<div dir="ltr">Dear <span style="color:rgb(0,0,0);white-space:pre-wrap">Liviu,</span><div><br></div><div><div>Regarding other UDP processes, I tried 1850 but wasn't showing backtraces with memory information on it or any other info It seemed to be normal but i have't kept them. 1851,1852 gave me almost the same results.</div><div><br></div><div>Also Cpu load started increasing by the time the problem started happening. Here is the screenshot to see. No other reason for High CPU load. </div><div>CPU Usage came back to normal after restarting processes.</div><div>here is the cpu graph on this server : <a href="http://oi68.tinypic.com/2m79irs.jpg">http://oi68.tinypic.com/2m79irs.jpg</a></div><div><br></div><div>I will try to enable DBG_MALLOC and see if this happens again , so as I can understand you don't see any obvious thing around....</div><div><br></div><div>Thank you</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div style="color:rgb(102,102,102);font-stretch:normal;font-size:13px;font-family:arial,sans-serif;margin-bottom:3px">
<b style="font-size:15px;color:rgb(17,17,17)">Vasilios Tzanoudakis</b> | Technology Director
</div>
<div style="font-stretch:normal;font-size:12px;font-family:arial,sans-serif;line-height:18px;padding-top:5px;color:rgb(102,102,102)">
t. +30-212-222-8003 | f. +30-212-222-8001
<br>2, Klisthenous Str., Metamorfosi, 144 52, Attica, Greece<br>
<a href="mailto:vasilios.tzanoudakis@voiceland.gr" style="color:rgb(70,22,107);text-decoration:none" target="_blank">vasilios.tzanoudakis@voiceland.gr</a>
|
<a href="http://www.voiceland.gr/" style="color:rgb(70,22,107);text-decoration:none" target="_blank">www.voiceland.gr</a>
</div>
<div style="padding-top:5px"><img src="https://docs.google.com/a/voiceland.gr/uc?id=0B-vM3smbJoi8YnZHWTNyR1RyVTQ&export=download" width="200" height="50"><br></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Wed, Mar 29, 2017 at 12:25 PM, Vasilios Tzanoudakis <span dir="ltr"><<a href="mailto:vasilios.tzanoudakis@voiceland.gr" target="_blank">vasilios.tzanoudakis@voiceland.gr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Dear all,</div><div><br></div><div>After 11 days os usage Opensips process was not responding at all.</div><div><br></div><div>This is the second time I am facing this kind of issue. </div><div>I am sending you all the information below: </div><div><br></div><div>System has 4 CPU Cores / 4 GB Ram</div><div><br></div><div>Opensips version 2.2.2<br></div><div><br></div><div><b>Run parameters : </b></div><div><div>S_MEMORY=2048<br></div><div>P_MEMORY=1024<br></div></div><div><br></div><div><b>uname -a</b></div><div>Linux 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux</div><div><br></div><div><b>Compiled with the following flags:</b></div><div><br></div><div>PKG_MALLOC</div><div>SHM_MMAP</div><div>USE_MCAST</div><div>DISABLE_NAGLE</div><div>STATISTICS</div><div>HAVE_RESOLV_RES</div><div>F_MALLOC</div><div>F_MALLOC_OPTIMIZATIONS</div><div>NO_DEBUG</div><div><br></div><div><b>and with the following extra modules : </b></div><div><br></div><div>db_mysql</div><div>dialplan</div><div>regex</div><div>snmpstats</div><div><br></div><div><b>Loaded modules in opensips.cfg</b></div><div><br></div><div><div>loadmodule "proto_udp.so"</div><div>loadmodule "mi_datagram.so"<br></div><div>loadmodule "signaling.so"</div><div>loadmodule "sl.so"</div><div>loadmodule "tm.so"</div><div>loadmodule "rr.so"</div><div>loadmodule "maxfwd.so"</div><div>loadmodule "sipmsgops.so"</div><div>loadmodule "mi_fifo.so"</div><div>loadmodule "uri.so"</div><div>loadmodule "db_mysql.so"</div><div>loadmodule "usrloc.so"</div><div>loadmodule "registrar.so"</div><div>loadmodule "auth.so"</div><div>loadmodule "auth_db.so"</div><div>loadmodule "dialog.so"</div><div>loadmodule "nathelper.so"</div><div>loadmodule "rtpproxy.so"</div><div>loadmodule "drouting.so"</div><div>loadmodule "uac.so"</div></div><div><br></div><div><b>/usr/local/opensips2/sbin/<wbr>opensipsctl fifo ps</b></div><div><br></div><div>Process:: ID=0 PID=1843 Type=attendant</div><div>Process:: ID=1 PID=1845 Type=MI FIFO</div><div>Process:: ID=2 PID=1846 Type=MI Datagram</div><div>Process:: ID=3 PID=1847 Type=time_keeper</div><div>Process:: ID=4 PID=1849 Type=timer</div><div>Process:: ID=5 PID=1850 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=6 PID=1851 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=7 PID=1852 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=8 PID=1853 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=9 PID=1854 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=10 PID=1855 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=11 PID=1856 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=12 PID=1857 Type=SIP receiver udp:X.X.X.X:65100 </div><div>Process:: ID=13 PID=1859 Type=Timer handler</div><div><br></div><div><b>gdb /usr/local/opensips2/sbin/<wbr>opensips 1851</b><br></div><div><br></div><div>Reading symbols from /lib/x86_64-linux-gnu/libgcc_<wbr>s.so.1...(no debugging symbols found)...done.</div><div>Loaded symbols for /lib/x86_64-linux-gnu/libgcc_<wbr>s.so.1</div><div>Reading symbols from /lib/x86_64-linux-gnu/libnss_<wbr>dns.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-<wbr>linux-gnu/<a href="http://libnss_dns-2.19.so">libnss_dns-2.19.so</a>..<wbr>.done.</div><div>done.</div><div>Loaded symbols for /lib/x86_64-linux-gnu/libnss_<wbr>dns.so.2</div><div>0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000, words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd, word=0x636f732e3379786f) at wordexp.c:222</div><div>222<span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>wordexp.c: No such file or directory.</div><div><br></div><div><b>(gdb) bt<br></b></div><div>#0 0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000, words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd, word=0x636f732e3379786f) at wordexp.c:222</div><div>#1 parse_backtick (ifs_white=0x722f7261762f0001 <error: <b>Cannot access memory at address</b> 0x722f7261762f0001>, ifs=0x7fae24ccc960 "", pwordexp=0x838fd4 <buf+916>, flags=0, offset=0x7ffc00000000, </div><div> words=<optimized out>, max_length=0x1, word_length=0x7ffc32c06060, word=0x7ffc32c060c0) at wordexp.c:2147</div><div>#2 wordexp (words=<optimized out>, pwordexp=<optimized out>, flags=0) at wordexp.c:2359</div><div>#3 0x00007ffc32c06220 in ?? ()</div><div>#4 0x0000000000000000 in ?? ()</div><div><br></div><div><b>(gdb) bt full<br></b></div><div>#0 0x00007faee52c36cd in parse_backslash (offset=0x7ffc00000000, words=<optimized out>, max_length=0x7ffc32c06040, word_length=0xd, word=0x636f732e3379786f) at wordexp.c:222</div><div>No locals.</div><div>#1 parse_backtick (ifs_white=0x722f7261762f0001 <error: <b>Cannot access memory at address </b>0x722f7261762f0001>, ifs=0x7fae24ccc960 "", pwordexp=0x838fd4 <buf+916>, flags=0, offset=0x7ffc00000000, </div><div> words=<optimized out>, max_length=0x1, word_length=0x7ffc32c06060, word=0x7ffc32c060c0) at wordexp.c:2147</div><div> squoting = <optimized out></div><div> comm_length = 8622050</div><div> <b>comm = 0x6 <error: Cannot access memory at address 0x6></b></div><div> error = <optimized out></div><div> comm_maxlen = 8700952</div><div>#2 wordexp (words=<optimized out>, pwordexp=<optimized out>, flags=0) at wordexp.c:2359</div><div> words_offset = 0</div><div> word_length = 0</div><div> max_length = 140388071391744</div><div> word = 0x7ffc32c065c8 ""</div><div> error = <optimized out></div><div> ifs = 0x7fae24ccc960 ""</div><div> ifs_white = "\000\000\000"</div><div> old_word = {we_wordc = 0, we_wordv = 0x0, we_offs = 0}</div><div>#3 0x00007ffc32c06220 in ?? ()</div><div>No symbol table info available.</div><div>#4 0x0000000000000000 in ?? ()</div><div>No symbol table info available.</div><div><br></div><div><b>ldd -v /usr/local/opensips2/sbin/<wbr>opensips</b></div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>linux-vdso.so.1 => (0x00007fff96f49000)</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libdl.so.2 => /lib/x86_64-linux-gnu/libdl.<wbr>so.2 (0x00007f75c45de000)</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libresolv.so.2 => /lib/x86_64-linux-gnu/<wbr>libresolv.so.2 (0x00007f75c43c3000)</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 => /lib/x86_64-linux-gnu/libc.so.<wbr>6 (0x00007f75c3ffa000)</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>/lib64/ld-linux-x86-64.so.2 (0x000055e9cad86000)</div><div><br></div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>Version information:</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>/usr/local/opensips2/sbin/<wbr>opensips:</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libresolv.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/<wbr>libresolv.so.2</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.<wbr>so.2</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>/lib/x86_64-linux-gnu/libdl.<wbr>so.2:</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>/lib/x86_64-linux-gnu/<wbr>libresolv.so.2:</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.<wbr>6</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>/lib/x86_64-linux-gnu/libc.so.<wbr>6:</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2</div><div><span class="gmail-m_6393084154644316687gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2</div><div><br></div><div><b># free -m</b></div><div> total used free shared buffers cached</div><div>Mem: 3951 2842 1109 28 66 2156</div><div>-/+ buffers/cache: 619 3332</div><div>Swap: 4091 90 4001</div><div><br></div><div><b># vmstat </b></div><div>procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----</div><div> r b swpd free buff cache si so bi bo in cs us sy id wa st</div><div> 2 0 92396 1136328 68576 2207768 0 0 5 14 10 2 1 2 97 0 0</div><div><br></div><div><b># w</b></div><div> 11:43:18 up 11 days, 23:16, 1 user, load average: 2.46, 2.39, 2.36</div><div><br></div><div><b># netstat -ulnp</b></div><div>Active Internet connections (only servers)</div><div>Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name</div><div>udp 426496 0 X.X.X.X:65100 0.0.0.0:* 1843/opensips </div><div><br></div><div>also after restarting opensips process system started to receive traffic again BUT I got the following errors on syslog regarding rtpproxy</div><div><br></div><div>Mar 29 09:57:32 rtp1 /usr/local/opensips2/sbin/<wbr>opensips[1850]: ERROR:rtpproxy:force_rtp_<wbr>proxy_body: incorrect port 0 in reply from rtp proxy</div><div>Mar 29 09:58:09 rtp1 /usr/local/opensips2/sbin/<wbr>opensips[1850]: ERROR:rtpproxy:force_rtp_<wbr>proxy_body: incorrect port 0 in reply from rtp proxy</div><div>Mar 29 09:58:10 rtp1 /usr/local/opensips2/sbin/<wbr>opensips[1857]: ERROR:rtpproxy:force_rtp_<wbr>proxy_body: incorrect port 0 in reply from rtp proxy</div><div>Mar 29 09:58:12 rtp1 /usr/local/opensips2/sbin/<wbr>opensips[1850]: ERROR:rtpproxy:force_rtp_<wbr>proxy_body: incorrect port 0 in reply from rtp proxy</div><div>Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/<wbr>opensips[1850]: ERROR:rtpproxy:force_rtp_<wbr>proxy_body: incorrect port 0 in reply from rtp proxy</div><div>Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/<wbr>opensips[1851]: ERROR:rtpproxy:force_rtp_<wbr>proxy_body: incorrect port 0 in reply from rtp proxy</div><div>Mar 29 09:58:26 rtp1 /usr/local/opensips2/sbin/<wbr>opensips[1853]: ERROR:rtpproxy:force_rtp_<wbr>proxy_body: incorrect port 0 in reply from rtp proxy</div><div><br></div><div>and I had to reboot rtpproxy processes too for traffic to pass through without problems. <br></div><div>I don't know if this is another problem with rtpproxy but I am just mentioning it.</div><div><br></div><div>thank you in advance</div><span class="gmail-HOEnZb"><font color="#888888"><div><br></div><br clear="all"><div><div class="gmail-m_6393084154644316687gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div style="color:rgb(102,102,102);font-stretch:normal;font-size:13px;font-family:arial,sans-serif;margin-bottom:3px">
<b style="font-size:15px;color:rgb(17,17,17)">Vasilios Tzanoudakis</b> | Technology Director
</div>
<div style="font-stretch:normal;font-size:12px;font-family:arial,sans-serif;line-height:18px;padding-top:5px;color:rgb(102,102,102)">
t. <a href="tel:+30%2021%202222%208003" value="+302122228003" target="_blank">+30-212-222-8003</a> | f. <a href="tel:+30%2021%202222%208001" value="+302122228001" target="_blank">+30-212-222-8001</a>
<br>2, Klisthenous Str., Metamorfosi, 144 52, Attica, Greece<br>
<a href="mailto:vasilios.tzanoudakis@voiceland.gr" style="color:rgb(70,22,107);text-decoration:none" target="_blank">vasilios.tzanoudakis@<wbr>voiceland.gr</a>
|
<a href="http://www.voiceland.gr/" style="color:rgb(70,22,107);text-decoration:none" target="_blank">www.voiceland.gr</a>
</div>
<div style="padding-top:5px"><img src="https://docs.google.com/a/voiceland.gr/uc?id=0B-vM3smbJoi8YnZHWTNyR1RyVTQ&export=download" width="200" height="50"><br></div></div></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div><br></div></div>