<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><tt>Hi Vasilios,</tt></p>
<p><tt>Some things to consider:</tt></p>
<p><tt> - what are some other UDP processes doing? (e.g. 1850,
1855, etc.)</tt></p>
<p><tt> - when doing "make install", the binaries will include
debug symbols regardless of any menuconfig options. However, if
you want to obtain more clear backtraces, you should also enable
"DBG_MALLOC" compile flag, which will also disable compiler
optimizations.<br>
</tt></p>
<p><tt> - any obvious bottlenecks which could lead to the UDP
queue being stuck? 100% CPU? Useful logs?</tt></p>
<p><tt>Regards,<br>
</tt></p>
<pre class="moz-signature" cols="72">Liviu Chircu
OpenSIPS Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a class="moz-txt-link-freetext" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a></pre>
<div class="moz-cite-prefix">On 29.03.2017 12:25, Vasilios
Tzanoudakis wrote:<br>
</div>
<blockquote
cite="mid:CADTki+BVr7DN7679v_AjvdPS+WT8U4LHKWv=fy7fJNNXGCxtzQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Dear all,</div>
<div><br>
</div>
<div>After 11 days os usage Opensips process was not responding
at all.</div>
<div><br>
</div>
<div>This is the second time I am facing this kind of issue. </div>
<div>I am sending you all the information below: </div>
<div><br>
</div>
<div>System has 4 CPU Cores / 4 GB Ram</div>
<div><br>
</div>
<div>Opensips version 2.2.2<br>
</div>
<div><br>
</div>
<div><b>Run parameters : </b></div>
<div>
<div>S_MEMORY=2048<br>
</div>
<div>P_MEMORY=1024<br>
</div>
</div>
<div><br>
</div>
<div><b>uname -a</b></div>
<div>Linux 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18
18:10:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux</div>
<div><br>
</div>
<div><b>Compiled with the following flags:</b></div>
<div><br>
</div>
<div>PKG_MALLOC</div>
<div>SHM_MMAP</div>
<div>USE_MCAST</div>
<div>DISABLE_NAGLE</div>
<div>STATISTICS</div>
<div>HAVE_RESOLV_RES</div>
<div>F_MALLOC</div>
<div>F_MALLOC_OPTIMIZATIONS</div>
<div>NO_DEBUG</div>
<div><br>
</div>
<div><b>and with the following extra modules : </b></div>
<div><br>
</div>
<div>db_mysql</div>
<div>dialplan</div>
<div>regex</div>
<div>snmpstats</div>
<div><br>
</div>
<div><b>Loaded modules in opensips.cfg</b></div>
<div><br>
</div>
<div>
<div>loadmodule "proto_udp.so"</div>
<div>loadmodule "mi_datagram.so"<br>
</div>
<div>loadmodule "signaling.so"</div>
<div>loadmodule "sl.so"</div>
<div>loadmodule "tm.so"</div>
<div>loadmodule "rr.so"</div>
<div>loadmodule "maxfwd.so"</div>
<div>loadmodule "sipmsgops.so"</div>
<div>loadmodule "mi_fifo.so"</div>
<div>loadmodule "uri.so"</div>
<div>loadmodule "db_mysql.so"</div>
<div>loadmodule "usrloc.so"</div>
<div>loadmodule "registrar.so"</div>
<div>loadmodule "auth.so"</div>
<div>loadmodule "auth_db.so"</div>
<div>loadmodule "dialog.so"</div>
<div>loadmodule "nathelper.so"</div>
<div>loadmodule "rtpproxy.so"</div>
<div>loadmodule "drouting.so"</div>
<div>loadmodule "uac.so"</div>
</div>
<div><br>
</div>
<div><b>/usr/local/opensips2/sbin/opensipsctl fifo ps</b></div>
<div><br>
</div>
<div>Process:: ID=0 PID=1843 Type=attendant</div>
<div>Process:: ID=1 PID=1845 Type=MI FIFO</div>
<div>Process:: ID=2 PID=1846 Type=MI Datagram</div>
<div>Process:: ID=3 PID=1847 Type=time_keeper</div>
<div>Process:: ID=4 PID=1849 Type=timer</div>
<div>Process:: ID=5 PID=1850 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=6 PID=1851 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=7 PID=1852 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=8 PID=1853 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=9 PID=1854 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=10 PID=1855 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=11 PID=1856 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=12 PID=1857 Type=SIP receiver
udp:X.X.X.X:65100 </div>
<div>Process:: ID=13 PID=1859 Type=Timer handler</div>
<div><br>
</div>
<div><b>gdb /usr/local/opensips2/sbin/opensips 1851</b><br>
</div>
<div><br>
</div>
<div>Reading symbols from
/lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols
found)...done.</div>
<div>Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1</div>
<div>Reading symbols from
/lib/x86_64-linux-gnu/libnss_dns.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libnss_dns-2.19.so...done.</div>
<div>done.</div>
<div>Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2</div>
<div>0x00007faee52c36cd in parse_backslash
(offset=0x7ffc00000000, words=<optimized out>,
max_length=0x7ffc32c06040, word_length=0xd,
word=0x636f732e3379786f) at wordexp.c:222</div>
<div>222<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>wordexp.c:
No such file or directory.</div>
<div><br>
</div>
<div><b>(gdb) bt<br>
</b></div>
<div>#0 0x00007faee52c36cd in parse_backslash
(offset=0x7ffc00000000, words=<optimized out>,
max_length=0x7ffc32c06040, word_length=0xd,
word=0x636f732e3379786f) at wordexp.c:222</div>
<div>#1 parse_backtick (ifs_white=0x722f7261762f0001 <error:
<b>Cannot access memory at address</b> 0x722f7261762f0001>,
ifs=0x7fae24ccc960 "", pwordexp=0x838fd4 <buf+916>,
flags=0, offset=0x7ffc00000000, </div>
<div> words=<optimized out>, max_length=0x1,
word_length=0x7ffc32c06060, word=0x7ffc32c060c0) at
wordexp.c:2147</div>
<div>#2 wordexp (words=<optimized out>,
pwordexp=<optimized out>, flags=0) at wordexp.c:2359</div>
<div>#3 0x00007ffc32c06220 in ?? ()</div>
<div>#4 0x0000000000000000 in ?? ()</div>
<div><br>
</div>
<div><b>(gdb) bt full<br>
</b></div>
<div>#0 0x00007faee52c36cd in parse_backslash
(offset=0x7ffc00000000, words=<optimized out>,
max_length=0x7ffc32c06040, word_length=0xd,
word=0x636f732e3379786f) at wordexp.c:222</div>
<div>No locals.</div>
<div>#1 parse_backtick (ifs_white=0x722f7261762f0001 <error:
<b>Cannot access memory at address </b>0x722f7261762f0001>,
ifs=0x7fae24ccc960 "", pwordexp=0x838fd4 <buf+916>,
flags=0, offset=0x7ffc00000000, </div>
<div> words=<optimized out>, max_length=0x1,
word_length=0x7ffc32c06060, word=0x7ffc32c060c0) at
wordexp.c:2147</div>
<div> squoting = <optimized out></div>
<div> comm_length = 8622050</div>
<div> <b>comm = 0x6 <error: Cannot access memory at
address 0x6></b></div>
<div> error = <optimized out></div>
<div> comm_maxlen = 8700952</div>
<div>#2 wordexp (words=<optimized out>,
pwordexp=<optimized out>, flags=0) at wordexp.c:2359</div>
<div> words_offset = 0</div>
<div> word_length = 0</div>
<div> max_length = 140388071391744</div>
<div> word = 0x7ffc32c065c8 ""</div>
<div> error = <optimized out></div>
<div> ifs = 0x7fae24ccc960 ""</div>
<div> ifs_white = "\000\000\000"</div>
<div> old_word = {we_wordc = 0, we_wordv = 0x0, we_offs =
0}</div>
<div>#3 0x00007ffc32c06220 in ?? ()</div>
<div>No symbol table info available.</div>
<div>#4 0x0000000000000000 in ?? ()</div>
<div>No symbol table info available.</div>
<div><br>
</div>
<div><b>ldd -v /usr/local/opensips2/sbin/opensips</b></div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>linux-vdso.so.1
=> (0x00007fff96f49000)</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libdl.so.2
=> /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f75c45de000)</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libresolv.so.2
=> /lib/x86_64-linux-gnu/libresolv.so.2
(0x00007f75c43c3000)</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
=> /lib/x86_64-linux-gnu/libc.so.6 (0x00007f75c3ffa000)</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>/lib64/ld-linux-x86-64.so.2
(0x000055e9cad86000)</div>
<div><br>
</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Version
information:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>/usr/local/opensips2/sbin/opensips:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libresolv.so.2
(GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libresolv.so.2</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libdl.so.2
(GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.15) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>/lib/x86_64-linux-gnu/libdl.so.2:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ld-linux-x86-64.so.2
(GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>/lib/x86_64-linux-gnu/libresolv.so.2:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>libc.so.6
(GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>/lib/x86_64-linux-gnu/libc.so.6:</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ld-linux-x86-64.so.2
(GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2</div>
<div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>ld-linux-x86-64.so.2
(GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2</div>
<div><br>
</div>
<div><b># free -m</b></div>
<div> total used free shared
buffers cached</div>
<div>Mem: 3951 2842 1109 28
66 2156</div>
<div>-/+ buffers/cache: 619 3332</div>
<div>Swap: 4091 90 4001</div>
<div><br>
</div>
<div><b># vmstat </b></div>
<div>procs -----------memory---------- ---swap-- -----io----
-system-- ------cpu-----</div>
<div> r b swpd free buff cache si so bi bo
in cs us sy id wa st</div>
<div> 2 0 92396 1136328 68576 2207768 0 0 5 14
10 2 1 2 97 0 0</div>
<div><br>
</div>
<div><b># w</b></div>
<div> 11:43:18 up 11 days, 23:16, 1 user, load average: 2.46,
2.39, 2.36</div>
<div><br>
</div>
<div><b># netstat -ulnp</b></div>
<div>Active Internet connections (only servers)</div>
<div>Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name</div>
<div>udp 426496 0 X.X.X.X:65100 0.0.0.0:*
1843/opensips </div>
<div><br>
</div>
<div>also after restarting opensips process system started to
receive traffic again BUT I got the following errors on syslog
regarding rtpproxy</div>
<div><br>
</div>
<div>Mar 29 09:57:32 rtp1
/usr/local/opensips2/sbin/opensips[1850]:
ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply
from rtp proxy</div>
<div>Mar 29 09:58:09 rtp1
/usr/local/opensips2/sbin/opensips[1850]:
ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply
from rtp proxy</div>
<div>Mar 29 09:58:10 rtp1
/usr/local/opensips2/sbin/opensips[1857]:
ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply
from rtp proxy</div>
<div>Mar 29 09:58:12 rtp1
/usr/local/opensips2/sbin/opensips[1850]:
ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply
from rtp proxy</div>
<div>Mar 29 09:58:26 rtp1
/usr/local/opensips2/sbin/opensips[1850]:
ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply
from rtp proxy</div>
<div>Mar 29 09:58:26 rtp1
/usr/local/opensips2/sbin/opensips[1851]:
ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply
from rtp proxy</div>
<div>Mar 29 09:58:26 rtp1
/usr/local/opensips2/sbin/opensips[1853]:
ERROR:rtpproxy:force_rtp_proxy_body: incorrect port 0 in reply
from rtp proxy</div>
<div><br>
</div>
<div>and I had to reboot rtpproxy processes too for traffic to
pass through without problems. <br>
</div>
<div>I don't know if this is another problem with rtpproxy but I
am just mentioning it.</div>
<div><br>
</div>
<div>thank you in advance</div>
<div><br>
</div>
<br clear="all">
<div>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div
style="color:rgb(102,102,102);font-stretch:normal;font-size:13px;font-family:arial,sans-serif;margin-bottom:3px">
<b style="font-size:15px;color:rgb(17,17,17)">Vasilios
Tzanoudakis</b> | Technology Director
</div>
<div
style="font-stretch:normal;font-size:12px;font-family:arial,sans-serif;line-height:18px;padding-top:5px;color:rgb(102,102,102)">t.
+30-212-222-8003 | f. +30-212-222-8001
<br>
2, Klisthenous Str., Metamorfosi, 144 52,
Attica, Greece<br>
<a moz-do-not-send="true"
href="mailto:vasilios.tzanoudakis@voiceland.gr"
style="color:rgb(70,22,107);text-decoration:none" target="_blank">vasilios.tzanoudakis@voiceland.gr</a>
|
<a moz-do-not-send="true"
href="http://www.voiceland.gr/"
style="color:rgb(70,22,107);text-decoration:none"
target="_blank">www.voiceland.gr</a>
</div>
<div style="padding-top:5px"><img
moz-do-not-send="true"
src="https://docs.google.com/a/voiceland.gr/uc?id=0B-vM3smbJoi8YnZHWTNyR1RyVTQ&export=download"
width="200" height="50"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</body>
</html>