<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi, Daniel!<br>
<br>
Are you doing any "expensive" processing on reply messages? Or in
failure route? Any DB or REST queries?<br>
It looks like the process is waiting for a lock in the transaction
module, which usually means that the transaction is stuck doing
something else.<br>
<br>
Best regards,<br>
</tt>
<pre class="moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a class="moz-txt-link-abbreviated" href="http://www.opensips-solutions.com">www.opensips-solutions.com</a></pre>
<div class="moz-cite-prefix">On 02/21/2017 05:43 PM, Daniel Zanutti
wrote:<br>
</div>
<blockquote
cite="mid:CAN0jgiPMvGv=MO39c=i6nUDSa1jGdH17tcebZ_oB+p9qJeGgew@mail.gmail.com"
type="cite">
<div dir="ltr">Hi
<div><br>
</div>
<div>We are having a weird issue on a high load server, could
you guys please help me find out a solution?</div>
<div><br>
</div>
<div>These opensips threads freeze for a few seconds at 100% CPU
and drop some packets. So some new calls fails to be
processed:</div>
<div><br>
</div>
<div><img src="cid:part1.D4E7CF96.D65FF5C4@opensips.org"
alt="Inline image 2" width="543" height="42"><br>
</div>
<div><br>
</div>
<div>Running on a Debian 8.6 x64 - 12 cores/48GB Ram server:</div>
<div>
<div><font color="#9900ff"># opensips -V</font></div>
<div><font color="#9900ff">version: opensips 1.11.6-notls
(x86_64/linux)</font></div>
<div><font color="#9900ff">flags: STATS: On, USE_IPV6,
USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</font></div>
<div><font color="#9900ff">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE
1024, BUF_SIZE 65535</font></div>
<div><font color="#9900ff">poll method support: poll,
epoll_lt, epoll_et, sigio_rt, select.</font></div>
<div><font color="#9900ff">main.c compiled on 10:46:04 Oct 20
2016 with gcc 4.9.2</font></div>
</div>
<div><br>
</div>
<div>I got the gdb of one of one frozen thread, it's seems to be
something related to fastlock:</div>
<div><br>
</div>
<div>
<div>(gdb) bt full</div>
<div><font color="#9900ff">#0 0x00007ff4cc9e4bb7 in
sched_yield () at ../sysdeps/unix/syscall-template.S:81</font></div>
<div><font color="#9900ff">No locals.</font></div>
<div><font color="#9900ff">#1 0x00007ff4cacd9d5d in get_lock
(lock=0x7ff3cb6e2678) at ../../evi/../fastlock.h:182</font></div>
<div><font color="#9900ff"> i = 0</font></div>
<div><font color="#9900ff">#2 _lock (s=0x7ff3cb6e2678) at
lock.h:102</font></div>
<div><font color="#9900ff">No locals.</font></div>
<div><font color="#9900ff">#3 reply_received
(p_msg=0x7ff4cc809c38) at t_reply.c:1556</font></div>
<div><font color="#9900ff"> msg_status = 302</font></div>
<div><font color="#9900ff"> last_uac_status = 503</font></div>
<div><font color="#9900ff"> branch = 1</font></div>
<div><font color="#9900ff"> reply_status =
<optimized out></font></div>
<div><font color="#9900ff"> timer = 140689351533517</font></div>
<div><font color="#9900ff"> cancel_bitmap = 0</font></div>
<div><font color="#9900ff"> uac = <optimized out></font></div>
<div><font color="#9900ff"> t = 0x7ff3cb6e0e20</font></div>
<div><font color="#9900ff"> backup_list = <optimized
out></font></div>
<div><font color="#9900ff"> has_reply_route = 1</font></div>
<div><font color="#9900ff"> __FUNCTION__ =
"reply_received"</font></div>
<div><font color="#9900ff">#4 0x000000000044d613 in
forward_reply (msg=msg@entry=0x7ff4cc809c38) at
forward.c:575</font></div>
<div><font color="#9900ff"> new_buf = 0x0</font></div>
<div><font color="#9900ff"> to = 0x0</font></div>
<div><font color="#9900ff"> new_len = 32755</font></div>
<div><font color="#9900ff"> mod = 0x7ff4cc71b2b0</font></div>
<div><font color="#9900ff"> proto = <optimized
out></font></div>
<div><font color="#9900ff"> id = 0</font></div>
<div><font color="#9900ff"> send_sock = <optimized
out></font></div>
<div><font color="#9900ff"> s = <optimized out></font></div>
<div><font color="#9900ff"> len = <optimized out></font></div>
<div><font color="#9900ff"> __FUNCTION__ =
"forward_reply"</font></div>
<div><font color="#9900ff">#5 0x00000000004340ff in
receive_msg (buf=<optimized out>, len=<optimized
out>, rcv_info=rcv_info@entry=0x7fff9f1182f0) at
receive.c:214</font></div>
<div><font color="#9900ff"> msg = 0x7ff4cc809c38</font></div>
<div><font color="#9900ff"> start = {tv_sec =
140735862113030, tv_usec = 140689389048549}</font></div>
<div><font color="#9900ff"> rc = 3</font></div>
<div><font color="#9900ff"> __FUNCTION__ =
"receive_msg"</font></div>
<div><font color="#9900ff">#6 0x0000000000462ef4 in
udp_rcv_loop () at udp_server.c:422</font></div>
<div><font color="#9900ff"> len = <optimized out></font></div>
<div><font color="#9900ff"> buf = "SIP/2.0 302 Moved
Temporarily\r\nVia: SIP/2.0/UDP
2.2.2.2:5060;rport=5060;received=2.2.2.2;branch=z9hG4bK9ef.5fb57785.1\r\nVia:
SIP/2.0/UDP 1.1.1.1:5060;rport=5060;received=1.1.1"...</font></div>
<div><font color="#9900ff"> from = 0x7ff4cc809c00</font></div>
<div><font color="#9900ff"> fromlen = 16</font></div>
<div><font color="#9900ff"> ri = {src_ip = {af = 2, len
= 4, u = {addrl = {41876154, 1}, addr32 = {41876154, 0, 1,
0}, addr16 = {64186, 638, 0, 0, 1, 0, 0, 0}, addr =
"\272\372~\002\000\000\000\000\001\000\000\000\000\000\000"}},
dst_ip = {af = 2,</font></div>
<div><font color="#9900ff"> len = 4, u = {addrl =
{769317832, 0}, addr32 = {769317832, 0, 0, 0}, addr16 =
{56264, 11738, 0, 0, 0, 0, 0, 0}, addr = "\310\333\332-",
'\000' <repeats 11 times>}}, src_port = 5060,
dst_port = 5060, proto = 1,</font></div>
<div><font color="#9900ff"> proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data
= "\023ĺ\372~\002\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
41876154},</font></div>
<div><font color="#9900ff"> sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2,
sin6_port = 50195, sin6_flowinfo = 41876154, sin6_addr =
{__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},</font></div>
<div><font color="#9900ff"> __u6_addr32 = {0,
0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7ff4cc71ade0}</font></div>
<div><font color="#9900ff"> p = <optimized out></font></div>
<div><font color="#9900ff"> msg = {</font></div>
<div><font color="#9900ff"> s = 0x81e780 <buf>
"SIP/2.0 302 Moved Temporarily\r\nVia: SIP/2.0/UDP
2.2.2.2:5060;rport=5060;received=2.2.2.2;branch=z9hG4bK9ef.5fb57785.1\r\nVia:
SIP/2.0/UDP 1.1.1.1:5060;rport=5060;received=1.1.1"...,
len = 537}</font></div>
<div><font color="#9900ff"> __FUNCTION__ =
"udp_rcv_loop"</font></div>
<div><font color="#9900ff">#7 0x00000000004161e0 in main_loop
() at main.c:929</font></div>
<div><font color="#9900ff"> i = 3</font></div>
<div><font color="#9900ff"> pid = <optimized out></font></div>
<div><font color="#9900ff"> si = 0x7ff4cc71ade0</font></div>
<div><font color="#9900ff"> startup_done = 0x0</font></div>
<div><font color="#9900ff"> chd_rank = 4</font></div>
<div><font color="#9900ff"> rc = <optimized out></font></div>
<div><font color="#9900ff"> load_p = 0x7ff3c61ac430</font></div>
<div><font color="#9900ff">#8 main (argc=<optimized
out>, argv=<optimized out>) at main.c:1613</font></div>
<div><font color="#9900ff"> cfg_log_stderr =
<optimized out></font></div>
<div><font color="#9900ff"> cfg_stream = <optimized
out></font></div>
<div><font color="#9900ff"> c = <optimized out></font></div>
<div><font color="#9900ff"> r = <optimized out></font></div>
<div><font color="#9900ff">---Type <return> to continue,
or q <return> to quit---</font></div>
<div><font color="#9900ff"> tmp = 0x7fff9f118efe ""</font></div>
<div><font color="#9900ff"> tmp_len = <optimized
out></font></div>
<div><font color="#9900ff"> port = <optimized
out></font></div>
<div><font color="#9900ff"> proto = <optimized
out></font></div>
<div><font color="#9900ff"> options = 0x5c0aa8
"f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"</font></div>
<div><font color="#9900ff"> ret = -1</font></div>
<div><font color="#9900ff"> seed = 3165191676</font></div>
<div><font color="#9900ff"> rfd = <optimized out></font></div>
<div><font color="#9900ff"> __FUNCTION__ = "main"</font></div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Any idea?</div>
<div><br>
</div>
<div>Thanks!</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</body>
</html>