<p>I've had problems in production where, after a while, some of the TCP listener opensips processes will get stuck and start consuming 100% CPU. The only way to resolve it is to restart opensips. This slowly starts to happen to each TCP listener until all are stuck and we stop serving requests. Even while there are still a few functioning listeners, service is degraded and some clients to opensips times out. I'm currently using the 1.11 branch at commit <a href="https://github.com/OpenSIPS/opensips/commit/17c5990ee35691451008380f13b2ffcbc59ccf79" class="commit-link"><tt>17c5990</tt></a>. I also observed this behavior on 1.8, and was hoping upgrading to 1.11 would solve the issue. Here's a typical stack trace while the process is stuck:</p>
<pre><code>#0 0x00007f7d3596ee87 in sched_yield () from /lib64/libc.so.6
#1 0x00000000004cfa4b in get_lock (lock=0x7f7cb1e48390) at fastlock.h:182
#2 tcp_send (send_sock=0x7f7d334f5298, type=type@entry=3,
buf=buf@entry=0x7f7cba888c68 "CANCEL sip:[redacted]@[redacted];transport=TLS SIP/2.0\r\nVia: SIP/2.0/TLS [redacted]:5061;branch=z9hG4bK0d93.b6c7a621.0;i=dc65\r\nVia: SIP/2.0/TLS [redacted]:51370;r"..., len=len@entry=559, to=to@entry=0x7f7cb4d4ba50, id=id@entry=0)
at tcp_main.c:1239
#3 0x00007f7d32419707 in msg_send (len=559,
buf=0x7f7cba888c68 "CANCEL sip:[redacted]@[redacted];transport=TLS SIP/2.0\r\nVia: SIP/2.0/TLS [redacted]:5061;branch=z9hG4bK0d93.b6c7a621.0;i=dc65\r\nVia: SIP/2.0/TLS [redacted]:51370;r"...,
id=0, to=0x7f7cb4d4ba50, proto=3, send_sock=<optimized out>)
at ../../forward.h:128
#4 send_pr_buffer (rb=rb@entry=0x7f7cb4d4ba30, buf=0x7f7cba888c68, len=559)
at t_funcs.c:69
#5 0x00007f7d3241eddf in t_forward_nonack (t=t@entry=0x7f7cb4d4b858,
p_msg=p_msg@entry=0x7f7d33cd8ed8, proxy=proxy@entry=0x0) at t_fwd.c:681
#6 0x00007f7d32419ccc in t_relay_to (p_msg=p_msg@entry=0x7f7d33cd8ed8,
proxy=proxy@entry=0x0, flags=flags@entry=0) at t_funcs.c:241
#7 0x00007f7d3243125d in w_t_relay (p_msg=0x7f7d33cd8ed8,
proxy=<optimized out>, flags=0x0) at tm.c:1206
#8 0x000000000041b071 in do_action (a=a@entry=0x7f7d3350b1b8,
msg=msg@entry=0x7f7d33cd8ed8) at action.c:1838
#9 0x0000000000420a39 in run_action_list (a=<optimized out>,
msg=0x7f7d33cd8ed8) at action.c:171
#10 0x000000000041df58 in do_action (a=a@entry=0x7f7d3350c3e0,
msg=msg@entry=0x7f7d33cd8ed8) at action.c:1105
#11 0x0000000000420a39 in run_action_list (a=<optimized out>,
msg=0x7f7d33cd8ed8) at action.c:171
#12 0x000000000041df58 in do_action (a=a@entry=0x7f7d3350c978,
msg=msg@entry=0x7f7d33cd8ed8) at action.c:1105
#13 0x0000000000420a39 in run_action_list (a=<optimized out>,
msg=0x7f7d33cd8ed8) at action.c:171
#14 0x000000000041efbd in do_action (a=a@entry=0x7f7d3350ca58,
msg=msg@entry=0x7f7d33cd8ed8) at action.c:1111
#15 0x0000000000420a39 in run_action_list (a=<optimized out>,
msg=0x7f7d33cd8ed8) at action.c:171
#16 0x000000000041efbd in do_action (a=a@entry=0x7f7d3350cb38,
msg=msg@entry=0x7f7d33cd8ed8) at action.c:1111
#17 0x0000000000420a39 in run_action_list (a=<optimized out>,
msg=0x7f7d33cd8ed8) at action.c:171
#18 0x000000000041efbd in do_action (a=a@entry=0x7f7d3350cc18,
msg=msg@entry=0x7f7d33cd8ed8) at action.c:1111
#19 0x0000000000420a39 in run_action_list (a=<optimized out>,
msg=0x7f7d33cd8ed8) at action.c:171
#20 0x000000000041efbd in do_action (a=a@entry=0x7f7d3350ccf8,
msg=msg@entry=0x7f7d33cd8ed8) at action.c:1111
#21 0x0000000000420c24 in run_action_list (msg=<optimized out>,
a=<optimized out>) at action.c:171
#22 run_actions (msg=0x7f7d33cd8ed8, a=<optimized out>) at action.c:136
#23 run_top_route (a=<optimized out>, msg=msg@entry=0x7f7d33cd8ed8)
at action.c:211
#24 0x0000000000489143 in receive_msg (
buf=buf@entry=0x876b08 <current_req+8> "CANCEL sip:[redacted]@[redacted];transport=TLS SIP/2.0\r\nVia: SIP/2.0/TLS [redacted]:51370;rport;branch=z9hG4bKPj4rfzZOxA8KjWkF2jX1wz5YLxkLNUxPsK\r\nMax-Forwards: 69\r\nF"..., len=len@entry=423, rcv_info=rcv_info@entry=0x7fffc506a590)
at receive.c:172
#25 0x00000000004d949f in tcp_read_req (con=con@entry=0x7f7cb2ff3bf8,
bytes_read=bytes_read@entry=0x7fffc506a670) at tcp_read.c:697
#26 0x00000000004da4e6 in handle_io (fm=0x7f7d3351f038, idx=idx@entry=-1,
event_type=1) at tcp_read.c:1033
#27 0x00000000004dc674 in io_wait_loop_epoll (h=<optimized out>,
t=<optimized out>, repeat=<optimized out>) at io_wait.h:845
#28 tcp_receive_loop (unix_sock=<optimized out>) at tcp_read.c:1141
#29 0x00000000004d4de4 in tcp_init_children (
chd_rank=chd_rank@entry=0x836a50 <chd_rank.22745>,
startup_done=startup_done@entry=0x0) at tcp_main.c:2373
#30 0x0000000000416e24 in main_loop () at main.c:1024
#31 main (argc=<optimized out>, argv=<optimized out>) at main.c:1634
</code></pre>
<p>We're using TLS here, but the same thing happens when using unencrypted TCP.</p>
<p>Unfortunately, I've been unable to reproduce this under controlled conditions; it just seems to happen randomly in production.</p>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br>Reply to this email directly or <a href="https://github.com/OpenSIPS/opensips/issues/410">view it on GitHub</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AFOciTrD1aXjxVyV9C4dv455qYWmZGqFks5nozf6gaJpZM4DcGMP.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/OpenSIPS/opensips/issues/410"></link>
<meta itemprop="name" content="View Issue"></meta>
</div>
<meta itemprop="description" content="View this Issue on GitHub"></meta>
</div>