<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi William,<br>
<br>
That it's interesting. Most of the processes are idle (waiting in
the I/O reactor) and there are a bunch of them blocked in a lock
(same pattern). Nevertheless, the weird thing is there is no
active process (like doing something) that may hold the lock. All
procs are either blocked, either idle.<br>
<br>
What opensips version you have?<br>
<br>
Also, is opensips recovering from this state? or you need to do a
reboot ?<br>
<br>
Regards,<br>
</tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
OpenSIPS Summit, Amsterdam, May 2020
<a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2020Amsterdam/">https://www.opensips.org/events/Summit-2020Amsterdam/</a>
</pre>
<div class="moz-cite-prefix">On 2/28/20 4:20 PM, William Simon
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:EBE69AD5-62CC-4AB1-B9E0-C3275669B5A4@stratusvideo.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan-Andrei,
thank you for your insight. Yes, we also use SIP TCP &
TLS. I do not see any locks in the rest of the “opensipsctl
trap.” Perhaps you will be able to understand it better. The
trap is posted at <a href="https://pastebin.com/1rs8fVEB"
moz-do-not-send="true">https://pastebin.com/1rs8fVEB</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thank you<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">William
Simon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Friday, February 28, 2020 at 4:23 AM<br>
<b>To: </b>OpenSIPS users mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a>, William Simon
<a class="moz-txt-link-rfc2396E" href="mailto:wsimon@stratusvideo.com"><wsimon@stratusvideo.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] opensips udp workers
lock up with sched_yield<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
William,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>That sched_yield translates into waiting for a lock. As
the backtrace (a bit crippled) shows as coming from
"send_pr_buffer" (which is responsible for sending out on
the network the buffer of a SIP msg), I suspect the
transport is TCP or TLS (missing frame #1), as they are
using locking. So you have the backtraces from all the
procs? this will help to identify the proc holding the
lock and blocking all the other procs.</tt><br>
<br>
<tt>Best regards,</tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit, Amsterdam, May 2020<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2020Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2020Amsterdam/</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp, Miami, March 2020<o:p></o:p></pre>
<pre> <a href="https://opensips.org/training/OpenSIPS_Bootcamp_2020/" moz-do-not-send="true">https://opensips.org/training/OpenSIPS_Bootcamp_2020/</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<div>
<p class="MsoNormal">On 2/28/20 3:58 AM, William Simon wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">In a SIP
video environment we have a pair of opensips servers load
balancing traffic to freeswitch. The call volume is modest
among the two proxies, about 400 concurrent calls at peak
times.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We are
occasionally seeing opensips lock up and stop responding
to SIP traffic. There is no error in the syslog and no
indication of resource exhaustion on the VM (it is a
4-core VMware instance with 4GB of RAM). Once opensips
locks up, CPU soon reaches 100%, but before that, it was
not using even 50% of the CPU.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Get_statistics
shows that neither the shared memory nor pkg memory are
heavily used. They are set at 64M / 4M</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">opensipsctl
trace shows this on the udp worker threads (children=8 in
config – it was previously set to children=4 and showed
the same behavior)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[Thread
debugging using libthread_db enabled]</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Using host
libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">0x00007f5b14028bb7
in sched_yield () at ../sysdeps/unix/syscall-template.S:81</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#0
0x00007f5b14028bb7 in sched_yield () at
../sysdeps/unix/syscall-template.S:81</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No locals.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#1
0x00000000005323a5 in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#2
0x00007f5b0ec6c48f in send_pr_buffer () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#3
0x00007f5b0ec9eb9b in t_forward_nonack () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#4
0x00007f5b0ec6defe in t_relay_to () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#5
0x00007f5b0ec815ee in ?? () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#6
0x000000000042b20a in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#7
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#8
0x000000000046d3bc in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#9
0x000000000046cc1d in eval_expr ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#10
0x000000000046cc39 in eval_expr ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#11
0x000000000046cc09 in eval_expr ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#12
0x000000000042b19a in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#13
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#14
0x00000000004306ba in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#15
0x000000000042da9a in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#16
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#17
0x000000000042e62e in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#18
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#19
0x000000000042e62e in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#20
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#21
0x00000000004308d0 in run_top_route ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#22
0x0000000000436ef3 in receive_msg ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#23
0x000000000052d5c5 in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#24
0x000000000051536d in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#25
0x000000000051837a in udp_rcv_loop ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#26
0x0000000000519c38 in udp_start_processes ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#27
0x000000000041c38a in main ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No symbol
table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---end
82753
-------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<br>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>