<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi William,<br>
<br>
I suspect you may hit some deadlock (most like a wild guess, as
there is not much data to check). And the only advice I can give
you is to upgrade to 2.4 (simple one) or 3.0 (a bit more complex).<br>
<br>
Regards,<br>
</tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
OpenSIPS Summit, Amsterdam, May 2020
<a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2020Amsterdam/">https://www.opensips.org/events/Summit-2020Amsterdam/</a>
</pre>
<div class="moz-cite-prefix">On 3/9/20 8:22 PM, William Simon wrote:<br>
</div>
<blockquote type="cite"
cite="mid:72CEB63D-CBD7-415B-B7AA-92E58B344E15@stratusvideo.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">This is
opensips 2.2.7 (I understand no longer supported) . We found
last time this happened a very large TCP send-q from
opensips to a remote SIP TCP endpoint when running netstat.
It seems TCP gets blocked and does not recover. We do not
need to reboot the server to restore the service, only
restart opensips.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Monday, March 9, 2020 at 5:02 AM<br>
<b>To: </b>William Simon <a class="moz-txt-link-rfc2396E" href="mailto:wsimon@stratusvideo.com"><wsimon@stratusvideo.com></a>,
OpenSIPS users mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] opensips udp workers
lock up with sched_yield<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
William,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>That it's interesting. Most of the processes are idle
(waiting in the I/O reactor) and there are a bunch of them
blocked in a lock (same pattern). Nevertheless, the weird
thing is there is no active process (like doing something)
that may hold the lock. All procs are either blocked,
either idle.</tt><br>
<br>
<tt>What opensips version you have?</tt><br>
<br>
<tt>Also, is opensips recovering from this state? or you
need to do a reboot ?</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit, Amsterdam, May 2020<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2020Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2020Amsterdam/</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<div>
<p class="MsoNormal">On 2/28/20 4:20 PM, William Simon wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan-Andrei,
thank you for your insight. Yes, we also use SIP TCP &
TLS. I do not see any locks in the rest of the
“opensipsctl trap.” Perhaps you will be able to understand
it better. The trap is posted at <a
href="https://pastebin.com/1rs8fVEB"
moz-do-not-send="true">https://pastebin.com/1rs8fVEB</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thank you</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">William
Simon</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Friday, February 28, 2020 at 4:23 AM<br>
<b>To: </b>OpenSIPS users mailling list <a
href="mailto:users@lists.opensips.org"
moz-do-not-send="true">
<users@lists.opensips.org></a>, William Simon <a
href="mailto:wsimon@stratusvideo.com"
moz-do-not-send="true">
<wsimon@stratusvideo.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] opensips udp
workers lock up with sched_yield</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
William,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New""><br>
<br>
<tt>That sched_yield translates into waiting for a lock.
As the backtrace (a bit crippled) shows as coming from
"send_pr_buffer" (which is responsible for sending out
on the network the buffer of a SIP msg), I suspect the
transport is TCP or TLS (missing frame #1), as they are
using locking. So you have the backtraces from all the
procs? this will help to identify the proc holding the
lock and blocking all the other procs.</tt><br>
<br>
<tt>Best regards,</tt><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit, Amsterdam, May 2020<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2020Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2020Amsterdam/</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp, Miami, March 2020<o:p></o:p></pre>
<pre> <a href="https://opensips.org/training/OpenSIPS_Bootcamp_2020/" moz-do-not-send="true">https://opensips.org/training/OpenSIPS_Bootcamp_2020/</a><o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<div>
<p class="MsoNormal">On 2/28/20 3:58 AM, William Simon
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">In a SIP
video environment we have a pair of opensips servers
load balancing traffic to freeswitch. The call volume is
modest among the two proxies, about 400 concurrent calls
at peak times.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We are
occasionally seeing opensips lock up and stop responding
to SIP traffic. There is no error in the syslog and no
indication of resource exhaustion on the VM (it is a
4-core VMware instance with 4GB of RAM). Once opensips
locks up, CPU soon reaches 100%, but before that, it was
not using even 50% of the CPU.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Get_statistics
shows that neither the shared memory nor pkg memory are
heavily used. They are set at 64M / 4M</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">opensipsctl
trace shows this on the udp worker threads (children=8
in config – it was previously set to children=4 and
showed the same behavior)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[Thread
debugging using libthread_db enabled]</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Using
host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">0x00007f5b14028bb7
in sched_yield () at
../sysdeps/unix/syscall-template.S:81</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#0
0x00007f5b14028bb7 in sched_yield () at
../sysdeps/unix/syscall-template.S:81</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
locals.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#1
0x00000000005323a5 in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#2
0x00007f5b0ec6c48f in send_pr_buffer () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#3
0x00007f5b0ec9eb9b in t_forward_nonack () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#4
0x00007f5b0ec6defe in t_relay_to () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#5
0x00007f5b0ec815ee in ?? () from
/usr/lib/x86_64-linux-gnu/opensips/modules/tm.so</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#6
0x000000000042b20a in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#7
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#8
0x000000000046d3bc in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#9
0x000000000046cc1d in eval_expr ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#10
0x000000000046cc39 in eval_expr ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#11
0x000000000046cc09 in eval_expr ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#12
0x000000000042b19a in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#13
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#14
0x00000000004306ba in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#15
0x000000000042da9a in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#16
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#17
0x000000000042e62e in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#18
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#19
0x000000000042e62e in do_action ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#20
0x0000000000430590 in run_action_list ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#21
0x00000000004308d0 in run_top_route ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#22
0x0000000000436ef3 in receive_msg ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#23
0x000000000052d5c5 in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#24
0x000000000051536d in ?? ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#25
0x000000000051837a in udp_rcv_loop ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#26
0x0000000000519c38 in udp_start_processes ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#27
0x000000000041c38a in main ()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">No
symbol table info available.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">---end
82753
-------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</blockquote>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
</div>
<br>
<br>
“The information transmitted is intended only for the person or
entity to which it is addressed and may contain proprietary,
business-confidential and/or privileged material. If you are not
the intended recipient of this message you are hereby notified
that any use, review, retransmission, dissemination, distribution,
reproduction or any action taken in reliance upon this message is
prohibited. If you received this in error, please contact the
sender and delete the material from any computer.”
</blockquote>
<br>
</body>
</html>