<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi Be<tt>n,<br>
<br>
<tt>DO you have the <tt>backtraces from <tt>more <tt>similar
crashes ? may <tt>there is a<tt> pattern there.<br>
<br>
<tt>Regards,</tt><br>
</tt></tt></tt></tt></tt></tt></tt></tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a>
OpenSIPS Bootcamp 2018
<a class="moz-txt-link-freetext" href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a>
</pre>
<div class="moz-cite-prefix">On 11/15/2018 05:01 PM, Ben Newlin
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:F1AA3509-48E6-4C40-957B-C1EEC531C91A@genesys.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \,serif ";
panose-1:2 7 3 9 2 2 5 2 4 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;}
tt
{mso-style-priority:99;
font-family:"Courier New",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle29
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle30
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">It’s
happening every few days, so it is pretty frequent. There
was another one yesterday but the DBG compile flags had been
temporarily removed for that one.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We have not
been able to determine a sequence to reproduce it yet.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben
Newlin </span>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Thursday, November 15, 2018 at 7:06 AM<br>
<b>To: </b>Ben Newlin <a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@genesys.com"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:devel@lists.opensips.org"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New",serif"><br>
<br>
<tt>How often this crash happens ? are you able to reproduce
it ?</tt><br>
<br>
<tt>The acc extra should work in the branch route, no
problem. Out of curiosity, I will try to reproduce you
case (timeout -> failure route -> t_relay ->
branch_route) to see if I can reproduce it.</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/13/2018 07:41 PM, Ben Newlin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Yes, we
are setting acc_extra variables in our branch routes,
which are sometimes (but not always) called from failure
route. Are acc_extra variables not available for use in
branch_routes?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We don’t
currently use drop_accounting anywhere in our script. If I
call it before that branch_route then it will stop
accounting for that call, right? We need to have
accounting records for the call, so I’m not sure how that
would resolve the issue?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Tuesday, November 13, 2018 at 9:13 AM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New",serif"><br>
<br>
<tt>Thanks for the info. The crash happens when you try to
set an acc extra variable in branch route (when a
creating a new branch via failure route, on timeout).</tt><br>
<br>
<tt>Now, do you use the drop accounting in your script ?
and considering the above scenario, it is possible to
have the drop acc before the branch route ?</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/12/2018 08:55 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We
upgraded to 2.4.3 and the crash reproduced today.
Backtrace is available here:
<a href="https://pastebin.com/CZxQnZdR"
moz-do-not-send="true">
https://pastebin.com/CZxQnZdR</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Wednesday, November 7, 2018 at 6:18 AM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a>, Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true">
<Ben.Newlin@genesys.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">The BT indicates
a double free for the accounting context - and I
noticed you use 2.4.1 version. And yes, there was an
issue related to acc context, issue that was fixed
starting 2.4.2. So, could you upgrade to the latest
2.4 and see if the crash still happens ? As I think
the fix is already there.</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif ""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/06/2018 11:13 PM, Bogdan-Andrei
Iancu wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Jackpot
- you get it right !! I will start digging into the
trace, but please keep the corefile, I might need it
later.</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">Thanks and
regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif ""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/06/2018 10:24 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
have reproduced this crash and verified this time
that the flags were set.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New \,serif "">$ opensips -V</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New \,serif "">version: opensips 2.4.1
(x86_64/linux)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New \,serif "">flags: STATS: On, DISABLE_NAGLE,
USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC,
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New \,serif "">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New \,serif "">poll method support: poll,
epoll, sigio_rt, select.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New \,serif "">git revision: 5d042cffc</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New \,serif "">main.c compiled on 23:38:55 Nov
5 2018 with gcc 7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Backtrace
is available here: <a
href="https://pastebin.com/KTQjkCwq"
moz-do-not-send="true">
https://pastebin.com/KTQjkCwq</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Bogdan-Andrei
Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Thursday, November 1, 2018 at 1:19
PM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">According to
the backtrace, the memory debugger was not
activated. Do an "opensips -V" to check the
resulting compile flags - do you see the
DBG_MALLOC and QM_MALLOC ?</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 10/31/2018 05:04 PM, Ben
Newlin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
was able to compile with those options and the
crash has occurred again. Backtrace is here:
<a href="https://pastebin.com/dezi9xUU"
moz-do-not-send="true">
https://pastebin.com/dezi9xUU</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Even
though I had `memdump=1` set in my script, there
was no extra memory debugging information in the
logs prior to or at the time of the crash. I’m not
sure if that is expected or not.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Bogdan-Andrei
Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Monday, October 29, 2018 at 8:11
AM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span
style="font-size:10.0pt">Hi Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">You can
change the compile flags via the Makefile.conf
file - the menuconfig is also updating that
file. So during your build you can simply push a
pre-modified Makefile.conf file with the options
needed for memory debugging. </span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 10/26/2018 05:14 PM, Ben
Newlin wrote:<o:p></o:p></p>
</div>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Unfortunately,
we have run into a similar issue before. Our
build system is completely automated and there
is no way to inject the `make menuconfig`
interactive step into that process. If I were to
be testing this locally I might be able to work
something out, but I could never get such a
build into our testing environment which is
where the crashes are occurring.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Do
you have instructions for enabling memory
debugging that do not require using the
interactive TUI tool? What does the menuconfig
program do when these options are selected? Are
there some defines or other settings we can
change ourselves and bypass menuconfig?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin
</span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Bogdan-Andrei
Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Friday, October 26, 2018 at 4:59
AM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a>, Ben
Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true">
<Ben.Newlin@genesys.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span
style="font-size:10.0pt">Hi Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New \,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">all the
BT's points to crashes while doing memory ops.
I suspect a memory corruption that randomly
triggers crashes in different parts of the
code.</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">Could
you try to re-compile with memory debugging
support ? See
<a
href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem"
moz-do-not-send="true">
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a>, the
"How to handle it" section.</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif ""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New \,serif ""><br>
<br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 10/24/2018 04:28 AM, Ben
Newlin wrote:<o:p></o:p></p>
</div>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span
style="font-size:11.0pt">We have had 2 more
crashes today.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">Crash 2: <a
href="https://pastebin.com/rMruBQcZ"
moz-do-not-send="true">
https://pastebin.com/rMruBQcZ</a></span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">This crash appears to
have occurred while processing an initial
INVITE request. I could not see anything
unusual about the request. I cannot tell if
this crash is related to the others.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">Crash 3: <a
href="https://pastebin.com/Gmk1m4NT"
moz-do-not-send="true">
https://pastebin.com/Gmk1m4NT</a></span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">This crash follows
the pattern of the original crash I reported.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben
Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="color:black">From: </span></b><span
style="color:black">Devel
<a
href="mailto:devel-bounces@lists.opensips.org"
moz-do-not-send="true"><devel-bounces@lists.opensips.org></a>
on behalf of Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Reply-To: </b>OpenSIPS devel mailling
list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Date: </b>Monday, October 22, 2018 at
4:45 PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel]
OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span
style="font-size:11.0pt">Here is a better
trace of the call:
<a href="https://pastebin.com/gWpQR8E7"
moz-do-not-send="true">
https://pastebin.com/gWpQR8E7</a></span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben
Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="color:black">From: </span></b><span
style="color:black">Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Date: </b>Monday, October 22, 2018 at
4:34 PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span
style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">We have been having
sporadic crashes and I was recently able to
recover a core dump for one. I have uploaded
it here:
<a href="https://pastebin.com/ABktcYcH"
moz-do-not-send="true">
https://pastebin.com/ABktcYcH</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">I picked out a
Call-ID from the crash data and took a look in
our tracing. I have uploaded it here:
<a href="https://pastebin.com/ZEzUUKZ5"
moz-do-not-send="true">
https://pastebin.com/ZEzUUKZ5</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">It appears that a
downstream server was extremely lagged and
failed to respond to an INVITE. We sent the
INVITE to another server and the call was
connected, but then eventually the original
server “caught up” and sent a burst of 200 OK
responses. The crash seems to have occurred
processing the ACK to one of these responses.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben
Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Devel mailing list<o:p></o:p></pre>
<pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Devel mailing list<o:p></o:p></pre>
<pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>