<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi Ben,<br>
<br>
<tt>With some<tt> <tt>additional information that we <tt>got
from a different feed, we managed to rep<tt>roduce the <tt>cra<tt>sh<tt>
and Liviu is working <tt>on a fix as we spe<tt>ak.
I <tt>will keep yo<tt>u posted<tt>, but many
thanks for <tt>yo<tt>ur help in
investigating this issue.<br>
<br>
<tt>Best regards,</tt><br>
</tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a>
OpenSIPS Bootcamp 2018
<a class="moz-txt-link-freetext" href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a>
</pre>
<div class="moz-cite-prefix">On 11/13/2018 07:41 PM, Ben Newlin
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:7032F04A-043B-417A-90CB-B0C7D28BE19A@genesys.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \,serif";
panose-1:2 7 3 9 2 2 5 2 4 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle29
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Yes, we are
setting acc_extra variables in our branch routes, which are
sometimes (but not always) called from failure route. Are
acc_extra variables not available for use in branch_routes?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We don’t
currently use drop_accounting anywhere in our script. If I
call it before that branch_route then it will stop
accounting for that call, right? We need to have accounting
records for the call, so I’m not sure how that would resolve
the issue?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben
Newlin </span>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Tuesday, November 13, 2018 at 9:13 AM<br>
<b>To: </b>Ben Newlin <a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@genesys.com"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:devel@lists.opensips.org"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>Thanks for the info. The crash happens when you try to
set an acc extra variable in branch route (when a creating
a new branch via failure route, on timeout).</tt><br>
<br>
<tt>Now, do you use the drop accounting in your script ? and
considering the above scenario, it is possible to have the
drop acc before the branch route ?</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/12/2018 08:55 PM, Ben Newlin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We
upgraded to 2.4.3 and the crash reproduced today.
Backtrace is available here:
<a href="https://pastebin.com/CZxQnZdR"
moz-do-not-send="true">
https://pastebin.com/CZxQnZdR</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Wednesday, November 7, 2018 at 6:18 AM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a>, Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true">
<Ben.Newlin@genesys.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">The BT indicates a
double free for the accounting context - and I noticed
you use 2.4.1 version. And yes, there was an issue
related to acc context, issue that was fixed starting
2.4.2. So, could you upgrade to the latest 2.4 and see
if the crash still happens ? As I think the fix is
already there.</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/06/2018 11:13 PM, Bogdan-Andrei
Iancu wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Jackpot
- you get it right !! I will start digging into the
trace, but please keep the corefile, I might need it
later.</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">Thanks and
regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/06/2018 10:24 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have
reproduced this crash and verified this time that the
flags were set.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
\,serif"">$ opensips -V</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
\,serif"">version: opensips 2.4.1 (x86_64/linux)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
\,serif"">flags: STATS: On, DISABLE_NAGLE,
USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC,
DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
\,serif"">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
\,serif"">poll method support: poll, epoll,
sigio_rt, select.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
\,serif"">git revision: 5d042cffc</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier New
\,serif"">main.c compiled on 23:38:55 Nov 5 2018
with gcc 7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Backtrace
is available here: <a
href="https://pastebin.com/KTQjkCwq"
moz-do-not-send="true">
https://pastebin.com/KTQjkCwq</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Bogdan-Andrei
Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Thursday, November 1, 2018 at 1:19 PM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">According to
the backtrace, the memory debugger was not
activated. Do an "opensips -V" to check the
resulting compile flags - do you see the DBG_MALLOC
and QM_MALLOC ?</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
\,serif""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 10/31/2018 05:04 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
was able to compile with those options and the crash
has occurred again. Backtrace is here:
<a href="https://pastebin.com/dezi9xUU"
moz-do-not-send="true">
https://pastebin.com/dezi9xUU</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Even
though I had `memdump=1` set in my script, there was
no extra memory debugging information in the logs
prior to or at the time of the crash. I’m not sure
if that is expected or not.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Bogdan-Andrei
Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Monday, October 29, 2018 at 8:11 AM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">You can
change the compile flags via the Makefile.conf
file - the menuconfig is also updating that file.
So during your build you can simply push a
pre-modified Makefile.conf file with the options
needed for memory debugging. </span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif""><br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 10/26/2018 05:14 PM, Ben
Newlin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Unfortunately,
we have run into a similar issue before. Our build
system is completely automated and there is no way
to inject the `make menuconfig` interactive step
into that process. If I were to be testing this
locally I might be able to work something out, but
I could never get such a build into our testing
environment which is where the crashes are
occurring.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Do
you have instructions for enabling memory
debugging that do not require using the
interactive TUI tool? What does the menuconfig
program do when these options are selected? Are
there some defines or other settings we can change
ourselves and bypass menuconfig?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Bogdan-Andrei
Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Friday, October 26, 2018 at 4:59
AM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a>, Ben
Newlin <a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true">
<Ben.Newlin@genesys.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span
style="font-size:10.0pt">Hi Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">all the
BT's points to crashes while doing memory ops. I
suspect a memory corruption that randomly
triggers crashes in different parts of the code.</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">Could you
try to re-compile with memory debugging support
? See
<a
href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem"
moz-do-not-send="true">
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a>, the
"How to handle it" section.</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif""><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New \,serif""><br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 10/24/2018 04:28 AM, Ben
Newlin wrote:<o:p></o:p></p>
</div>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">We
have had 2 more crashes today.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Crash
2: <a href="https://pastebin.com/rMruBQcZ"
moz-do-not-send="true">
https://pastebin.com/rMruBQcZ</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This
crash appears to have occurred while processing
an initial INVITE request. I could not see
anything unusual about the request. I cannot
tell if this crash is related to the others.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Crash
3: <a href="https://pastebin.com/Gmk1m4NT"
moz-do-not-send="true">
https://pastebin.com/Gmk1m4NT</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This
crash follows the pattern of the original crash
I reported.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin
</span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Devel
<a
href="mailto:devel-bounces@lists.opensips.org"
moz-do-not-send="true"><devel-bounces@lists.opensips.org></a>
on behalf of Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Reply-To: </b>OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Date: </b>Monday, October 22, 2018 at 4:45
PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Here
is a better trace of the call:
<a href="https://pastebin.com/gWpQR8E7"
moz-do-not-send="true">
https://pastebin.com/gWpQR8E7</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin
</span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Ben
Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Date: </b>Monday, October 22, 2018 at 4:34
PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We
have been having sporadic crashes and I was
recently able to recover a core dump for one. I
have uploaded it here:
<a href="https://pastebin.com/ABktcYcH"
moz-do-not-send="true">
https://pastebin.com/ABktcYcH</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
picked out a Call-ID from the crash data and
took a look in our tracing. I have uploaded it
here:
<a href="https://pastebin.com/ZEzUUKZ5"
moz-do-not-send="true">
https://pastebin.com/ZEzUUKZ5</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">It
appears that a downstream server was extremely
lagged and failed to respond to an INVITE. We
sent the INVITE to another server and the call
was connected, but then eventually the original
server “caught up” and sent a burst of 200 OK
responses. The crash seems to have occurred
processing the ACK to one of these responses.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin
</span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Devel mailing list<o:p></o:p></pre>
<pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Devel mailing list<o:p></o:p></pre>
<pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>