<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi <tt>Ben,<br>
<br>
You can c<tt>hange the comp<tt>ile flags via <tt>the Make<tt>file.conf
file - the menuconfig is also updating th<tt>at file. So
d<tt>u<tt>ring your build you can simpl<tt>y push a <tt>p<tt>re</tt>-modified
</tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt><tt><tt><tt><tt><tt><tt><tt><tt><tt><tt><tt><tt><tt><tt><tt><tt>Make<tt>file.conf
file with<tt> the options needed
for <tt>memory debugging.</tt></tt></tt></tt></tt></tt></tt></tt></tt><tt><tt><tt>
</tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt><br>
<br>
<tt><tt>Regards,</tt></tt><br>
</tt></tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a>
OpenSIPS Bootcamp 2018
<a class="moz-txt-link-freetext" href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a>
</pre>
<div class="moz-cite-prefix">On 10/26/2018 05:14 PM, Ben Newlin
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:78E544C4-D735-4EE2-B4C8-7A49ECA15F84@genesys.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;}
tt
{mso-style-priority:99;
font-family:"Courier New",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Unfortunately,
we have run into a similar issue before. Our build system is
completely automated and there is no way to inject the `make
menuconfig` interactive step into that process. If I were to
be testing this locally I might be able to work something
out, but I could never get such a build into our testing
environment which is where the crashes are occurring.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Do you have
instructions for enabling memory debugging that do not
require using the interactive TUI tool? What does the
menuconfig program do when these options are selected? Are
there some defines or other settings we can change ourselves
and bypass menuconfig?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben
Newlin </span>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Friday, October 26, 2018 at 4:59 AM<br>
<b>To: </b>OpenSIPS devel mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:devel@lists.opensips.org"><devel@lists.opensips.org></a>, Ben Newlin
<a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@genesys.com"><Ben.Newlin@genesys.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New",serif"><br>
<br>
<tt>all the BT's points to crashes while doing memory ops. I
suspect a memory corruption that randomly triggers crashes
in different parts of the code.</tt><br>
<br>
<tt>Could you try to re-compile with memory debugging
support ? See <a
href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem"
moz-do-not-send="true">
http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a>, the
"How to handle it" section.</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="http://www.opensips-solutions.com" moz-do-not-send="true">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Bootcamp 2018<o:p></o:p></pre>
<pre> <a href="http://opensips.org/training/OpenSIPS_Bootcamp_2018/" moz-do-not-send="true">http://opensips.org/training/OpenSIPS_Bootcamp_2018/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 10/24/2018 04:28 AM, Ben Newlin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">We have
had 2 more crashes today.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Crash 2: <a
href="https://pastebin.com/rMruBQcZ"
moz-do-not-send="true">
https://pastebin.com/rMruBQcZ</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This crash
appears to have occurred while processing an initial
INVITE request. I could not see anything unusual about the
request. I cannot tell if this crash is related to the
others.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Crash 3: <a
href="https://pastebin.com/Gmk1m4NT"
moz-do-not-send="true">
https://pastebin.com/Gmk1m4NT</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This crash
follows the pattern of the original crash I reported.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Devel
<a href="mailto:devel-bounces@lists.opensips.org"
moz-do-not-send="true"><devel-bounces@lists.opensips.org></a>
on behalf of Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Reply-To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Date: </b>Monday, October 22, 2018 at 4:45 PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Here is a
better trace of the call:
<a href="https://pastebin.com/gWpQR8E7"
moz-do-not-send="true">
https://pastebin.com/gWpQR8E7</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Date: </b>Monday, October 22, 2018 at 4:34 PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We have
been having sporadic crashes and I was recently able to
recover a core dump for one. I have uploaded it here:
<a href="https://pastebin.com/ABktcYcH"
moz-do-not-send="true">
https://pastebin.com/ABktcYcH</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I picked
out a Call-ID from the crash data and took a look in our
tracing. I have uploaded it here:
<a href="https://pastebin.com/ZEzUUKZ5"
moz-do-not-send="true">
https://pastebin.com/ZEzUUKZ5</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">It appears
that a downstream server was extremely lagged and failed
to respond to an INVITE. We sent the INVITE to another
server and the call was connected, but then eventually the
original server “caught up” and sent a burst of 200 OK
responses. The crash seems to have occurred processing the
ACK to one of these responses.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Devel mailing list<o:p></o:p></pre>
<pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>