<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi Ben,<br>
<br>
<tt>In any of the c<tt>ore<tt>file, in fr<tt>ame 2 could you print
<br>
<tt> p *((struct qm_frag*)((char*)(<tt>p</tt>)-sizeof(struct
qm_frag)))</tt><br>
<br>
<br>
<tt>Now, I <tt>see that the backtrace via </tt></tt>libcurl
& lib<tt>crypto <tt>may g<tt>o via <tt>some</tt> er<tt>ror
<tt>cleanup</tt><tt> </tt>- see frame #7 and #8</tt></tt></tt></tt><tt>
- so , do you see any curl error prior to the crash <tt>(ideally
from the sam<tt>e process) ?<br>
<br>
<tt>If you can <tt>reproduce <tt>it eas<tt>ily,
could you try <tt>with<tt>out the async
support, just as a PoC only.<br>
<br>
<tt><tt>Than<tt>ks,</tt></tt></tt><br>
</tt></tt></tt></tt></tt></tt></tt></tt></tt><br>
</tt></tt></tt></tt></tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
OpenSIPS Summit 2019
<a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2019Amsterdam/">https://www.opensips.org/events/Summit-2019Amsterdam/</a>
</pre>
<div class="moz-cite-prefix">On 06/25/2019 10:04 PM, Ben Newlin
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:36AD7E08-1279-4EE4-ABF0-ED8581606847@genesys.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;}
tt
{mso-style-priority:99;
font-family:"Courier New",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle29
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle30
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle31
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle32
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Yes, we are
using tls_mgm and proto_tls. We have the following version
of openssl-devel installed:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif"># yum info openssl-devel<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Name : openssl-devel<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Arch : x86_64<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Epoch : 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Version : 1.0.2k<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Release : 16.amzn2.1.1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Size : 3.1 M<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Repo : installed<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">From repo : amzn2-core<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Summary : Files for development of
applications which will use OpenSSL<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">URL : <a class="moz-txt-link-freetext" href="http://www.openssl.org/">http://www.openssl.org/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">License : OpenSSL<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif">Description : OpenSSL is a toolkit for
supporting cryptography. The openssl-devel<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif"> : package contains include
files needed to develop applications which<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Courier
New",serif"> : support various cryptographic
algorithms and protocols.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben
Newlin </span>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Tuesday, June 25, 2019 at 2:54 PM<br>
<b>To: </b>Ben Newlin <a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@genesys.com"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:devel@lists.opensips.org"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Thank
you Ben :)</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New",serif"><br>
<br>
<tt>The backtraces report a double free, but the mem ops are
performed via libcrypto ( opensips registers mem-specific
functions to lib crypto).</tt><br>
<br>
<tt>Are you using in your opensips cfg any other module that
might require ssl/crtypto, like tls_mgm, proto_wss ?</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit 2019<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 06/25/2019 06:49 PM, Ben Newlin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">You have
good timing. :) I was just collecting the data from a
crash this morning with the extra flags set.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"># opensips
-V</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">version:
opensips 2.4.5 (x86_64/linux)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">flags:
STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE
1024, BUF_SIZE 65535</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">poll
method support: poll, epoll, sigio_rt, select.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">git
revision: 60097425d</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">main.c
compiled on 22:27:12 Jun 24 2019 with gcc 7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[1] <a
href="https://pastebin.com/aj1mvk0N"
moz-do-not-send="true">
https://pastebin.com/aj1mvk0N</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2] <a
href="https://pastebin.com/kfSz9Ks0"
moz-do-not-send="true">
https://pastebin.com/kfSz9Ks0</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[3] <a
href="https://pastebin.com/hptH9yiA"
moz-do-not-send="true">
https://pastebin.com/hptH9yiA</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[4] <a
href="https://pastebin.com/GfYYZxzB"
moz-do-not-send="true">
https://pastebin.com/GfYYZxzB</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[5] <a
href="https://pastebin.com/CzYiNAch"
moz-do-not-send="true">
https://pastebin.com/CzYiNAch</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Tuesday, June 25, 2019 at 11:39 AM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New",serif"><br>
<br>
<tt>The backtrace + corefile should do it ;).</tt><br>
<br>
<tt>Best regards,</tt><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit 2019<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 06/20/2019 07:19 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Yes I
will enable those. This seems to be reproducing fairly
often so hopefully we will get some more info quickly.
Is there anything else I need to collect at time of the
crash with this enabled or is the core dump sufficient?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Thursday, June 20, 2019 at 12:15 PM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New",serif"><br>
<br>
<tt>This looks like very generic - a crash due a
previous memory corruption - is it possible for you to
run with QM_MALLOC + DBG_MALLOC ? otherwise it is
rather impossible to identify the original source of
corruption.</tt><br>
<br>
<tt>Thanks and regards,</tt><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit 2019<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 06/20/2019 05:34 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">We
have had another crash this morning:
</span><a href="https://pastebin.com/dmXYbhzC"
moz-do-not-send="true">https://pastebin.com/dmXYbhzC</a><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">OpenSIPS
version is the same as below.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Date: </b>Wednesday, June 19, 2019 at 2:38 PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a>,
Bogdan-Andrei Iancu <a
href="mailto:bogdan@opensips.org"
moz-do-not-send="true">
<bogdan@opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">The
Github issue has the version info for that, for all
the other crashes the version is below.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">$
opensips -V</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">version:
opensips 2.4.5 (x86_64/linux)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">flags:
STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">poll
method support: poll, epoll, sigio_rt, select.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">git
revision: 60097425d</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">main.c
compiled on 18:06:35 Jun 13 2019 with gcc 7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>