<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <tt>Hi Ben,<br>
      <br>
      <tt>OK, thanks for the conf<tt>irmation<tt>. Let me wor<tt>k out
              an add<tt>itional fix for this<tt> ;)<br>
                  <br>
                  <tt>Regards,</tt><br>
                </tt></tt></tt></tt></tt></tt></tt>
    <pre class="moz-signature" cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
OpenSIPS Summit 2019
  <a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2019Amsterdam/">https://www.opensips.org/events/Summit-2019Amsterdam/</a>
</pre>
    <div class="moz-cite-prefix">On 06/26/2019 08:55 PM, Ben Newlin
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:ABE9C9ED-36D6-41A7-8A53-72F5EE66BCAC@genesys.com">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <meta name="Generator" content="Microsoft Word 15 (filtered
        medium)">
      <style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Consolas;
        panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
        {font-family:"Courier New \,serif";
        panose-1:2 7 3 9 2 2 5 2 4 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:#0563C1;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:#954F72;
        text-decoration:underline;}
pre
        {mso-style-priority:99;
        mso-style-link:"HTML Preformatted Char";
        margin:0in;
        margin-bottom:.0001pt;
        font-size:10.0pt;
        font-family:"Courier New",serif;}
tt
        {mso-style-priority:99;
        font-family:"Courier New",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {mso-style-priority:34;
        margin-top:0in;
        margin-right:0in;
        margin-bottom:0in;
        margin-left:.5in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
        {mso-style-name:msonormal;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
        {mso-style-name:"HTML Preformatted Char";
        mso-style-priority:99;
        mso-style-link:"HTML Preformatted";
        font-family:Consolas;}
span.EmailStyle22
        {mso-style-type:personal;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
span.EmailStyle23
        {mso-style-type:personal;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
span.EmailStyle24
        {mso-style-type:personal;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
span.EmailStyle25
        {mso-style-type:personal;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
span.EmailStyle26
        {mso-style-type:personal;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
span.EmailStyle27
        {mso-style-type:personal;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
span.EmailStyle28
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
      <div class="WordSection1">
        <p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,<o:p></o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt">I was able
            to reproduce this crash on the newer 2.4 commit.<o:p></o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt">Backtrace: <a
              href="https://pastebin.com/RaMXTZA4"
              moz-do-not-send="true">
              https://pastebin.com/RaMXTZA4</a><o:p></o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif"># opensips -V<o:p></o:p></span></p>
        <p class="MsoNormal"><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif">version: opensips 2.4.6 (x86_64/linux)<o:p></o:p></span></p>
        <p class="MsoNormal"><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif">flags: STATS: On, DISABLE_NAGLE, USE_MCAST,
            SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC,
            FAST_LOCK-ADAPTIVE_WAIT<o:p></o:p></span></p>
        <p class="MsoNormal"><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif">ADAPTIVE_WAIT_LOOPS=1024,
            MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE
            1024, BUF_SIZE 65535<o:p></o:p></span></p>
        <p class="MsoNormal"><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif">poll method support: poll, epoll, sigio_rt,
            select.<o:p></o:p></span></p>
        <p class="MsoNormal"><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif">git revision: 714afb97f<o:p></o:p></span></p>
        <p class="MsoNormal"><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif">main.c compiled on 17:22:40 Jun 26 2019
            with gcc 7<o:p></o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben
            Newlin </span>
          <span style="font-size:11.0pt"><o:p></o:p></span></p>
        <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
        <div style="border:none;border-top:solid #B5C4DF
          1.0pt;padding:3.0pt 0in 0in 0in">
          <p class="MsoNormal"><b><span style="color:black">From: </span></b><span
              style="color:black">Bogdan-Andrei Iancu
              <a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
              <b>Date: </b>Wednesday, June 26, 2019 at 11:34 AM<br>
              <b>To: </b>Ben Newlin <a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@genesys.com"><Ben.Newlin@genesys.com></a>,
              OpenSIPS devel mailling list
              <a class="moz-txt-link-rfc2396E" href="mailto:devel@lists.opensips.org"><devel@lists.opensips.org></a><br>
              <b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash<o:p></o:p></span></p>
        </div>
        <div>
          <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
        </div>
        <p class="MsoNormal"><tt><span style="font-size:10.0pt">As per
              ticket, the issue you reported was already fixed, so going
              for the latest version in 2.4 branch should be fine with
              you.</span></tt><span
            style="font-size:10.0pt;font-family:"Courier
            New",serif"><br>
            <br>
            <tt>Regards,</tt><br>
            <br>
          </span><o:p></o:p></p>
        <pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
        <pre><o:p> </o:p></pre>
        <pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
        <pre>  <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
        <pre>OpenSIPS Summit 2019<o:p></o:p></pre>
        <pre>  <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
        <div>
          <p class="MsoNormal">On 06/26/2019 06:21 PM, Ben Newlin wrote:<o:p></o:p></p>
        </div>
        <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
          <p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">I will
              test both with the latest and with that specific commit,
              but I cannot use the latest in production due to the issue
              I reported in 2.4.6. [1] Perhaps that specific commit is
              before whatever change is causing that issue.</span><o:p></o:p></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
          <p class="MsoNormal"><span style="font-size:11.0pt">[1] <a
                href="https://github.com/OpenSIPS/opensips/issues/1736"
                moz-do-not-send="true">
                https://github.com/OpenSIPS/opensips/issues/1736</a></span><o:p></o:p></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
          <p class="MsoNormal"><span
              style="font-size:11.0pt;color:black">Ben Newlin </span>
            <o:p></o:p></p>
          <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
          <div style="border:none;border-top:solid #B5C4DF
            1.0pt;padding:3.0pt 0in 0in 0in">
            <p class="MsoNormal"><b><span style="color:black">From: </span></b><span
                style="color:black">Bogdan-Andrei Iancu
                <a href="mailto:bogdan@opensips.org"
                  moz-do-not-send="true"><bogdan@opensips.org></a><br>
                <b>Date: </b>Wednesday, June 26, 2019 at 10:56 AM<br>
                <b>To: </b>OpenSIPS devel mailling list <a
                  href="mailto:devel@lists.opensips.org"
                  moz-do-not-send="true">
                  <devel@lists.opensips.org></a>, Ben Newlin <a
                  href="mailto:Ben.Newlin@genesys.com"
                  moz-do-not-send="true">
                  <Ben.Newlin@genesys.com></a><br>
                <b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
          </div>
          <div>
            <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
          </div>
          <p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
                Ben,</span></tt><span
              style="font-size:10.0pt;font-family:"Courier New
              ,serif",serif"><br>
              <br>
            </span><tt><span style="font-size:10.0pt">While re-checking
                some of some data related to your report, I noticed that
                your opensips version (revision
              </span></tt><tt><span style="font-size:11.0pt">d025b4f61)
                is a b</span></tt><tt><span style="font-size:10.0pt">it
                old and it is missing some fixes that might be relevant
                for you (mainly
                3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).</span></tt><span
              style="font-size:10.0pt;font-family:"Courier New
              ,serif",serif"><br>
              <br>
            </span><tt><span style="font-size:10.0pt">Could you grab the
                latest 2.4 from git and give it a try please.</span></tt><span
              style="font-size:10.0pt;font-family:"Courier New
              ,serif",serif"><br>
              <br>
            </span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
              style="font-size:10.0pt;font-family:"Courier New
              ,serif",serif"><br>
              <br>
              <br>
            </span><o:p></o:p></p>
          <pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
          <pre> <o:p></o:p></pre>
          <pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
          <pre>  <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
          <pre>OpenSIPS Summit 2019<o:p></o:p></pre>
          <pre>  <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
          <div>
            <p class="MsoNormal">On 06/26/2019 02:37 PM, Bogdan-Andrei
              Iancu wrote:<o:p></o:p></p>
          </div>
          <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
            <p class="MsoNormal"><tt><span style="font-size:10.0pt">Thank
                  you Ben,</span></tt><span
                style="font-size:10.0pt;font-family:"Courier New
                ,serif",serif"><br>
                <br>
              </span><tt><span style="font-size:10.0pt">This backtrace
                  confirms the initial suspicion - I'm trying to find
                  out a way to get it fixed in a more generic way; what
                  you experience is just one face of the problem (with
                  many faces :P) and there are other related reports.</span></tt><span
                style="font-size:10.0pt;font-family:"Courier New
                ,serif",serif"><br>
                <br>
              </span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
                style="font-size:10.0pt;font-family:"Courier New
                ,serif",serif"><br>
                <br>
                <br>
              </span><o:p></o:p></p>
            <pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
            <pre> <o:p></o:p></pre>
            <pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
            <pre>  <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
            <pre>OpenSIPS Summit 2019<o:p></o:p></pre>
            <pre>  <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
            <div>
              <p class="MsoNormal">On 06/26/2019 02:34 AM, Ben Newlin
                wrote:<o:p></o:p></p>
            </div>
            <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
              <p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
              <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
              <p class="MsoNormal"><span style="font-size:11.0pt">I have
                  now been able to reproduce this crash reliably, I
                  believe. Please see the backtrace [1]. Is there some
                  other information you would like me to collect when
                  this occurs?</span><o:p></o:p></p>
              <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
              <p class="MsoNormal"><span style="font-size:11.0pt">[1] <a
                    href="https://pastebin.com/n0Ph8XH5"
                    moz-do-not-send="true">
                    https://pastebin.com/n0Ph8XH5</a></span><o:p></o:p></p>
              <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
              <p class="MsoNormal"><span
                  style="font-size:11.0pt;color:black">Ben Newlin </span>
                <o:p></o:p></p>
              <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
              <div style="border:none;border-top:solid #B5C4DF
                1.0pt;padding:3.0pt 0in 0in 0in">
                <p class="MsoNormal"><b><span style="color:black">From:
                    </span></b><span style="color:black">Bogdan-Andrei
                    Iancu
                    <a href="mailto:bogdan@opensips.org"
                      moz-do-not-send="true"><bogdan@opensips.org></a><br>
                    <b>Date: </b>Friday, June 7, 2019 at 9:15 AM<br>
                    <b>To: </b>Ben Newlin <a
                      href="mailto:Ben.Newlin@genesys.com"
                      moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
                    OpenSIPS devel mailling list
                    <a href="mailto:devel@lists.opensips.org"
                      moz-do-not-send="true"><devel@lists.opensips.org></a><br>
                    <b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
              </div>
              <div>
                <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
              </div>
              <p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
                    Ben,</span></tt><span
                  style="font-size:10.0pt;font-family:"Courier
                  New",serif"><br>
                  <br>
                  <tt>How often/easy is to reproduce this crash (if
                    possible) ? Brainstorming with Razvan, we suspect a
                    race (on the msg save in shmem in transaction)
                    between the process doing the cleanup after the
                    async resume and the process running the failure
                    route (due th 503).</tt><br>
                  <br>
                  <tt>But this is just a supposition, eventually you can
                    validate it or not by removing the async ??</tt><br>
                  <br>
                  <tt>And on the double ACK - I'm not 100% it is a
                    actually a double one, as the second has a smaller
                    MF value (69, versus the 70 on the first ACK).
                  </tt><br>
                  <br>
                  <tt>Regards,</tt><br>
                  <br>
                  <br>
                  <br>
                </span><o:p></o:p></p>
              <pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
              <pre> <o:p></o:p></pre>
              <pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
              <pre>  <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
              <pre>OpenSIPS Summit 2019<o:p></o:p></pre>
              <pre>  <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
              <div>
                <p class="MsoNormal">On 06/07/2019 03:52 PM, Ben Newlin
                  wrote:<o:p></o:p></p>
              </div>
              <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
                <p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt">Sorry,
                    I should have thought to actually look at the trace
                    and examine this call.</span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt">1)
                    Yes</span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt">2)
                    The Called Party is 10.32.20.60, which is another
                    OpenSIPS instance. The crashed instance received the
                    "503 Service Unavailable" approximately 8-10 ms
                    after sending the INVITE.</span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt">There
                    is a SIP trace of the exchange here:
                    <a href="https://pastebin.com/6bttsSVD"
                      moz-do-not-send="true">
                      https://pastebin.com/6bttsSVD</a>.</span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt">One
                    oddity I saw is that the crashed process appears to
                    send (or at least siptrace) the ACK twice.</span><o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                <p class="MsoNormal"><span
                    style="font-size:11.0pt;color:black">Ben Newlin </span>
                  <o:p></o:p></p>
                <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                <div style="border:none;border-top:solid #B5C4DF
                  1.0pt;padding:3.0pt 0in 0in 0in">
                  <p class="MsoNormal"><b><span style="color:black">From:
                      </span></b><span style="color:black">Bogdan-Andrei
                      Iancu
                      <a href="mailto:bogdan@opensips.org"
                        moz-do-not-send="true"><bogdan@opensips.org></a><br>
                      <b>Date: </b>Thursday, June 6, 2019 at 11:42 AM<br>
                      <b>To: </b>OpenSIPS devel mailling list <a
                        href="mailto:devel@lists.opensips.org"
                        moz-do-not-send="true">
                        <devel@lists.opensips.org></a>, Ben Newlin
                      <a href="mailto:Ben.Newlin@genesys.com"
                        moz-do-not-send="true">
                        <Ben.Newlin@genesys.com></a><br>
                      <b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
                      Crash</span><o:p></o:p></p>
                </div>
                <div>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                </div>
                <p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
                      Ben,</span></tt><span
                    style="font-size:10.0pt;font-family:"Courier
                    New",serif"><br>
                    <br>
                    <tt>Thanks for "another" report :).</tt><br>
                    <br>
                    <tt>Questions:</tt><br>
                    <tt>1) do you do any async for the INVITE in this
                      crash ?</tt><br>
                    <tt>2) if it is an YES to (1), is the caller party
                      generating the "503 Service Unavailable" (which
                      triggers the crash) - 10.32.20.60 ?? - a really
                      close (from net delay perspective) and fast to
                      answer party ?</tt><br>
                    <br>
                    <tt>Regards,</tt><br>
                    <br>
                    <br>
                    <br>
                    <br>
                  </span><o:p></o:p></p>
                <pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
                <pre> <o:p></o:p></pre>
                <pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
                <pre>  <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
                <pre>OpenSIPS Summit 2019<o:p></o:p></pre>
                <pre>  <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
                <div>
                  <p class="MsoNormal">On 06/05/2019 10:02 PM, Ben
                    Newlin wrote:<o:p></o:p></p>
                </div>
                <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
                  <p class="MsoNormal"><span style="font-size:11.0pt">We
                      have had another crash today.</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">Backtrace
                      is here: <a href="https://pastebin.com/q4RQC7kS"
                        moz-do-not-send="true">
                        https://pastebin.com/q4RQC7kS</a></span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">I
                      found this in the log at the time of the crash:</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">Jun 
                      5 17:54:10 [4978] CRITICAL:core:sig_usr: segfault
                      in process pid: 4978, id: 8</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">Please
                      let me know if any further information can be
                      useful.</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span
                      style="font-size:11.0pt;color:black">Ben Newlin </span>
                    <o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <div style="border:none;border-top:solid #B5C4DF
                    1.0pt;padding:3.0pt 0in 0in 0in">
                    <p class="MsoNormal"><b><span style="color:black">From:
                        </span></b><span style="color:black">Devel
                        <a
                          href="mailto:devel-bounces@lists.opensips.org"
                          moz-do-not-send="true"><devel-bounces@lists.opensips.org></a>
                        on behalf of Ben Newlin
                        <a href="mailto:Ben.Newlin@genesys.com"
                          moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
                        <b>Reply-To: </b>OpenSIPS devel mailling list <a
                          href="mailto:devel@lists.opensips.org"
                          moz-do-not-send="true">
                          <devel@lists.opensips.org></a><br>
                        <b>Date: </b>Friday, May 10, 2019 at 6:31 PM<br>
                        <b>To: </b>OpenSIPS devel mailling list <a
                          href="mailto:devel@lists.opensips.org"
                          moz-do-not-send="true">
                          <devel@lists.opensips.org></a><br>
                        <b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS
                        Crash</span><o:p></o:p></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  </div>
                  <p class="MsoNormal"><span style="font-size:11.0pt">I
                      found this in the log at the time of the crash:</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">kernel:
                      opensips[5003]: segfault at 30 ip 00007fbd4c8f59d0
                      sp 00007ffcaa850c80 error 6 in
                      tm.so[7fbd4c887000+8e000]</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span
                      style="font-size:11.0pt;color:black">Ben Newlin </span>
                    <o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <div style="border:none;border-top:solid #B5C4DF
                    1.0pt;padding:3.0pt 0in 0in 0in">
                    <p class="MsoNormal"><b><span style="color:black">From:
                        </span></b><span style="color:black">Devel
                        <a
                          href="mailto:devel-bounces@lists.opensips.org"
                          moz-do-not-send="true"><devel-bounces@lists.opensips.org></a>
                        on behalf of Ben Newlin
                        <a href="mailto:Ben.Newlin@genesys.com"
                          moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
                        <b>Reply-To: </b>OpenSIPS devel mailling list <a
                          href="mailto:devel@lists.opensips.org"
                          moz-do-not-send="true">
                          <devel@lists.opensips.org></a><br>
                        <b>Date: </b>Friday, May 10, 2019 at 5:44 PM<br>
                        <b>To: </b>OpenSIPS devel mailling list <a
                          href="mailto:devel@lists.opensips.org"
                          moz-do-not-send="true">
                          <devel@lists.opensips.org></a><br>
                        <b>Subject: </b>[OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  </div>
                  <p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">We
                      had a crash today of our OpenSIPS instance.</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">Backtrace
                      is here: </span><a
                      href="https://pastebin.com/QbRJimwx"
                      moz-do-not-send="true">https://pastebin.com/QbRJimwx</a><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">#
                      opensips -V</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">version:
                      opensips 2.4.5 (x86_64/linux)</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">flags:
                      STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
                      PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">ADAPTIVE_WAIT_LOOPS=1024,
                      MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
                      MAX_URI_SIZE 1024, BUF_SIZE 65535</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">poll
                      method support: poll, epoll, sigio_rt, select.</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">git
                      revision: d025b4f61</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt">main.c
                      compiled on 20:58:31 May  9 2019 with gcc 7</span><o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
                  <p class="MsoNormal"><span
                      style="font-size:11.0pt;color:black">Ben Newlin </span>
                    <o:p></o:p></p>
                  <p class="MsoNormal"><span style="font-size:11.0pt"><br>
                      <br>
                      <br>
                      <br>
                      <br>
                      <br>
                    </span><o:p></o:p></p>
                  <pre>_______________________________________________<o:p></o:p></pre>
                  <pre>Devel mailing list<o:p></o:p></pre>
                  <pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
                  <pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
                </blockquote>
                <p class="MsoNormal"><span style="font-size:11.0pt"><br>
                    <br>
                    <br>
                    <br>
                    <br>
                  </span><o:p></o:p></p>
              </blockquote>
              <p class="MsoNormal"><span style="font-size:11.0pt"><br>
                  <br>
                  <br>
                  <br>
                </span><o:p></o:p></p>
            </blockquote>
            <p class="MsoNormal"><span style="font-size:11.0pt"><br>
                <br>
                <br>
                <br>
                <br>
              </span><o:p></o:p></p>
            <pre>_______________________________________________<o:p></o:p></pre>
            <pre>Devel mailing list<o:p></o:p></pre>
            <pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
            <pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
          </blockquote>
          <p class="MsoNormal"><span style="font-size:11.0pt"><br>
              <br>
              <br>
            </span><o:p></o:p></p>
        </blockquote>
        <p class="MsoNormal"><span style="font-size:11.0pt"><br>
            <br>
            <o:p></o:p></span></p>
      </div>
    </blockquote>
    <br>
  </body>
</html>