<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Courier;}
span.EmailStyle20
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">I have identified that these crashes are occurring when the far end system is not returning the Record-Route headers in the 200 OK response. The headers are present in the 180 response, but not the 200 OK.
I have reproduced the scenario using SIPp and captured a SIP trace: <a href="http://pastebin.com/ckKk3EhY">
http://pastebin.com/ckKk3EhY</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The crash occurs on receipt of the ACK request and attempt to match the dialog.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I also captured a BT for this scenario as well, in case anything specific in the trace made the issue easier to find:
<a href="http://pastebin.com/cM3FhPiw">http://pastebin.com/cM3FhPiw</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am working with the other system to try to fix their behavior.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ideally the Record-Route headers from previous replies could be used in this case to allow the call to succeed, but I don’t know if that is possible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Ben Newlin</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">"Newlin, Ben" <Ben.Newlin@inin.com><br>
<b>Date: </b>Wednesday, July 27, 2016 at 9:44 AM<br>
<b>To: </b>Bogdan-Andrei Iancu <bogdan@opensips.org>, OpenSIPS users mailling list <users@lists.opensips.org><br>
<b>Subject: </b>Re: [OpenSIPS-Users] OpenSIPS fix_route_dialog crashes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This is a different scenario than the other you responded to. As I said, we have two types of servers that work together. One is a load-balancer and runs as a proxy. It uses double Record-Route because it
sends messages between public and private networks. Then we have our other servers using TH which receive those requests. We are not using TH and RR on the same server (although I would like to).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If validate_dialog() and fix_route_dialog() (and possibly loose_route()) should not be called when using TH, I believe the documentation should reference that. It states that match_dialog() must be used with
TH, but does not indicate that the other functions should not be used or that the functionality won’t work. There is also no documentation of the incompatibility between RR and TH.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Either way, I ran a test where I removed all calls to loose_route(), validate_dialog(), and fix_route_dialog() from my script. The crash still occurred and the BT still pointed to fix_route_dialog() function.
So it must be getting called from within Dialog module somewhere. That BT is here:
<a href="http://pastebin.com/wu2X2Hxh">http://pastebin.com/wu2X2Hxh</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I collected this BT with loose_route() being called from my script, but not validate_dialog() or fix_route_dialog():
<a href="http://pastebin.com/6V7yPaHF">http://pastebin.com/6V7yPaHF</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This BT was collected with all three functions being called from my script:
<a href="http://pastebin.com/fZYYdndn">http://pastebin.com/fZYYdndn</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Ben Newlin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Bogdan-Andrei Iancu <bogdan@opensips.org><br>
<b>Date: </b>Wednesday, July 27, 2016 at 3:57 AM<br>
<b>To: </b>OpenSIPS users mailling list <users@lists.opensips.org>, "Newlin, Ben" <Ben.Newlin@inin.com><br>
<b>Subject: </b>Re: [OpenSIPS-Users] OpenSIPS fix_route_dialog crashes</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Ben,</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>First, if you use TH, makes no sense to do Record-Routing - there are 2 SIP concepts that overlaps. You either act as an end-point (by doing TH), either as a proxy (doing RR).</tt><br>
<br>
<tt>If doing TH, makes no sense to use validate + fix as these functions check and repair the routing information in the request (like Route and Contact headers). if you do TH, this routing info is actually hidden and added by OpenSIPS, so there is nothing
to fix and repair.</tt><br>
<br>
<tt>Nevertheless, this should not crash or corrupt OpenSIPS. HAve you managed to get a corefile ?</tt><br>
<br>
<tt>Also if you suspect memory corruption, you can compile-in the memory debugger - see
<a href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a> .</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre><a href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 26.07.2016 23:20, Newlin, Ben wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">I have had 3 OpenSIPS server crashes in the last week. All were due to segmentation faults. I was not able to capture core dumps; I am configuring that now to catch the next crash.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">My logs leading up to the crash are full of errors from fix_route_dialog() complaining about invalid URIs for sequential requests:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul 26 19:34:02 [220] ERROR:dialog:fix_route_dialog: Failed to parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul 26 19:34:02 [220] ERROR:core:parse_uri: bad uri, state 0 parsed: <ip:1> (4) / <ip:10.18.8.18:5060;ftag=gK0448f137;lr;r2=on>> (44)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul 26 19:11:19 [218] ERROR:dialog:fix_route_dialog: Failed to parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul 26 19:11:19 [218] ERROR:core:parse_uri: bad uri, state 0 parsed: <b0i2> (4) / <b0i2yjor;transport=udp<sip:10.18.8.17:5060;ftag=7207ce89;lr;r2=on> (65)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul 26 17:43:19 [220] ERROR:dialog:fix_route_dialog: Failed to parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul 26 17:43:19 [220] ERROR:core:parse_uri: bad uri, state 0 parsed: <ervi> (4) / <ervice_id6fdbc70f-2438-4726-807c-0d081df4d87> (44)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Many times the “URI” displayed in the error message is actually internal OpenSIPS variables, as in the last error above. When they are from the SIP message, I have verified that the messages themselves are
correctly formatted. This leads me to believe there is memory corruption occurring.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This all started when I updated my load-balancer servers to use Record-Routing, specifically the “double_rr” mechanism for when multiple interfaces exist. The Record-Routing is occurring on different servers
which have not crashed. Only the servers receiving the Record-Routed messages are experiencing the errors.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Here is a piece of the code processing sequential requests. I am using the topology_hiding() functionality of the Dialog module. Are validate_dialog() and fix_route_dialog() still valid in a topology_hiding
scenario?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">if (t_check_trans())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> setflag(SEQ_REQUEST);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> if (has_totag())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> loose_route();</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> if (match_dialog())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> if (!validate_dialog())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> fix_route_dialog();</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> if (is_method("BYE"))</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> setflag(ACC_FLAG);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> setflag(SEQ_REQUEST);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> else if (!isflagset(SEQ_REQUEST))</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> if (!is_method("ACK")) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> route(rlog, LV_ERROR, "check_sequential", "Sequential request not matched");</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> route(reply_error, "481", "Call Does Not Exist");</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> return(EXIT);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I will attempt to get core dumps of future crashes.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ben Newlin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><o:p></o:p></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>