<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<tt>Ben<tt>,<br>
<br>
<tt>What <tt>OpenSIPS version <tt>is this (the crashing one) ?
1.11 or 2.1 ?<br>
<br>
<tt>Regards,</tt><br>
</tt></tt></tt></tt></tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a></pre>
<div class="moz-cite-prefix">On 27.07.2016 19:02, Newlin, Ben wrote:<br>
</div>
<blockquote cite="mid:204A9049-5F70-4462-B6E6-1E00F59D8088@inin.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Courier;}
span.EmailStyle20
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">I have
identified that these crashes are occurring when the far end
system is not returning the Record-Route headers in the 200
OK response. The headers are present in the 180 response,
but not the 200 OK. I have reproduced the scenario using
SIPp and captured a SIP trace: <a moz-do-not-send="true"
href="http://pastebin.com/ckKk3EhY">
http://pastebin.com/ckKk3EhY</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The crash
occurs on receipt of the ACK request and attempt to match
the dialog.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I also
captured a BT for this scenario as well, in case anything
specific in the trace made the issue easier to find:
<a moz-do-not-send="true"
href="http://pastebin.com/cM3FhPiw">http://pastebin.com/cM3FhPiw</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am working
with the other system to try to fix their behavior.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ideally the
Record-Route headers from previous replies could be used in
this case to allow the call to succeed, but I don’t know if
that is possible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Ben
Newlin</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">"Newlin, Ben"
<a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@inin.com"><Ben.Newlin@inin.com></a><br>
<b>Date: </b>Wednesday, July 27, 2016 at 9:44 AM<br>
<b>To: </b>Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>, OpenSIPS users mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] OpenSIPS
fix_route_dialog crashes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New
Roman","serif""><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This is
a different scenario than the other you responded to. As
I said, we have two types of servers that work together.
One is a load-balancer and runs as a proxy. It uses
double Record-Route because it sends messages between
public and private networks. Then we have our other
servers using TH which receive those requests. We are
not using TH and RR on the same server (although I would
like to).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If
validate_dialog() and fix_route_dialog() (and possibly
loose_route()) should not be called when using TH, I
believe the documentation should reference that. It
states that match_dialog() must be used with TH, but
does not indicate that the other functions should not be
used or that the functionality won’t work. There is also
no documentation of the incompatibility between RR and
TH.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Either
way, I ran a test where I removed all calls to
loose_route(), validate_dialog(), and fix_route_dialog()
from my script. The crash still occurred and the BT
still pointed to fix_route_dialog() function. So it must
be getting called from within Dialog module somewhere.
That BT is here:
<a moz-do-not-send="true"
href="http://pastebin.com/wu2X2Hxh">http://pastebin.com/wu2X2Hxh</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
collected this BT with loose_route() being called from
my script, but not validate_dialog() or
fix_route_dialog():
<a moz-do-not-send="true"
href="http://pastebin.com/6V7yPaHF">http://pastebin.com/6V7yPaHF</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This BT
was collected with all three functions being called from
my script:
<a moz-do-not-send="true"
href="http://pastebin.com/fZYYdndn">http://pastebin.com/fZYYdndn</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black">Ben Newlin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Wednesday, July 27, 2016 at 3:57 AM<br>
<b>To: </b>OpenSIPS users mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a>, "Newlin, Ben"
<a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@inin.com"><Ben.Newlin@inin.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] OpenSIPS
fix_route_dialog crashes</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times
New Roman","serif""> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New""><br>
<br>
<tt>First, if you use TH, makes no sense to do
Record-Routing - there are 2 SIP concepts that
overlaps. You either act as an end-point (by doing
TH), either as a proxy (doing RR).</tt><br>
<br>
<tt>If doing TH, makes no sense to use validate +
fix as these functions check and repair the
routing information in the request (like Route and
Contact headers). if you do TH, this routing info
is actually hidden and added by OpenSIPS, so there
is nothing to fix and repair.</tt><br>
<br>
<tt>Nevertheless, this should not crash or corrupt
OpenSIPS. HAve you managed to get a corefile ?</tt><br>
<br>
<tt>Also if you suspect memory corruption, you can
compile-in the memory debugger - see
<a moz-do-not-send="true"
href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a>
.</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 26.07.2016 23:20, Newlin, Ben
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">I
have had 3 OpenSIPS server crashes in the last
week. All were due to segmentation faults. I was
not able to capture core dumps; I am configuring
that now to catch the next crash.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">My
logs leading up to the crash are full of errors
from fix_route_dialog() complaining about invalid
URIs for sequential requests:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:34:02 [220] ERROR:dialog:fix_route_dialog:
Failed to parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:34:02 [220] ERROR:core:parse_uri: bad uri,
state 0 parsed: <ip:1> (4) /
<ip:10.18.8.18:5060;ftag=gK0448f137;lr;r2=on>>
(44)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-family:"Times New
Roman","serif""> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:11:19 [218] ERROR:dialog:fix_route_dialog:
Failed to parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:11:19 [218] ERROR:core:parse_uri: bad uri,
state 0 parsed: <b0i2> (4) /
<b0i2yjor;transport=udp<sip:10.18.8.17:5060;ftag=7207ce89;lr;r2=on>
(65)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 17:43:19 [220] ERROR:dialog:fix_route_dialog:
Failed to parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 17:43:19 [220] ERROR:core:parse_uri: bad uri,
state 0 parsed: <ervi> (4) /
<ervice_id6fdbc70f-2438-4726-807c-0d081df4d87>
(44)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Many
times the “URI” displayed in the error message is
actually internal OpenSIPS variables, as in the
last error above. When they are from the SIP
message, I have verified that the messages
themselves are correctly formatted. This leads me
to believe there is memory corruption occurring.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This
all started when I updated my load-balancer
servers to use Record-Routing, specifically the
“double_rr” mechanism for when multiple interfaces
exist. The Record-Routing is occurring on
different servers which have not crashed. Only the
servers receiving the Record-Routed messages are
experiencing the errors.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Here
is a piece of the code processing sequential
requests. I am using the topology_hiding()
functionality of the Dialog module. Are
validate_dialog() and fix_route_dialog() still
valid in a topology_hiding scenario?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">if
(t_check_trans())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
setflag(SEQ_REQUEST);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
if (has_totag())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
{</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
loose_route();</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
if (match_dialog())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
{</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
if (!validate_dialog())</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
fix_route_dialog();</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
if (is_method("BYE"))</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
setflag(ACC_FLAG);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
setflag(SEQ_REQUEST);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
else if (!isflagset(SEQ_REQUEST))</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
{</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
if (!is_method("ACK")) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
route(rlog, LV_ERROR, "check_sequential",
"Sequential request not matched");</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
route(reply_error, "481", "Call Does Not
Exist");</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
return(EXIT);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
will attempt to get core dumps of future crashes.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ben
Newlin</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-family:"Times New
Roman","serif""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Users mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span
style="font-family:"Times New
Roman","serif""> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>