<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi Be<tt>n,<br>
<br>
<tt>According to the BT,<tt> the cras<tt>h is in a pkg_malloc</tt></tt></tt>()
call:<br>
<tt> route = pkg_malloc(size);<br>
<tt><tt>Please double check <tt>t</tt>his with gdb <tt>i<tt>nfo.<br>
<br>
<tt><tt>I<tt>f<tt> so, this indicate <tt>a mem<tt>o<tt>ry
corruption<tt> and we have 2 options
here:<br>
<tt> - <tt>you comp<tt>il<tt>e
with memory debugger (see my
previous emails)<br>
<tt> - provide <tt>step-by-step
ind<tt>ication<tt>s on how
to rep<tt>roduce this
crash.<br>
</tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt></tt><br>
<tt>Thanks and Regards,</tt><br>
</tt></tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a></pre>
<div class="moz-cite-prefix">On 29.07.2016 15:54, Newlin, Ben wrote:<br>
</div>
<blockquote cite="mid:273E1DB2-DD7E-412E-B762-2E1826B2AC7D@inin.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New","serif";}
tt
{mso-style-priority:99;
font-family:"Courier New","serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Courier;}
span.EmailStyle20
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">This is
1.11.6, running on CentOS 7.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">Ben
Newlin</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black"><a class="moz-txt-link-rfc2396E" href="mailto:users-bounces@lists.opensips.org"><users-bounces@lists.opensips.org></a>
on behalf of Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Reply-To: </b>OpenSIPS users mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a><br>
<b>Date: </b>Friday, July 29, 2016 at 8:50 AM<br>
<b>To: </b>"Newlin, Ben" <a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@inin.com"><Ben.Newlin@inin.com></a>,
OpenSIPS users mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] OpenSIPS
fix_route_dialog crashes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New
Roman""><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New","serif""><br>
<br>
<tt>What OpenSIPS version is this (the crashing one) ?
1.11 or 2.1 ?</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 27.07.2016 19:02, Newlin, Ben
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">I have
identified that these crashes are occurring when the
far end system is not returning the Record-Route
headers in the 200 OK response. The headers are
present in the 180 response, but not the 200 OK. I
have reproduced the scenario using SIPp and captured a
SIP trace: <a moz-do-not-send="true"
href="http://pastebin.com/ckKk3EhY">
http://pastebin.com/ckKk3EhY</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The
crash occurs on receipt of the ACK request and attempt
to match the dialog.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I also
captured a BT for this scenario as well, in case
anything specific in the trace made the issue easier
to find:
<a moz-do-not-send="true"
href="http://pastebin.com/cM3FhPiw">http://pastebin.com/cM3FhPiw</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am
working with the other system to try to fix their
behavior.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ideally
the Record-Route headers from previous replies could
be used in this case to allow the call to succeed, but
I don’t know if that is possible.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black">Ben Newlin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">"Newlin, Ben"
<a moz-do-not-send="true"
href="mailto:Ben.Newlin@inin.com"><Ben.Newlin@inin.com></a><br>
<b>Date: </b>Wednesday, July 27, 2016 at 9:44 AM<br>
<b>To: </b>Bogdan-Andrei Iancu <a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>,
OpenSIPS users mailling list
<a moz-do-not-send="true"
href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] OpenSIPS
fix_route_dialog crashes</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-family:"Times New Roman""> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This
is a different scenario than the other you
responded to. As I said, we have two types of
servers that work together. One is a load-balancer
and runs as a proxy. It uses double Record-Route
because it sends messages between public and
private networks. Then we have our other servers
using TH which receive those requests. We are not
using TH and RR on the same server (although I
would like to).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If
validate_dialog() and fix_route_dialog() (and
possibly loose_route()) should not be called when
using TH, I believe the documentation should
reference that. It states that match_dialog() must
be used with TH, but does not indicate that the
other functions should not be used or that the
functionality won’t work. There is also no
documentation of the incompatibility between RR
and TH.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Either
way, I ran a test where I removed all calls to
loose_route(), validate_dialog(), and
fix_route_dialog() from my script. The crash still
occurred and the BT still pointed to
fix_route_dialog() function. So it must be getting
called from within Dialog module somewhere. That
BT is here:
<a moz-do-not-send="true"
href="http://pastebin.com/wu2X2Hxh">http://pastebin.com/wu2X2Hxh</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
collected this BT with loose_route() being called
from my script, but not validate_dialog() or
fix_route_dialog():
<a moz-do-not-send="true"
href="http://pastebin.com/6V7yPaHF">http://pastebin.com/6V7yPaHF</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This
BT was collected with all three functions being
called from my script:
<a moz-do-not-send="true"
href="http://pastebin.com/fZYYdndn">http://pastebin.com/fZYYdndn</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span
style="font-size:10.5pt;color:black">Ben Newlin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Bogdan-Andrei
Iancu
<a moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Wednesday, July 27, 2016 at 3:57
AM<br>
<b>To: </b>OpenSIPS users mailling list <a
moz-do-not-send="true"
href="mailto:users@lists.opensips.org">
<users@lists.opensips.org></a>, "Newlin,
Ben" <a moz-do-not-send="true"
href="mailto:Ben.Newlin@inin.com">
<Ben.Newlin@inin.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Users] OpenSIPS
fix_route_dialog crashes</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-family:"Times New Roman""> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><tt><span
style="font-size:10.0pt">Hi Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New","serif""><br>
<br>
<tt>First, if you use TH, makes no sense to do
Record-Routing - there are 2 SIP concepts
that overlaps. You either act as an
end-point (by doing TH), either as a proxy
(doing RR).</tt><br>
<br>
<tt>If doing TH, makes no sense to use
validate + fix as these functions check and
repair the routing information in the
request (like Route and Contact headers). if
you do TH, this routing info is actually
hidden and added by OpenSIPS, so there is
nothing to fix and repair.</tt><br>
<br>
<tt>Nevertheless, this should not crash or
corrupt OpenSIPS. HAve you managed to get a
corefile ?</tt><br>
<br>
<tt>Also if you suspect memory corruption, you
can compile-in the memory debugger - see
<a moz-do-not-send="true"
href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a>
.</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 26.07.2016 23:20,
Newlin, Ben wrote:<o:p></o:p></p>
</div>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span
style="font-size:11.0pt">I have had 3
OpenSIPS server crashes in the last week.
All were due to segmentation faults. I was
not able to capture core dumps; I am
configuring that now to catch the next
crash.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">My logs leading up
to the crash are full of errors from
fix_route_dialog() complaining about invalid
URIs for sequential requests:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:34:02 [220]
ERROR:dialog:fix_route_dialog: Failed to
parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:34:02 [220] ERROR:core:parse_uri: bad
uri, state 0 parsed: <ip:1> (4) /
<ip:10.18.8.18:5060;ftag=gK0448f137;lr;r2=on>>
(44)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-family:"Times New
Roman""> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:11:19 [218]
ERROR:dialog:fix_route_dialog: Failed to
parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 19:11:19 [218] ERROR:core:parse_uri: bad
uri, state 0 parsed: <b0i2> (4) /
<b0i2yjor;transport=udp<sip:10.18.8.17:5060;ftag=7207ce89;lr;r2=on>
(65)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 17:43:19 [220]
ERROR:dialog:fix_route_dialog: Failed to
parse SIP uri</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:8.5pt;font-family:Menlo;color:#2CA221;background:white">Jul
26 17:43:19 [220] ERROR:core:parse_uri: bad
uri, state 0 parsed: <ervi> (4) /
<ervice_id6fdbc70f-2438-4726-807c-0d081df4d87>
(44)</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">Many times the
“URI” displayed in the error message is
actually internal OpenSIPS variables, as in
the last error above. When they are from the
SIP message, I have verified that the
messages themselves are correctly formatted.
This leads me to believe there is memory
corruption occurring.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">This all started
when I updated my load-balancer servers to
use Record-Routing, specifically the
“double_rr” mechanism for when multiple
interfaces exist. The Record-Routing is
occurring on different servers which have
not crashed. Only the servers receiving the
Record-Routed messages are experiencing the
errors.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">Here is a piece of
the code processing sequential requests. I
am using the topology_hiding() functionality
of the Dialog module. Are validate_dialog()
and fix_route_dialog() still valid in a
topology_hiding scenario?</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">if
(t_check_trans())</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">
setflag(SEQ_REQUEST);</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> if (has_totag())</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> {</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> loose_route();</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> if
(match_dialog())</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> {</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> if
(!validate_dialog())</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">
fix_route_dialog();</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> if
(is_method("BYE"))</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">
setflag(ACC_FLAG);</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">
setflag(SEQ_REQUEST);</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> else if
(!isflagset(SEQ_REQUEST))</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> {</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> if
(!is_method("ACK")) {</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> route(rlog,
LV_ERROR, "check_sequential", "Sequential
request not matched");</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">
route(reply_error, "481", "Call Does Not
Exist");</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> return(EXIT);</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">I will attempt to
get core dumps of future crashes.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt">Ben Newlin</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-family:"Times New
Roman""><br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Users mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span
style="font-family:"Times New
Roman""> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Times
New Roman""><o:p> </o:p></span></p>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</body>
</html>