<div dir="ltr">Hi Bogdan, yes, that seems to be fixed it. Please merge into 2.3 at your earliest convenience. Thanks!<div><br></div><div>-Max</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 21, 2017 at 6:29 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<tt>Hi MAxim,<br>
<br>
Could you validate this fix:<br>
<a class="m_2422082274593172953moz-txt-link-freetext" href="https://github.com/OpenSIPS/opensips/commit/4aa5fb91a0a8b06d886ef3f2a093c8f5526aaa4a" target="_blank">https://github.com/OpenSIPS/<wbr>opensips/commit/<wbr>4aa5fb91a0a8b06d886ef3f2a093c8<wbr>f5526aaa4a</a><br>
<br>
Regards,<br>
</tt><span class="">
<pre class="m_2422082274593172953moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="m_2422082274593172953moz-txt-link-freetext" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.<wbr>com</a>
OpenSIPS Bootcamp 2017, Houston, US
<a class="m_2422082274593172953moz-txt-link-freetext" href="http://opensips.org/training/OpenSIPS_Bootcamp_2017.html" target="_blank">http://opensips.org/training/<wbr>OpenSIPS_Bootcamp_2017.html</a>
</pre>
</span><div><div class="h5"><div class="m_2422082274593172953moz-cite-prefix">On 08/21/2017 01:16 PM, Bogdan-Andrei
Iancu wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<tt>Hi Maxim,<br>
<br>
Sorry for the delay, finally back on tracks after ClueCon,<br>
<br>
And thank you for your latest data (especially for the "trap"
output) - I have a good lead on what might be the problem.
Expect the fix soon.<br>
<br>
Best regards,<br>
</tt>
<pre class="m_2422082274593172953moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="m_2422082274593172953moz-txt-link-freetext" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.<wbr>com</a>
OpenSIPS Bootcamp 2017, Houston, US
<a class="m_2422082274593172953moz-txt-link-freetext" href="http://opensips.org/training/OpenSIPS_Bootcamp_2017.html" target="_blank">http://opensips.org/training/<wbr>OpenSIPS_Bootcamp_2017.html</a>
</pre>
<div class="m_2422082274593172953moz-cite-prefix">On 08/19/2017 01:39 AM, Maxim Sobolev
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Bogdan, any luck looking into this issue?
Looks like a serious problem with 2.3 / master to me. Backing
out the commit in question fixes it for us.
<div><br>
</div>
<div>-Max</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 7, 2017 at 12:09 PM,
Maxim Sobolev <span dir="ltr"><<a href="mailto:sobomax@sippysoft.com" target="_blank">sobomax@sippysoft.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Bogdan, I've collected some "opensipsctl
trap" output here for you, see line 2611 onwards:
<div><br>
</div>
<div><a href="https://travis-ci.org/sippy/voiptests/jobs/261951017" target="_blank">https://travis-ci.org/sippy/vo<wbr>iptests/jobs/261951017</a><br>
</div>
<div><br>
</div>
<div>This is 10 seconds after starting the test,
CANCELs arrive at about 2-3s mark, so it's well into
"dead zone". Yet, I see some CANCEL messages in the
traces, so it looks like some locking issue indeed.</div>
<div><br>
</div>
<div>Please let me know if it helps or if you need any
more debug. For the time being, I am successfully
running with 2.3 and master by reverting change 1eb4ec0f78f43f6ff546de4<wbr>9bc72e513876fb86b</div>
<div><br>
</div>
<div>-Max</div>
</div>
<div class="gmail_extra">
<div>
<div class="m_2422082274593172953h5"><br>
<div class="gmail_quote">On Wed, Aug 2, 2017 at
8:03 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <tt>Thank
you Maxim for your troubleshooting.<br>
<br>
If you say this code makes a difference in
your test, I have the say that the newly
added locking may do the "blocking" . To
see if this is true, while OpenSIPS is
"blocked" could you run several times the
"opensipsctl trap" ?<br>
<br>
Best regards,<br>
</tt><span>
<pre class="m_2422082274593172953m_7405539153921155859m_278828822372435955moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="m_2422082274593172953m_7405539153921155859m_278828822372435955moz-txt-link-freetext" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.<wbr>com</a>
OpenSIPS Bootcamp 2017, Houston, US
<a class="m_2422082274593172953m_7405539153921155859m_278828822372435955moz-txt-link-freetext" href="http://opensips.org/training/OpenSIPS_Bootcamp_2017.html" target="_blank">http://opensips.org/training/O<wbr>penSIPS_Bootcamp_2017.html</a>
</pre>
</span>
<div>
<div class="m_2422082274593172953m_7405539153921155859h5">
<div class="m_2422082274593172953m_7405539153921155859m_278828822372435955moz-cite-prefix">On
08/02/2017 03:59 AM, Maxim Sobolev
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Bogdan & Co, I
think I've narrowed down failure to
a specific revision in the 2.3
branch, see <a href="https://travis-ci.org/sippy/voiptests/builds/260009545" target="_blank">https://travis-ci.org/sipp<wbr>y/voiptests/builds/260009545</a>:
<div><br>
</div>
<div>
<div>commit
1eb4ec0f78f43f6ff546de49bc72e5<wbr>13876fb86b</div>
<div>Author: Bogdan-Andrei Iancu
<<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></div>
<div>Date: Thu Mar 9 15:37:01
2017 +0200</div>
<div><br>
</div>
<div> Added support for dynamic
branch manipulation</div>
<div><br>
</div>
<div> New functions were added
to allow remote injecting of new
branches into an ongoing
transaction:</div>
<div> *
t_inject_branches(source, flags)
- injects new branches;</div>
<div> *
t_wait_for_new_branches() -
instruct transaction to wait
more for possible new
injections.</div>
</div>
<div><br>
</div>
<div>Looking at the diff in
question, the following piece of
code looks suspicious, but it
might be something else from that
change:<br>
</div>
<div><br>
</div>
<div>
<div>@@ -1244,7 +1301,18 @@ inline
static int w_t_relay( struct
sip_msg *p_msg , char *proxy,
char *flags)</div>
<div><br>
</div>
<div>
update_cloned_msg_from_msg(
t->uas.request, p_msg);</div>
<div><br>
</div>
<div>- ret =
t_forward_nonack( t, p_msg, p);</div>
<div>+ if
(route_type==FAILURE_ROUTE) {</div>
<div>+ /* If
called from failure route we
need reset the branch counter to</div>
<div>+ *
ignore the previous set of
branches (already terminated) */</div>
<div>+ ret =
t_forward_nonack( t, p_msg, p,
1/*reset*/);</div>
<div>+ } else {</div>
<div>+ /* if
called from request route and
the transaction was previously</div>
<div>+ *
created, better lock here to
avoid any overlapping with</div>
<div>+ *
branch injection from other
processes */</div>
<div>+
LOCK_REPLIES(t);</div>
<div>+ ret =
t_forward_nonack( t, p_msg, p,
1/*reset*/);</div>
<div>+
UNLOCK_REPLIES(t);</div>
<div>+ }</div>
<div> if (ret<=0
) {</div>
<div>
LM_ERR("t_forward_nonack
failed\n");</div>
<div> ret =
t_relay_inerr2scripterr();</div>
<div>@@ -1262,7 +1330,8 @@
route_err:</div>
</div>
<div><br>
</div>
<div>The revisions right before that
work, none after does. Hopefully
it will give you some clues. Let
me know if any other info is
needed.</div>
<div><br>
</div>
<div>-Max</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
</div>
</div>
<span>-- <br>
<div class="m_2422082274593172953m_7405539153921155859gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Maksym Sobolyev<br>
Sippy Software, Inc.<br>
Internet Telephony (VoIP) Experts<br>
Tel (Canada): <a href="tel:%28778%29%20783-0474" value="+17787830474" target="_blank">+1-778-783-0474</a><br>
Tel (Toll-Free): <a href="tel:%28855%29%20747-7779" value="+18557477779" target="_blank">+1-855-747-7779</a><br>
Fax: <a href="tel:%28866%29%20857-6942" value="+18668576942" target="_blank">+1-866-857-6942</a><br>
Web: <a href="http://www.sippysoft.com" target="_blank">http://www.sippysoft.com</a><br>
MSN: <a href="mailto:sales@sippysoft.com" target="_blank">sales@sippysoft.com</a><br>
Skype: SippySoft<br>
</div>
</div>
</span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_2422082274593172953gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Maksym Sobolyev<br>
Sippy Software, Inc.<br>
Internet Telephony (VoIP) Experts<br>
Tel (Canada): <a href="tel:(778)%20783-0474" value="+17787830474" target="_blank">+1-778-783-0474</a><br>
Tel (Toll-Free): <a href="tel:(855)%20747-7779" value="+18557477779" target="_blank">+1-855-747-7779</a><br>
Fax: <a href="tel:(866)%20857-6942" value="+18668576942" target="_blank">+1-866-857-6942</a><br>
Web: <a href="http://www.sippysoft.com" target="_blank">http://www.sippysoft.com</a><br>
MSN: <a href="mailto:sales@sippysoft.com" target="_blank">sales@sippysoft.com</a><br>
Skype: SippySoft<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<br>
<fieldset class="m_2422082274593172953mimeAttachmentHeader"></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
Devel mailing list
<a class="m_2422082274593172953moz-txt-link-abbreviated" href="mailto:Devel@lists.opensips.org" target="_blank">Devel@lists.opensips.org</a>
<a class="m_2422082274593172953moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/devel</a>
</pre>
</blockquote>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Maksym Sobolyev<br>Sippy Software, Inc.<br>Internet Telephony (VoIP) Experts<br>Tel (Canada): +1-778-783-0474<br>Tel (Toll-Free): +1-855-747-7779<br>Fax: +1-866-857-6942<br>Web: <a href="http://www.sippysoft.com" target="_blank">http://www.sippysoft.com</a><br>MSN: <a href="mailto:sales@sippysoft.com" target="_blank">sales@sippysoft.com</a><br>Skype: SippySoft<br></div></div>
</div>