<div dir="auto">Hi Ben<div dir="auto"><br><div dir="auto">I had a crash issue and asked for help too, no response. It's a terrible feeling, that you are alone with your problem and no one even respond.</div><div dir="auto"><br></div><div dir="auto">Opensips guys, please respond to us, even with: sorry I have no time to look at this.</div><div dir="auto"><br></div><div dir="auto">Regards</div><div dir="auto">Daniel</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em qua, 8 de mai de 2019 19:14, Ben Newlin <<a href="mailto:Ben.Newlin@genesys.com">Ben.Newlin@genesys.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_2242085418476537979WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Has anyone had a chance to look into this issue? I was able to mitigate by inserting small delays in my test clients, but now I am running into an issue where responses from one OpenSIPS to another are being
sent quickly enough to cause this problem. Making changes in my testbed is one thing, but I really don’t want to start injecting delays into my processing script to work around this issue.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The issue is that final responses to a transaction that are received within a few milliseconds of the transaction being sent are not stopping the retransmission timer. The response is handled properly by OpenSIPS,
but approximately 500ms later OpenSIPS will retransmit the request anyway. I can reproduce it very regularly in our testbed and it should also be easily reproducible with a simple SIPp scenario.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Trace from the latest occurrence is here:
</span><a href="https://pastebin.com/YAXFwNLb" target="_blank" rel="noreferrer">https://pastebin.com/YAXFwNLb</a><span style="font-size:11.0pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’d be happy to open an issue on Github as well if that is desired.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben Newlin </span>
<span style="font-size:11.0pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Devel <<a href="mailto:devel-bounces@lists.opensips.org" target="_blank" rel="noreferrer">devel-bounces@lists.opensips.org</a>> on behalf of Ben Newlin <<a href="mailto:Ben.Newlin@genesys.com" target="_blank" rel="noreferrer">Ben.Newlin@genesys.com</a>><br>
<b>Reply-To: </b>OpenSIPS devel mailling list <<a href="mailto:devel@lists.opensips.org" target="_blank" rel="noreferrer">devel@lists.opensips.org</a>><br>
<b>Date: </b>Thursday, April 25, 2019 at 11:46 AM<br>
<b>To: </b>OpenSIPS devel mailling list <<a href="mailto:devel@lists.opensips.org" target="_blank" rel="noreferrer">devel@lists.opensips.org</a>><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] Erroneous Retransmissions<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">I have found that this issue seems to only occur when the responses are received immediately after the request has been sent; within a few milliseconds. Inserting a delay in the response seems to have made
the erroneous retransmissions stop. This does not make it any less of a bug, but does seem to be a workaround.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben Newlin </span>
<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Devel <<a href="mailto:devel-bounces@lists.opensips.org" target="_blank" rel="noreferrer">devel-bounces@lists.opensips.org</a>> on behalf of Ben Newlin <<a href="mailto:Ben.Newlin@genesys.com" target="_blank" rel="noreferrer">Ben.Newlin@genesys.com</a>><br>
<b>Reply-To: </b>OpenSIPS devel mailling list <<a href="mailto:devel@lists.opensips.org" target="_blank" rel="noreferrer">devel@lists.opensips.org</a>><br>
<b>Date: </b>Friday, April 19, 2019 at 8:47 AM<br>
<b>To: </b>OpenSIPS devel mailling list <<a href="mailto:devel@lists.opensips.org" target="_blank" rel="noreferrer">devel@lists.opensips.org</a>><br>
<b>Subject: </b>[OpenSIPS-Devel] Erroneous Retransmissions</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Resending the issue described below to the devel mailing list as I believe this is a bug, not a configuration issue.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben Newlin </span>
<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Users <<a href="mailto:users-bounces@lists.opensips.org" target="_blank" rel="noreferrer">users-bounces@lists.opensips.org</a>> on behalf of Ben Newlin <<a href="mailto:Ben.Newlin@genesys.com" target="_blank" rel="noreferrer">Ben.Newlin@genesys.com</a>><br>
<b>Reply-To: </b>OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank" rel="noreferrer">users@lists.opensips.org</a>><br>
<b>Date: </b>Monday, April 15, 2019 at 7:20 PM<br>
<b>To: </b>OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank" rel="noreferrer">users@lists.opensips.org</a>><br>
<b>Subject: </b>Re: [OpenSIPS-Users] Erroneous Retransmissions</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Just wanted to bump this to see if anyone has had a chance to take a look. We are continuing to see this failing our tests intermittently. I have a new example here:
<a href="https://pastebin.com/kQJSGE56" target="_blank" rel="noreferrer">https://pastebin.com/kQJSGE56</a>.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In this example you can see that the retransmission of the INVITE is occurring not just after the 100 Trying has been received, but also after a 500 final response has already been received and is being processed,
but the INVITE is still retransmitted!</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Any help on this would be greatly appreciated.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""># opensips -V</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">version: opensips 2.4.5 (x86_64/linux)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">poll method support: poll, epoll, sigio_rt, select.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">git revision: d025b4f61</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">main.c compiled on 21:18:18 Apr 12 2019 with gcc 7</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben Newlin </span>
<u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Users <<a href="mailto:users-bounces@lists.opensips.org" target="_blank" rel="noreferrer">users-bounces@lists.opensips.org</a>> on behalf of Ben Newlin <<a href="mailto:Ben.Newlin@genesys.com" target="_blank" rel="noreferrer">Ben.Newlin@genesys.com</a>><br>
<b>Reply-To: </b>OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank" rel="noreferrer">users@lists.opensips.org</a>><br>
<b>Date: </b>Thursday, April 4, 2019 at 3:10 PM<br>
<b>To: </b>OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank" rel="noreferrer">users@lists.opensips.org</a>><br>
<b>Subject: </b>[OpenSIPS-Users] Erroneous Retransmissions</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We are seeing an issue which is causing our tests to fail intermittently. It appears that sometimes OpenSIPS will retransmit a relayed INVITE request, even though a 100 Trying response has already been received
and processed. This only happens intermittently.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have captured debug logs for the scenario:
<a href="https://pastebin.com/bHcBJmtv" target="_blank" rel="noreferrer">https://pastebin.com/bHcBJmtv</a>. In the log, you can see the 100 Trying response is received on line 707 and it appears to be matched to the correct transaction on line 729. However then on line 774 you can see the
retransmission of the INVITE anyway.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have verified using tcpdump that the INVITE is being retransmitted; it is not just a log error. And our tests are failing due to the unexpected extra 100 Trying response to the retransmission.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Can anyone explain why this would occur?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben Newlin </span>
<u></u><u></u></p>
</div>
</div>
_______________________________________________<br>
Devel mailing list<br>
<a href="mailto:Devel@lists.opensips.org" target="_blank" rel="noreferrer">Devel@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" rel="noreferrer noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><br>
</blockquote></div>