<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<div>Hi Vlad,</div>
<div> </div>
<div>Thanks for taking up this issue.</div>
<div>1. Yes. <span style="font-family: Verdana; font-size: 12px; line-height: 19.200000762939453px;">tcp_async=1</span></div>
<div>2. Running on CentOS ( 2.6.32-358.6.2.el6.x86_64 )</div>
<div>3. It's occurring on every INVITE</div>
<div> </div>
<div>May be unrelated to this bug; but I am also seeing error messages like:</div>
<div>ERROR:script_helper:run_helper_logic: failed to match dialog, ci '4E14BD6E-53B17A8F00026071-6B1EA700'</div>
<div> </div>
<div>Thanks again...</div>
<div>BR</div>
<div>-Gary</div>
<div>
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Sent:</b> Monday, June 30, 2014 at 5:06 AM<br/>
<b>From:</b> "Vlad Paiu" <vladpaiu@opensips.org><br/>
<b>To:</b> users@lists.opensips.org<br/>
<b>Cc:</b> gn62@gmx.us<br/>
<b>Subject:</b> Re: [OpenSIPS-Users] TCP Errors</div>
<div name="quoted-content">
<div style="background-color: rgb(255,255,255);">
<div class="moz-cite-prefix">Hello,<br/>
<br/>
First of all, are you using the async_tcp option in your OpenSIPS script ?<br/>
Also, what architecture / OS are you running on ?<br/>
What steps are you taking in reproducing this ?<br/>
<br/>
Best Regards,
<pre class="moz-signature">Vlad Paiu
OpenSIPS Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a> </pre>
On 27.06.2014 19:59, Gary Nyquist wrote:</div>
<blockquote>
<div style="font-family: Verdana;font-size: 12.0px;">
<div>Thanks Bogdan for looking into it.</div>
<div> </div>
<div>Here is the version:</div>
<div>opensips -V<br/>
version: opensips 1.11.1-tls (x86_64/linux)<br/>
flags: STATS: On, USE_TCP, USE_TLS, DISABLE_NAGLE, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_SHM_MEM, FAST_LOCK-ADAPTIVE_WAIT<br/>
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535<br/>
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.<br/>
git revision: dbc8653<br/>
main.c compiled on 16:17:43 Jun 26 2014 with gcc 4.4.7</div>
<div>
<div> </div>
<div>Yes, this bug is reproducable.</div>
<div><span style="font-family: Verdana , sans-serif , Arial , "Trebuchet MS";font-size: 13.0px;line-height: 1.6em;">When it appears, the log file instantly gets filled with hundreds of repeating lines like this:</span></div>
<div> </div>
<div>Jun 26 20:30:31 ip-10-0-0-30 /usr/sbin/opensips[15229]: DBG:core:handle_tcpconn_ev: data available on 0x7f40bcec7398 6<br/>
Jun 26 20:30:31 ip-10-0-0-30 /usr/sbin/opensips[15229]: DBG:core:io_watch_del: io_watch_del op on index -1 6 (0x7dff20, 6, -1, 0x0,0x1) fd_no=110 called<br/>
Jun 26 20:30:31 ip-10-0-0-30 /usr/sbin/opensips[15229]: ERROR:core:io_watch_del: BUG - trying to del fd 6 with flags 2 1<br/>
Jun 26 20:30:31 ip-10-0-0-30 /usr/sbin/opensips[15229]: DBG:core:handle_tcpconn_ev: data available on 0x7f40bcec7398 6</div>
<div>Jun 26 20:30:31 ip-10-0-0-30 /usr/sbin/opensips[15229]: DBG:core:io_watch_del: io_watch_del op on index -1 6 (0x7dff20, 6, -1, 0x0,0x1) fd_no=110 called<br/>
Jun 26 20:30:31 ip-10-0-0-30 /usr/sbin/opensips[15229]: ERROR:core:io_watch_del: BUG - trying to del fd 6 with flags 2 1</div>
<div> </div>
<div>Thanks again for your help.</div>
<div> </div>
<div>BR</div>
<div>-Gary</div>
<div> </div>
<div> </div>
<div style="margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0 10.0px 10.0px;border-left: 2.0px solid rgb(195,217,229);">
<div style="margin: 0 0 10.0px 0;"><b>Sent:</b> Thursday, June 26, 2014 at 10:56 AM<br/>
<b>From:</b> "Bogdan-Andrei Iancu" <a class="moz-txt-link-rfc2396E" href="bogdan@opensips.org" target="_parent"><bogdan@opensips.org></a><br/>
<b>To:</b> "Gary Nyquist" <a class="moz-txt-link-rfc2396E" href="gn62@gmx.us" target="_parent"><gn62@gmx.us></a><br/>
<b>Cc:</b> "OpenSIPS users mailling list" <a class="moz-txt-link-rfc2396E" href="users@lists.opensips.org" target="_parent"><users@lists.opensips.org></a><br/>
<b>Subject:</b> Re: [OpenSIPS-Users] TCP Errors</div>
<div>
<div style="background-color: rgb(255,255,255);">
<div class="moz-cite-prefix"><tt>Hi,<br/>
<br/>
That bug log actually says OpenSIPS tries to remove a connection marked as READ from a list for WRITEs :)..<br/>
<br/>
I will look into that. What exact version do you use (opensips -V) ? also, can you reproduce this bug ?<br/>
<br/>
Thanks and regards,</tt>
<pre class="moz-signature">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
On 25.06.2014 20:27, Gary Nyquist wrote:</div>
<blockquote>
<div style="font-family: Verdana;font-size: 12.0px;">
<div>
<div>Thanks Bogdan for the detailed reply.</div>
<div> </div>
<div>I implemented your advice. </div>
<div>Those error messages are now gone.</div>
<div> </div>
<div>But seeing a new error now; not sure if it is related to that...</div>
<div> </div>
<div>ERROR:core:io_watch_del: BUG - trying to del fd 36 with flags 2 1</div>
<div> </div>
<div>Any advice?</div>
<div> </div>
<div>BR</div>
<div>-Gary</div>
<div>
<div style="margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0 10.0px 10.0px;border-left: 2.0px solid rgb(195,217,229);">
<div style="margin: 0 0 10.0px 0;"><b>Sent:</b> Wednesday, June 25, 2014 at 5:54 AM<br/>
<b>From:</b> "Bogdan-Andrei Iancu" <a class="moz-txt-link-rfc2396E"><bogdan@opensips.org></a><br/>
<b>To:</b> "OpenSIPS users mailling list" <a class="moz-txt-link-rfc2396E"><users@lists.opensips.org></a>, <a class="moz-txt-link-abbreviated">gn62@gmx.us</a><br/>
<b>Subject:</b> Re: [OpenSIPS-Users] TCP Errors</div>
<div>
<div style="background-color: rgb(255,255,255);">
<div class="moz-cite-prefix"><tt>Hi,<br/>
<br/>
Those messages say that OpenSIPS tried to open a TCP connection to a party which does not respond -> there was a timeout for connect in 10 seconds. Because of this blocking in connects, there were not more opensips workers available to handle other traffic.<br/>
<br/>
So what you need to do is :<br/>
- minimize the impact of the blocking connect - see my previous email on reducing the connect timeout<br/>
- you may configure OpenSIPS not to open new TCP connect (but to reuse the existing ones, open by clients). See tcp_no_new_conn_bflag<br/>
<a class="moz-txt-link-freetext" href="http://www.opensips.org/Documentation/Script-CoreParameters-1-11#toc96" target="_blank">http://www.opensips.org/Documentation/Script-CoreParameters-1-11#toc96</a><br/>
- try to understand the SIP patterns where such TCP connect fails so you can avoid them at script level.<br/>
<br/>
Regards,</tt>
<pre class="moz-signature">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
On 24.06.2014 21:34, Gary Nyquist wrote:</div>
<blockquote>
<div style="font-family: Verdana;font-size: 12.0px;">
<div style="font-family: Verdana;font-size: 12.0px;">
<p style="margin: 0.0px;padding: 0.0px;"><span style="font-family: Verdana;"><span style="font-size: 12.0px;"><font face="monospace" size="3"><span>Hi,</span></font></span></span></p>
<p style="margin: 0.0px;padding: 0.0px;"> </p>
<p style="margin: 0.0px;padding: 0.0px;"><span style="font-family: Verdana;"><span style="font-size: 12.0px;"><font face="monospace" size="3"><span>The following lines are repeating in the log.</span></font></span></span></p>
<p style="margin: 0.0px;padding: 0.0px;"> </p>
<p style="margin: 0.0px;padding: 0.0px;">ERROR:core:tcp_blocking_connect: timeout 10 s elapsed from 10 s<br/>
ERROR:core:tcpconn_connect: tcp_blocking_connect failed<br/>
ERROR:core:tcp_send: connect failed<br/>
ERROR:tm:msg_send: tcp_send failed</p>
<p style="margin: 0.0px;padding: 0.0px;"> </p>
<p style="margin: 0.0px;padding: 0.0px;"><a class="moz-txt-link-freetext">INFO:core:send2child</a>: no free tcp receiver, connection passed to the least busy one</p>
<p style="margin: 0.0px;padding: 0.0px;"> </p>
<p style="margin: 0.0px;padding: 0.0px;"><span style="font-family: Verdana;"><span style="font-size: 12.0px;"><font face="monospace" size="3"><span>Any guess, what could be the reason?</span></font></span></span></p>
<p style="margin: 0.0px;padding: 0.0px;"><span style="font-family: Verdana;"><span style="font-size: 12.0px;"><font face="monospace" size="3"><span> </span></font></span></span></p>
<p style="margin: 0.0px;padding: 0.0px;"><span style="font-family: Verdana;"><span style="font-size: 12.0px;"><font face="monospace" size="3"><span>Best</span></font></span></span></p>
<p style="margin: 0.0px;padding: 0.0px;"><span style="font-family: Verdana;"><span style="font-size: 12.0px;"><font face="monospace" size="3"><span>--Gary</span></font></span></span></p>
<div><span style="font-family: Verdana;"><span style="font-size: 12.0px;"> </span></span></div>
</div>
</div>
<fieldset class="mimeAttachmentHeader"> </fieldset>
<pre>_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><font face="Verdana, sans-serif, Arial, Trebuchet MS" size="3"><span style="font-style: normal;line-height: 20.796875px;white-space: normal;">
</span></font></pre>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
<fieldset class="mimeAttachmentHeader"> </fieldset>
<pre>_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="Users@lists.opensips.org" target="_parent">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><font face="Verdana, sans-serif, Arial, Trebuchet MS" size="3"><span style="font-style: normal; line-height: 20.799999237060547px; white-space: normal;">
</span></font></pre>
</blockquote>
</div>
</div>
</div>
</div>
</div></div></body></html>