<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \, serif";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New","serif";
color:black;}
tt
{mso-style-priority:99;
font-family:"Courier New","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle29
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Bogdan, I have created an issue with opensips github<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><a href="https://github.com/OpenSIPS/opensips/issues/591">https://github.com/OpenSIPS/opensips/issues/591</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">And uploaded the logs specified in the link with the description along with timestamps.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Rahul Gupta<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Bogdan-Andrei Iancu [mailto:bogdan@opensips.org]
<br>
<b>Sent:</b> Tuesday, August 04, 2015 10:51 AM<br>
<b>To:</b> Gupta, Rahul; users@lists.opensips.org<br>
<b>Subject:</b> Re: ERROR:core:io_watch_del: BUG - trying to del fd 38 with flags 2 1<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Rahul,</span></tt><span style="font-size:10.0pt;font-family:"Courier New","serif""><br>
<br>
<tt>Any logs will be useful (use some pastebin or file transfer to upload).</tt><br>
<br>
<tt>On your observations : when a connection timeout does expire (via __tcpconn_lifetime()), reactor_del_all() removes the fd with both IO_WATCH_READ and IO_WATCH_WRITE flags. While in the logs it shows "1", removing with READ only.</tt><br>
<br>
<tt>Going back </tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre><a href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 04.08.2015 16:29, Gupta, Rahul wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Hi Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I tried the scenario with 1.11.3 as well as 1.11.5 latest code and could reproduce it consistently. I can turn on the higher debug level logging and provide you the log file. Let me know where can I upload these
logs ?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Rahul </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Bogdan-Andrei Iancu [<a href="mailto:bogdan@opensips.org">mailto:bogdan@opensips.org</a>]
<br>
<b>Sent:</b> Tuesday, August 04, 2015 8:18 AM<br>
<b>To:</b> Gupta, Rahul; <a href="mailto:users@lists.opensips.org">users@lists.opensips.org</a><br>
<b>Subject:</b> Re: ERROR:core:io_watch_del: BUG - trying to del fd 38 with flags 2 1</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Rahul,</span></tt><span style="font-size:10.0pt;font-family:"Courier New","serif""><br>
<br>
<tt>I tried to reproduce your scenario during the last 2 days. Thanks to the detailed instructions and stepping, I mananged to get (as described) to the last step. But instead of getting the error you mentioned, the tcp conn timeout event was properly handled:</tt><br>
<br>
<tt>[23203] DBG:core:__tcpconn_lifetime: timeout for hash=3 - 0x7f94f298fea0 (322 > 317)</tt><br>
<tt>[23203] DBG:core:io_watch_del: [TCP_main] io_watch_del op on index -1 29 (0x8828a0, 29, -1, 0x10,0x3) fd_no=22 called</tt><br>
<br>
<tt>Are you sure you are using the latest GIT code (I tested both 2.1 and 1.11).</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre><a href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 31.07.2015 17:40, Gupta, Rahul wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Hi Bogdan, thanks for looking into this issue. Let me know if anyway I can help to get to the bottom of this issue, like providing logs etc</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Rahul</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Bogdan-Andrei Iancu [<a href="mailto:bogdan@opensips.org">mailto:bogdan@opensips.org</a>]
<br>
<b>Sent:</b> Tuesday, July 28, 2015 1:13 PM<br>
<b>To:</b> Gupta, Rahul; <a href="mailto:users@lists.opensips.org">users@lists.opensips.org</a><br>
<b>Subject:</b> Re: ERROR:core:io_watch_del: BUG - trying to del fd 38 with flags 2 1</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Rahul,</span></tt><span style="font-size:10.0pt;font-family:"Courier New","serif""><br>
<br>
<tt>These are great news - that you find a way to reproduce it. Let me do it and start debugging.</tt><br>
<br>
<tt>I really do appreciate your effort in dealing with this error.</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre><a href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 24.07.2015 00:19, Gupta, Rahul wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Hi Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Almost a month ago, I have raised the issue regarding the opensips log file filling up with the following message
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">ERROR:core:io_watch_del: BUG - trying to del fd 38 with flags 2 1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Now I can reproduce it every time with the following scenario:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">1)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">UAC and UAS are connected to opensips proxy with TCP transport</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">2)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">Connections looks good</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">3)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">Netstat shows the connection to UAC in ESTABLISHED state and the tcp_conn_lists of opensips also looks fine for that UAC</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">4)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">Now unplug the Ethernet cable on UAC</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">5)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">After tcp connection timeout (set to 5 mins in opensips.cfg), the tcp connection goes away from netstat as well as from tcp_conn_lists</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">6)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">UAS tries another tcp call to the UAC which is still unplugged</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">7)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">tcp_conn_lists shows the tcp connection to the UAC and netstat shows the connection in SYN_SENT state</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">8)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">After the tcp connection timeout (set to 5 mins in opensips.cfg), the connection goes away from netstat however it remains there in tcp_conn_lists and at that moment the “BUG - trying to del fd 38 with flags 2 1” starts printing
in infinite loop.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I looked at the source code and observed the following</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">1)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">When the network cable is plugged in </span><o:p></o:p></p>
<p class="MsoListParagraph"><span style="color:#1F497D">The io_watch_add happens with flag IO_WATCH_READ in tcp_main.c when the command is CONN_NEW</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">2)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">When the network cable is unplugged</span><o:p></o:p></p>
<p class="MsoListParagraph"><span style="color:#1F497D">The io_watch_add happens with flag IO_WATCH_WRITE in tcp_main.c when the command is ASYNC_CONNECT</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">3)</span><span style="font-size:7.0pt">
</span><span style="color:#1F497D">While doing io_watch_del after timeout, from handle_tcpconn_ev method, io_watch_del always uses IO_WATCH_READ to delete the fd, which gives this error in case of unplugged cable.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Please look into this and suggest how can I fix this issue ?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Rahul Gupta</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:gray">------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:gray">DISCLAIMER: This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not
duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unintended recipients are prohibited from taking action on the basis of information in this e-mail.E-mail messages may
contain computer viruses or other defects, may not be accurately replicated on other systems, or may be intercepted, deleted or interfered with without the knowledge of the sender or the intended recipient. If you are not comfortable with the risks associated
with e-mail messages, you may decide not to use e-mail to communicate with IPC. IPC reserves the right, to the extent and under circumstances permitted by applicable law, to retain, monitor and intercept e-mail messages to and from its systems.</span><o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
</body>
</html>