Hi Binnan,<br><br>Thanks for all your help.<br><br>I've made a network trace and the other party is sending the TCP's ACKs with delay, so it's seems opensips is not causing this issue.<br>I will keep you updated if anything else comes out.<br>
<br>Thanks<br>Diego<br><br><div class="gmail_quote">On Thu, Oct 11, 2012 at 6:01 PM, Binan AL Halabi <span dir="ltr"><<a href="mailto:binanalhalabi@yahoo.com" target="_blank">binanalhalabi@yahoo.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div><div style="font-size:10pt;font-family:times new roman,new york,times,serif"><div>
<span><br></span></div>Hi Diego,<br><div style="font-family:times new roman,new york,times,serif;font-size:10pt"><div style="font-family:times new roman,new york,times,serif;font-size:12pt"><div><div><div style="font-size:10pt;font-family:times new roman,new york,times,serif">
I forgot to tell that opening large number of tcp sockets means opening large amount of file descriptors.<br><br>echo 128000 > /proc/sys/fs/inode-max
<br>echo 64000 > /proc/sys/fs/file-max<br>ulimit -n 64000<br><br>// Binan<br><br><div><span><br></span></div><div><br></div> <div style="font-family:times new roman,new york,times,serif;font-size:10pt"> <div style="font-family:times new roman,new york,times,serif;font-size:12pt">
<div dir="ltr"> <font face="Arial"> <hr size="1"> <b><span style="font-weight:bold">Från:</span></b> Diego Barberio <<a href="mailto:diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>><br>
<b><span style="font-weight:bold">Till:</span></b> Binan AL Halabi <<a href="mailto:binanalhalabi@yahoo.com" target="_blank">binanalhalabi@yahoo.com</a>>; OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a>> <br>
<b><span style="font-weight:bold">Skickat:</span></b> fredag, 12 oktober 2012 1:38<br> <b><span style="font-weight:bold">Ämne:</span></b> Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load<br>
</font> </div> <br>
<div>Hi Binnan,<br><br>Thanks for all your help.<br>I've applied all you suggestions and nothing changed.<br><br>I generated a trace of OpenSIPs connections using TCP dump (attached). I noticed that the send queues have a lot of data between 6 pm and 8 pm (when is the period the problem appears).<br>
As far as I know this means opensips is calling send funcion but the OS is not actually sending data, am I right?<br><br>What can be causing this problem? Do you have any other ideas?<br><br>Again, thanks a lot for your help.<br>
<br>Regards<br>Diego<br><br><div>On Tue, Oct 9, 2012 at 7:20 PM, Binan AL Halabi <span dir="ltr"><<a rel="nofollow" href="mailto:binanalhalabi@yahoo.com" target="_blank">binanalhalabi@yahoo.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:10pt;font-family:times new roman,new york,times,serif"><br><div>Do the following to increase TCP performance:</div>
<div><br></div><div>1- Increase the number of available local ports: <br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif"><span> </span><span> </span><span> </span><span> </span>echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range</div>
<div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif"><br></div><div>2- Increase the amount of memory associated with socket buffers (socket input and output queues):</div>
<div>For input
queues:<br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif"><span> </span><span> </span><span> </span><span> </span>echo 262143 > /proc/sys/net/core/rmem_max <br>
</div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif"><span> </span><span> </span><span> </span><span> </span>echo 262143 > /proc/sys/net/core/rmem_default</div>
<div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif">And wmem_max ,
wmem_default for output queues.</div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif"><br></div><div>3- TCP stack<br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif">
<span> </span><span> </span><span> </span>echo 0 > /proc/sys/net/ipv4/tcp_sack <br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif">
<span> </span><span> </span><span> </span>echo 0 > /proc/sys/net/ipv4/tcp_timestamps <br></div><div style="font-style:normal;font-size:13.3333px;background-color:transparent;font-family:times new roman,new york,times,serif">
<br></div>4- opensips memory pool is well configured : <a href="http://www.opensips.org/Resources/DocsTsMem" target="_blank">http://www.opensips.org/Resources/DocsTsMem</a><br><br><br>// Binan<br><div> </div> <div style="font-family:times new roman,new york,times,serif;font-size:10pt">
<div style="font-family:times new roman,new york,times,serif;font-size:12pt"> <div dir="ltr"> <font face="Arial"> <hr size="1"> <b><span style="font-weight:bold">Från:</span></b> Diego Barberio <<a rel="nofollow" href="mailto:diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>><br>
<b><span style="font-weight:bold">Till:</span></b> OpenSIPS users mailling list <<a rel="nofollow" href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a>> <br> <b><span style="font-weight:bold">Skickat:</span></b> tisdag, 9 oktober 2012 20:08<br>
<b><span style="font-weight:bold">Ämne:</span></b> Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't
send an OK response under high load<br> </font> </div><div><div> <br>
<div>Hi Binan,<br><br>Thank's to Bogdan's patch I was able to run opensips 1.8.1. However, the problem with TCP under highload is still present.<br><br>Do you have any other ideas?<br><br>Thanks<br>Diego<br><br><div>
On Mon, Sep 10, 2012 at 4:59 PM, Diego Barberio <span dir="ltr"><<a rel="nofollow" href="mailto:diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Binan,<br><br>I tried upgrading to 1.8.1 but I'm having some issues I'm duscussin with Bogdan on the following thread:<br>
<br><a href="http://lists.opensips.org/pipermail/users/2012-August/022764.html" target="_blank">http://lists.opensips.org/pipermail/users/2012-August/022764.html</a><br>
<br>Thanks<span><font color="#888888"><br>Diego</font></span><div><div><br><br><div>On Mon, Sep 10, 2012 at 4:47 PM, Binan AL Halabi <span dir="ltr"><<a rel="nofollow" href="mailto:binanalhalabi@yahoo.com" target="_blank">binanalhalabi@yahoo.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td style="font:inherit" valign="top">hi Diego,<br><br>1- As you dont have state in database so you dont need to check that.<br>
2- Why you dont upgrade to opensips 1.8.1 ? since it contains TCP fix.<br><br>Regards.<br>//Binan<br><br>--- On <b>Mon, 9/10/12, Binan AL Halabi <i><<a rel="nofollow" href="mailto:binanalhalabi@yahoo.com" target="_blank">binanalhalabi@yahoo.com</a>></i></b> wrote:<br>
<blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px"><br>From: Binan AL Halabi <<a rel="nofollow" href="mailto:binanalhalabi@yahoo.com" target="_blank">binanalhalabi@yahoo.com</a>><div>
<br>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load<br>To: "OpenSIPS users mailling list" <<a rel="nofollow" href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a>><br>
</div>Date: Monday, September 10, 2012, 12:19 PM<div><div><br><br><div><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td style="font:inherit" valign="top">If you have<br><br>--- On <b>Mon, 9/10/12, Diego Barberio
<i><<a rel="nofollow" href="mailto:diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>></i></b> wrote:<br><blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px">
<br>From: Diego Barberio <<a rel="nofollow" href="mailto:diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>><br>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load<br>
To: "OpenSIPS users mailling list" <<a rel="nofollow" href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a>><br>Date: Monday, September 10, 2012, 10:51 AM<br><br><div>Hi Binan,<br>
<br>I understand what do you say with "play around TCP connection lifetime" however I don't get the part you say "check the database". I don't have any database, opensips is working without any DB.<br>
Which database are you talking about?<br><br>Thanks<br>Diego<br><br><div>On Mon, Sep 10, 2012 at 1:37 PM, Binan AL Halabi <span dir="ltr"><<a rel="nofollow">binanalhalabi@yahoo.com</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td style="font:inherit" valign="top">Hi Diego,<br>play around TCP connection lifetime, you could find something.<br>
<br>one thing more check the database during the busy hour and see if something expired.<br><br>//Binan<br><br><br>--- On <b>Mon, 9/10/12, Diego Barberio <i><<a rel="nofollow">diego.barberio@redmondsoftware.com</a>></i></b> wrote:<br>
<blockquote style="border-left:2px solid rgb(16,16,255);margin-left:5px;padding-left:5px"><br>From: Diego Barberio <<a rel="nofollow">diego.barberio@redmondsoftware.com</a>><br>
Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load<br>To: "OpenSIPS users mailling list" <<a rel="nofollow">users@lists.opensips.org</a>><br>
Date: Monday, September 10, 2012, 6:39 AM<div><div><br><br><div>Hi All,<br><br>I'm still having this issue. However, I went on with my investigation and I've discovered that this issue only happens with my SIP over TCP connections
(I have 10 TCP connections)<br>I've upgraded to version:<br>
Server:: OpenSIPS (1.7.2-notls (i386/linux))<br><br>And the issue is still there.<br><br><br>Do you have any other ideas? (Disabling TCP is not an option)<br><br>Thanks<br>Diego<br><div><br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<br>
<br>
From: <a rel="nofollow" href="http://mc/compose?to=users-bounces@lists.opensips.org" target="_blank">users-bounces@lists.opensips.org</a><br>
[mailto:<a rel="nofollow" href="http://mc/compose?to=users-bounces@lists.opensips.org" target="_blank">users-bounces@lists.opensips.org</a>] On Behalf Of Ali Pey<br>
Sent: viernes, 10 de agosto de 2012 01:30 p.m.<br>
To: OpenSIPS users mailling list<br>
Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response<br>
under high load<br>
<br>
<br>
<br>
I can't think of anything else.<br>
<br>
<br>
<br>
Dear opensips team,<br>
<br>
<br>
<br>
Hope you had a good flight-terminator :)<br>
<br>
Can you take a look at this please?<br>
<br>
<br>
<br>
Thanks,<br>
<br>
Ali Pey<br>
<br>
<br>
<br>
<br>
<br>
On Fri, Aug 10, 2012 at 12:05 PM, Diego Barberio<br>
<<a rel="nofollow" href="http://mc/compose?to=diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>> wrote:<br>
<br>
Hi Ali,<br>
<br>
<br>
<br>
Thanks again for rour feedbak.<br>
<br>
Unfortunately that's not the case. I'm attaching an example with the same<br>
Via that works perfectly.<br>
<br>
<br>
<br>
Also, opensips works perfectly all day and only stops sending some OKs<br>
during peak hour.<br>
<br>
<br>
<br>
Do you have any other idea?<br>
<br>
<br>
<br>
Thanks<br>
<br>
Diego<br>
<br>
<br>
<br>
From: <a rel="nofollow" href="http://mc/compose?to=users-bounces@lists.opensips.org" target="_blank">users-bounces@lists.opensips.org</a><br>
[mailto:<a rel="nofollow" href="http://mc/compose?to=users-bounces@lists.opensips.org" target="_blank">users-bounces@lists.opensips.org</a>] On Behalf Of Ali Pey<br>
Sent: viernes, 10 de agosto de 2012 11:00 a.m.<br>
<br>
<br>
To: OpenSIPS users mailling list<br>
Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response<br>
under high load<br>
<br>
<br>
<br>
Hi Diego,<br>
<br>
<br>
<br>
I looked at your pcap file and I see that your second via has this:<br>
<br>
<a rel="nofollow" href="http://bc01.mbcscr4.mss.ims.tigo.com.gt/" target="_blank">BC01.MBCSCR4.MSS.IMS.TIGO.COM.GT</a><br>
<br>
<br>
<br>
Is this a valid address? Maybe opensips can not resolve it so it ignores the<br>
via. Have you tried adding it to /etc/hosts.<br>
<br>
<br>
<br>
Compare a good 200OK with one of these and see if there is any difference.<br>
<br>
<br>
<br>
Regards,<br>
<br>
Ali Pey<br>
<br>
On Fri, Aug 10, 2012 at 9:20 AM, Diego Barberio<br>
<<a rel="nofollow" href="http://mc/compose?to=diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>> wrote:<br>
<br>
Hi Ali,<br>
<br>
<br>
<br>
I removed set the debug flag to 0 and added the "-" before the file path,<br>
and I sill have the same issue.<br>
<br>
Is there any other thing I can try?<br>
<br>
<br>
<br>
Thanks<br>
<br>
Diego<br>
<br>
<br>
<br>
From: <a rel="nofollow" href="http://mc/compose?to=users-bounces@lists.opensips.org" target="_blank">users-bounces@lists.opensips.org</a><br>
[mailto:<a rel="nofollow" href="http://mc/compose?to=users-bounces@lists.opensips.org" target="_blank">users-bounces@lists.opensips.org</a>] On Behalf Of Ali Pey<br>
Sent: jueves, 09 de agosto de 2012 03:30 p.m.<br>
To: OpenSIPS users mailling list<br>
Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response<br>
under high load<br>
<br>
<br>
<br>
Hi Diego,<br>
<br>
<br>
<br>
Have you tried disabling the logging? That's usually the problem. Syslog<br>
slows down opensips.<br>
<br>
<br>
<br>
Also in /etc/syslog.conf as '-' before the file path to make it<br>
asynchronous.<br>
<br>
<br>
<br>
Regards,<br>
<br>
Ali Pey<br>
<br>
<br>
<br>
On Thu, Aug 9, 2012 at 2:19 PM, Diego Barberio<br>
<<a rel="nofollow" href="http://mc/compose?to=diego.barberio@redmondsoftware.com" target="_blank">diego.barberio@redmondsoftware.com</a>> wrote:<br>
<br>
Hello,<br>
<br>
I'm using opensips to forward incoming INVITEs to a set of 8 applications.<br>
I'm using dispatcher module to achieve this.<br>
During peak hour, opensips receives about 350 INVITEs per second, during<br>
this period about 30 OK responses per second my application sends correctly<br>
aren't forwarded by opensips.<br>
<br>
I've recompiled opensips so it can use more memory by modifing the config.h<br>
file:<br>
<br>
#define PKG_MEM_POOL_SIZE 10*1024*1024 /*!< Used only if PKG_MALLOC<br>
is defined*/<br>
#define SHM_MEM_SIZE 320 /*!< Used if SH_MEM is<br>
defined*/<br>
<br>
<br>
The incoming INVITEs come from UDP and TCP.<br>
<br>
<br>
For each of the OK responses that aren't forwarded this entry appears in the<br>
messages file:<br>
<br>
usr/local/sbin/opensips[24554]: ERROR:core:forward_reply: no 2nd via found<br>
in reply<br>
<br>
According to my research on the web this message means that the OK only has<br>
one Via header, however this is not the case and the capture attached (made<br>
on server which runs opensips) shows it.<br>
<br>
I've also tried to increase the log level, but it made opensips to work slow<br>
even in low traffic hours.<br>
<br>
<br>
I run some test in my lab and opensips worked perfectly even under more than<br>
350 INVITES per second, this problem only occurs in production<br>
<br>
I really don't know what else to do.<br>
Could you help me to fix this issue or at least move on with the<br>
investigation?<br>
<br>
Thanks<br>
Diego<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a rel="nofollow" href="http://mc/compose?to=Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a rel="nofollow" href="http://mc/compose?to=Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a rel="nofollow" href="http://mc/compose?to=Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
<br>
<br>
</blockquote></div><br>
</div><br></div></div>-----Inline Attachment Follows-----<div><br><br><div>_______________________________________________<br>Users mailing list<br><a rel="nofollow" href="http://mc/compose?to=Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br></div></div></blockquote></td></tr></tbody></table><br>_______________________________________________<br>
Users mailing list<br>
<a rel="nofollow">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br>
</div><br>-----Inline Attachment Follows-----<br><br><div>_______________________________________________<br>Users mailing list<br><a rel="nofollow">Users@lists.opensips.org</a><br><a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div></blockquote></td></tr></tbody></table></div><br>-----Inline Attachment Follows-----<br><br><div>_______________________________________________<br>Users mailing list<br><a rel="nofollow" href="http://mc/compose?to=Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br></div></div></div></blockquote></td></tr></tbody></table><br>
_______________________________________________<br>
Users mailing list<br>
<a rel="nofollow" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br>
</div></div></blockquote></div><br>
</div><br>_______________________________________________<br>Users mailing list<br><a rel="nofollow" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br><a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br><br> </div></div></div> </div> </div></div><br>_______________________________________________<br>
Users mailing list<br>
<a rel="nofollow" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a rel="nofollow" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br>
</div><br><br> </div> </div> </div></div></div><br><br> </div> </div> </div></div></div></div></blockquote></div><br>