[OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load

Binan AL Halabi binanalhalabi at yahoo.com
Fri Oct 12 02:01:34 CEST 2012 

Hi Diego,

I forgot to tell that opening large number of tcp sockets means opening large amount of file descriptors.

echo 128000 > /proc/sys/fs/inode-max 
echo 64000 > /proc/sys/fs/file-max
ulimit -n 64000

// Binan





________________________________
 Från: Diego Barberio <diego.barberio at redmondsoftware.com>
Till: Binan AL Halabi <binanalhalabi at yahoo.com>; OpenSIPS users mailling list <users at lists.opensips.org> 
Skickat: fredag, 12 oktober 2012 1:38
Ämne: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load
 

Hi Binnan,

Thanks for all your help.
I've applied all you suggestions and nothing changed.

I generated a trace of OpenSIPs connections using TCP dump (attached). I noticed that the send queues have a lot of data between 6 pm and 8 pm (when is the period the problem appears).
As far as I know this means opensips is calling send funcion but the OS is not actually sending data, am I right?

What can be causing this problem? Do you have any other ideas?

Again, thanks a lot for your help.

Regards
Diego


On Tue, Oct 9, 2012 at 7:20 PM, Binan AL Halabi <binanalhalabi at yahoo.com> wrote:


>
>Do the following to increase TCP performance:
>
>
>1- Increase the number of  available local ports:  
>
>                echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
>
>
>2- Increase the amount of memory associated with socket buffers (socket input and output queues):
>For input queues:
>
>                echo 262143 > /proc/sys/net/core/rmem_max 
>
>                echo 262143 > /proc/sys/net/core/rmem_default
>And wmem_max , wmem_default for output queues.
>
>
>3- TCP stack
>
>            echo 0 > /proc/sys/net/ipv4/tcp_sack 
>
>            echo 0 > /proc/sys/net/ipv4/tcp_timestamps 
>
>
>4- opensips memory pool is well configured : http://www.opensips.org/Resources/DocsTsMem
>
>
>// Binan
>
>
>________________________________
> Från: Diego Barberio <diego.barberio at redmondsoftware.com>
>Till: OpenSIPS users mailling list <users at lists.opensips.org> 
>Skickat: tisdag, 9 oktober 2012 20:08
>Ämne: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load
> 
>
>
>Hi Binan,
>
>Thank's to Bogdan's patch I was able to run opensips 1.8.1. However, the problem with TCP under highload is still present.
>
>Do you have any other ideas?
>
>Thanks
>Diego
>
>
>On Mon, Sep 10, 2012 at 4:59 PM, Diego Barberio <diego.barberio at redmondsoftware.com> wrote:
>
>Hi Binan,
>>
>>I tried upgrading to 1.8.1 but I'm having some issues I'm duscussin with Bogdan on the following thread:
>>
>>http://lists.opensips.org/pipermail/users/2012-August/022764.html
>>
>>Thanks
>>Diego
>>
>>
>>
>>On Mon, Sep 10, 2012 at 4:47 PM, Binan AL Halabi <binanalhalabi at yahoo.com> wrote:
>>
>>hi Diego,
>>>
>>>1- As you dont have state in database so you dont need to check that.
>>>2- Why you dont upgrade to opensips 1.8.1 ? since it contains TCP fix.
>>>
>>>Regards.
>>>//Binan
>>>
>>>--- On Mon, 9/10/12, Binan AL Halabi <binanalhalabi at yahoo.com> wrote:
>>>
>>>
>>>>From: Binan AL Halabi <binanalhalabi at yahoo.com>
>>>>
>>>>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load
>>>>To: "OpenSIPS users mailling list" <users at lists.opensips.org>
>>>>Date: Monday, September 10, 2012, 12:19 PM
>>>>
>>>>
>>>>
>>>>If you have
>>>>
>>>>--- On Mon, 9/10/12, Diego Barberio <diego.barberio at redmondsoftware.com> wrote:
>>>>
>>>>
>>>>>From: Diego Barberio <diego.barberio at redmondsoftware.com>
>>>>>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load
>>>>>To: "OpenSIPS users mailling list" <users at lists.opensips.org>
>>>>>Date: Monday, September 10, 2012, 10:51 AM
>>>>>
>>>>>
>>>>>Hi Binan,
>>>>>
>>>>>I understand what do you say with "play around TCP connection lifetime" however I don't get the part you say "check the database". I don't have any database, opensips is working without any DB.
>>>>>Which database are you talking about?
>>>>>
>>>>>Thanks
>>>>>Diego
>>>>>
>>>>>
>>>>>On Mon, Sep 10, 2012 at 1:37 PM, Binan AL Halabi <binanalhalabi at yahoo.com> wrote:
>>>>>
>>>>>Hi Diego,
>>>>>>play around TCP connection lifetime, you could find something.
>>>>>>
>>>>>>one thing more check the database during the busy hour and see if something expired.
>>>>>>
>>>>>>//Binan
>>>>>>
>>>>>>
>>>>>>--- On Mon, 9/10/12, Diego Barberio <diego.barberio at redmondsoftware.com> wrote:
>>>>>>
>>>>>>
>>>>>>>From: Diego Barberio <diego.barberio at redmondsoftware.com>
>>>>>>>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load
>>>>>>>To: "OpenSIPS users mailling list" <users at lists.opensips.org>
>>>>>>>Date: Monday, September 10, 2012, 6:39 AM
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Hi All,
>>>>>>>
>>>>>>>I'm still having this issue. However, I went on with my investigation and I've discovered that this issue only happens with my SIP over TCP connections
 (I have 10 TCP connections)
>>>>>>>I've upgraded to version:
>>>>>>>Server:: OpenSIPS (1.7.2-notls (i386/linux))
>>>>>>>
>>>>>>>And the issue is still there.
>>>>>>>
>>>>>>>
>>>>>>>Do you have any other ideas? (Disabling TCP is not an option)
>>>>>>>
>>>>>>>Thanks
>>>>>>>Diego
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>From: users-bounces at lists.opensips.org
>>>>>>>>[mailto:users-bounces at lists.opensips.org] On Behalf Of Ali Pey
>>>>>>>>Sent: viernes, 10 de agosto de 2012 01:30 p.m.
>>>>>>>>To: OpenSIPS users mailling list
>>>>>>>>Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
>>>>>>>>under high load
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>I can't think of anything else.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Dear opensips team,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Hope you had a good flight-terminator :)
>>>>>>>>
>>>>>>>>Can you take a look at this please?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks,
>>>>>>>>
>>>>>>>>Ali Pey
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>On Fri, Aug 10, 2012 at 12:05 PM, Diego Barberio
>>>>>>>><diego.barberio at redmondsoftware.com> wrote:
>>>>>>>>
>>>>>>>>Hi Ali,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks again for rour feedbak.
>>>>>>>>
>>>>>>>>Unfortunately that's not the case. I'm attaching an example with the same
>>>>>>>>Via that works perfectly.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Also, opensips works perfectly all day and only stops sending some OKs
>>>>>>>>during peak hour.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Do you have any other idea?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>
>>>>>>>>Diego
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>From: users-bounces at lists.opensips.org
>>>>>>>>[mailto:users-bounces at lists.opensips.org] On Behalf Of Ali Pey
>>>>>>>>Sent: viernes, 10 de agosto de 2012 11:00 a.m.
>>>>>>>>
>>>>>>>>
>>>>>>>>To: OpenSIPS users mailling list
>>>>>>>>Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
>>>>>>>>under high load
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Hi Diego,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>I looked at your pcap file and I see that your second via has this:
>>>>>>>>
>>>>>>>>BC01.MBCSCR4.MSS.IMS.TIGO.COM.GT
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Is this a valid address? Maybe opensips can not resolve it so it ignores the
>>>>>>>>via. Have you tried adding it to /etc/hosts.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Compare a good 200OK with one of these and see if there is any difference.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Regards,
>>>>>>>>
>>>>>>>>Ali Pey
>>>>>>>>
>>>>>>>>On Fri, Aug 10, 2012 at 9:20 AM, Diego Barberio
>>>>>>>><diego.barberio at redmondsoftware.com> wrote:
>>>>>>>>
>>>>>>>>Hi Ali,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>I removed set the debug flag to 0 and added the "-" before the file path,
>>>>>>>>and I sill have the same issue.
>>>>>>>>
>>>>>>>>Is there any other thing I can try?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>
>>>>>>>>Diego
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>From: users-bounces at lists.opensips.org
>>>>>>>>[mailto:users-bounces at lists.opensips.org] On Behalf Of Ali Pey
>>>>>>>>Sent: jueves, 09 de agosto de 2012 03:30 p.m.
>>>>>>>>To: OpenSIPS users mailling list
>>>>>>>>Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
>>>>>>>>under high load
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Hi Diego,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Have you tried disabling the logging? That's usually the problem. Syslog
>>>>>>>>slows down opensips.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Also in /etc/syslog.conf as '-' before the file path to make it
>>>>>>>>asynchronous.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Regards,
>>>>>>>>
>>>>>>>>Ali Pey
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>On Thu, Aug 9, 2012 at 2:19 PM, Diego Barberio
>>>>>>>><diego.barberio at redmondsoftware.com> wrote:
>>>>>>>>
>>>>>>>>Hello,
>>>>>>>>
>>>>>>>>I'm using opensips to forward incoming INVITEs to a set of 8 applications.
>>>>>>>>I'm using dispatcher module to achieve this.
>>>>>>>>During peak hour, opensips receives about 350 INVITEs per second, during
>>>>>>>>this period about 30 OK responses per second my application sends correctly
>>>>>>>>aren't forwarded by opensips.
>>>>>>>>
>>>>>>>>I've recompiled opensips so it can use more memory by modifing the config.h
>>>>>>>>file:
>>>>>>>>
>>>>>>>>#define PKG_MEM_POOL_SIZE 10*1024*1024          /*!< Used only if PKG_MALLOC
>>>>>>>>is defined*/
>>>>>>>>#define SHM_MEM_SIZE 320                        /*!< Used if SH_MEM is
>>>>>>>>defined*/
>>>>>>>>
>>>>>>>>
>>>>>>>>The incoming INVITEs come from UDP and TCP.
>>>>>>>>
>>>>>>>>
>>>>>>>>For each of the OK responses that aren't forwarded this entry appears in the
>>>>>>>>messages file:
>>>>>>>>
>>>>>>>>usr/local/sbin/opensips[24554]: ERROR:core:forward_reply: no 2nd via found
>>>>>>>>in reply
>>>>>>>>
>>>>>>>>According to my research on the web this message means that the OK only has
>>>>>>>>one Via header, however this is not the case and the capture attached (made
>>>>>>>>on server which runs opensips) shows it.
>>>>>>>>
>>>>>>>>I've also tried to increase the log level, but it made opensips to work slow
>>>>>>>>even in low traffic hours.
>>>>>>>>
>>>>>>>>
>>>>>>>>I run some test in my lab and opensips worked perfectly even under more than
>>>>>>>>350 INVITES per second, this problem only occurs in production
>>>>>>>>
>>>>>>>>I really don't know what else to do.
>>>>>>>>Could you help me to fix this issue or at least move on with the
>>>>>>>>investigation?
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>Diego
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>Users mailing list
>>>>>>>>Users at lists.opensips.org
>>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>Users mailing list
>>>>>>>>Users at lists.opensips.org
>>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>Users mailing list
>>>>>>>>Users at lists.opensips.org
>>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>-----Inline Attachment Follows-----
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>Users mailing list
>>>>>>>Users at lists.opensips.org
>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>> 
>>>>>>_______________________________________________
>>>>>>Users mailing list
>>>>>>Users at lists.opensips.org
>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>
>>>>>-----Inline Attachment Follows-----
>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>Users mailing list
>>>>>Users at lists.opensips.org
>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>> 
>>>>-----Inline Attachment Follows-----
>>>>
>>>>
>>>>_______________________________________________
>>>>Users mailing list
>>>>Users at lists.opensips.org
>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>> 
>>>_______________________________________________
>>>Users mailing list
>>>Users at lists.opensips.org
>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>
>
>_______________________________________________
>Users mailing list
>Users at lists.opensips.org
>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>_______________________________________________
>Users mailing list
>Users at lists.opensips.org
>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20121011/d839ed38/attachment-0001.htm>

More information about the Users mailing list