[OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under high load

Diego Barberio diego.barberio at redmondsoftware.com
Thu Oct 18 02:29:53 CEST 2012


Hi Binnan,

Thanks for all your help.

I've made a network trace and the other party is sending the TCP's ACKs
with delay, so it's seems opensips is not causing this issue.
I will keep you updated if anything else comes out.

Thanks
Diego

On Thu, Oct 11, 2012 at 6:01 PM, Binan AL Halabi <binanalhalabi at yahoo.com>wrote:

>
> Hi Diego,
> I forgot to tell that opening large number of tcp sockets means opening
> large amount of file descriptors.
>
> echo 128000 > /proc/sys/fs/inode-max
> echo 64000 > /proc/sys/fs/file-max
> ulimit -n 64000
>
> // Binan
>
>
>
>   ------------------------------
> *Från:* Diego Barberio <diego.barberio at redmondsoftware.com>
> *Till:* Binan AL Halabi <binanalhalabi at yahoo.com>; OpenSIPS users
> mailling list <users at lists.opensips.org>
> *Skickat:* fredag, 12 oktober 2012 1:38
> *Ämne:* Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
> response under high load
>
> Hi Binnan,
>
> Thanks for all your help.
> I've applied all you suggestions and nothing changed.
>
> I generated a trace of OpenSIPs connections using TCP dump (attached). I
> noticed that the send queues have a lot of data between 6 pm and 8 pm (when
> is the period the problem appears).
> As far as I know this means opensips is calling send funcion but the OS is
> not actually sending data, am I right?
>
> What can be causing this problem? Do you have any other ideas?
>
> Again, thanks a lot for your help.
>
> Regards
> Diego
>
> On Tue, Oct 9, 2012 at 7:20 PM, Binan AL Halabi <binanalhalabi at yahoo.com>wrote:
>
>
> Do the following to increase TCP performance:
>
> 1- Increase the number of  available local ports:
>                 echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
>
> 2- Increase the amount of memory associated with socket buffers (socket
> input and output queues):
> For input queues:
>                 echo 262143 > /proc/sys/net/core/rmem_max
>                 echo 262143 > /proc/sys/net/core/rmem_default
> And wmem_max , wmem_default for output queues.
>
> 3- TCP stack
>             echo 0 > /proc/sys/net/ipv4/tcp_sack
>             echo 0 > /proc/sys/net/ipv4/tcp_timestamps
>
> 4- opensips memory pool is well configured :
> http://www.opensips.org/Resources/DocsTsMem
>
>
> // Binan
>    ------------------------------
> *Från:* Diego Barberio <diego.barberio at redmondsoftware.com>
> *Till:* OpenSIPS users mailling list <users at lists.opensips.org>
> *Skickat:* tisdag, 9 oktober 2012 20:08
> *Ämne:* Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
> response under high load
>
> Hi Binan,
>
> Thank's to Bogdan's patch I was able to run opensips 1.8.1. However, the
> problem with TCP under highload is still present.
>
> Do you have any other ideas?
>
> Thanks
> Diego
>
> On Mon, Sep 10, 2012 at 4:59 PM, Diego Barberio <
> diego.barberio at redmondsoftware.com> wrote:
>
> Hi Binan,
>
> I tried upgrading to 1.8.1 but I'm having some issues I'm duscussin with
> Bogdan on the following thread:
>
> http://lists.opensips.org/pipermail/users/2012-August/022764.html
>
> Thanks
> Diego
>
>
> On Mon, Sep 10, 2012 at 4:47 PM, Binan AL Halabi <binanalhalabi at yahoo.com>wrote:
>
> hi Diego,
>
> 1- As you dont have state in database so you dont need to check that.
> 2- Why you dont upgrade to opensips 1.8.1 ? since it contains TCP fix.
>
> Regards.
> //Binan
>
> --- On *Mon, 9/10/12, Binan AL Halabi <binanalhalabi at yahoo.com>* wrote:
>
>
> From: Binan AL Halabi <binanalhalabi at yahoo.com>
>
> Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
> response under high load
> To: "OpenSIPS users mailling list" <users at lists.opensips.org>
> Date: Monday, September 10, 2012, 12:19 PM
>
>
> If you have
>
> --- On *Mon, 9/10/12, Diego Barberio <diego.barberio at redmondsoftware.com>*wrote:
>
>
> From: Diego Barberio <diego.barberio at redmondsoftware.com>
> Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
> response under high load
> To: "OpenSIPS users mailling list" <users at lists.opensips.org>
> Date: Monday, September 10, 2012, 10:51 AM
>
> Hi Binan,
>
> I understand what do you say with "play around TCP connection lifetime"
> however I don't get the part you say "check the database". I don't have any
> database, opensips is working without any DB.
> Which database are you talking about?
>
> Thanks
> Diego
>
> On Mon, Sep 10, 2012 at 1:37 PM, Binan AL Halabi <binanalhalabi at yahoo.com>wrote:
>
> Hi Diego,
> play around TCP connection lifetime, you could find something.
>
> one thing more check the database during the busy hour and see if
> something expired.
>
> //Binan
>
>
> --- On *Mon, 9/10/12, Diego Barberio <diego.barberio at redmondsoftware.com>*wrote:
>
>
> From: Diego Barberio <diego.barberio at redmondsoftware.com>
> Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
> response under high load
> To: "OpenSIPS users mailling list" <users at lists.opensips.org>
> Date: Monday, September 10, 2012, 6:39 AM
>
>
> Hi All,
>
> I'm still having this issue. However, I went on with my investigation and
> I've discovered that this issue only happens with my SIP over TCP
> connections (I have 10 TCP connections)
> I've upgraded to version:
> Server:: OpenSIPS (1.7.2-notls (i386/linux))
>
> And the issue is still there.
>
>
> Do you have any other ideas? (Disabling TCP is not an option)
>
> Thanks
> Diego
>
>
>
>
>
> From: users-bounces at lists.opensips.org<http://mc/compose?to=users-bounces@lists.opensips.org>
> [mailto:users-bounces at lists.opensips.org<http://mc/compose?to=users-bounces@lists.opensips.org>]
> On Behalf Of Ali Pey
> Sent: viernes, 10 de agosto de 2012 01:30 p.m.
> To: OpenSIPS users mailling list
> Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
> under high load
>
>
>
> I can't think of anything else.
>
>
>
> Dear opensips team,
>
>
>
> Hope you had a good flight-terminator :)
>
> Can you take a look at this please?
>
>
>
> Thanks,
>
> Ali Pey
>
>
>
>
>
> On Fri, Aug 10, 2012 at 12:05 PM, Diego Barberio
> <diego.barberio at redmondsoftware.com<http://mc/compose?to=diego.barberio@redmondsoftware.com>>
> wrote:
>
> Hi Ali,
>
>
>
> Thanks again for rour feedbak.
>
> Unfortunately that's not the case. I'm attaching an example with the same
> Via that works perfectly.
>
>
>
> Also, opensips works perfectly all day and only stops sending some OKs
> during peak hour.
>
>
>
> Do you have any other idea?
>
>
>
> Thanks
>
> Diego
>
>
>
> From: users-bounces at lists.opensips.org<http://mc/compose?to=users-bounces@lists.opensips.org>
> [mailto:users-bounces at lists.opensips.org<http://mc/compose?to=users-bounces@lists.opensips.org>]
> On Behalf Of Ali Pey
> Sent: viernes, 10 de agosto de 2012 11:00 a.m.
>
>
> To: OpenSIPS users mailling list
> Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
> under high load
>
>
>
> Hi Diego,
>
>
>
> I looked at your pcap file and I see that your second via has this:
>
> BC01.MBCSCR4.MSS.IMS.TIGO.COM.GT<http://bc01.mbcscr4.mss.ims.tigo.com.gt/>
>
>
>
> Is this a valid address? Maybe opensips can not resolve it so it ignores
> the
> via. Have you tried adding it to /etc/hosts.
>
>
>
> Compare a good 200OK with one of these and see if there is any difference.
>
>
>
> Regards,
>
> Ali Pey
>
> On Fri, Aug 10, 2012 at 9:20 AM, Diego Barberio
> <diego.barberio at redmondsoftware.com<http://mc/compose?to=diego.barberio@redmondsoftware.com>>
> wrote:
>
> Hi Ali,
>
>
>
> I removed set the debug flag to 0 and added the "-" before the file path,
> and I sill have the same issue.
>
> Is there any other thing I can try?
>
>
>
> Thanks
>
> Diego
>
>
>
> From: users-bounces at lists.opensips.org<http://mc/compose?to=users-bounces@lists.opensips.org>
> [mailto:users-bounces at lists.opensips.org<http://mc/compose?to=users-bounces@lists.opensips.org>]
> On Behalf Of Ali Pey
> Sent: jueves, 09 de agosto de 2012 03:30 p.m.
> To: OpenSIPS users mailling list
> Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
> under high load
>
>
>
> Hi Diego,
>
>
>
> Have you tried disabling the logging? That's usually the problem. Syslog
> slows down opensips.
>
>
>
> Also in /etc/syslog.conf as '-' before the file path to make it
> asynchronous.
>
>
>
> Regards,
>
> Ali Pey
>
>
>
> On Thu, Aug 9, 2012 at 2:19 PM, Diego Barberio
> <diego.barberio at redmondsoftware.com<http://mc/compose?to=diego.barberio@redmondsoftware.com>>
> wrote:
>
> Hello,
>
> I'm using opensips to forward incoming INVITEs to a set of 8 applications.
> I'm using dispatcher module to achieve this.
> During peak hour, opensips receives about 350 INVITEs per second, during
> this period about 30 OK responses per second my application sends correctly
> aren't forwarded by opensips.
>
> I've recompiled opensips so it can use more memory by modifing the config.h
> file:
>
> #define PKG_MEM_POOL_SIZE 10*1024*1024          /*!< Used only if
> PKG_MALLOC
> is defined*/
> #define SHM_MEM_SIZE 320                        /*!< Used if SH_MEM is
> defined*/
>
>
> The incoming INVITEs come from UDP and TCP.
>
>
> For each of the OK responses that aren't forwarded this entry appears in
> the
> messages file:
>
> usr/local/sbin/opensips[24554]: ERROR:core:forward_reply: no 2nd via found
> in reply
>
> According to my research on the web this message means that the OK only has
> one Via header, however this is not the case and the capture attached (made
> on server which runs opensips) shows it.
>
> I've also tried to increase the log level, but it made opensips to work
> slow
> even in low traffic hours.
>
>
> I run some test in my lab and opensips worked perfectly even under more
> than
> 350 INVITES per second, this problem only occurs in production
>
> I really don't know what else to do.
> Could you help me to fix this issue or at least move on with the
> investigation?
>
> Thanks
> Diego
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <http://mc/compose?to=Users@lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <http://mc/compose?to=Users@lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <http://mc/compose?to=Users@lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>
>
> -----Inline Attachment Follows-----
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <http://mc/compose?to=Users@lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> -----Inline Attachment Follows-----
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <http://mc/compose?to=Users@lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20121017/ee8077cc/attachment-0001.htm>


More information about the Users mailing list