[OpenSIPS-Devel] [ opensips-Bugs-3611677 ] Offiline RTP Proxy causes system freeze

[SourceForge.net]

Bugs item #3611677, was opened at 2013-04-23 14:10
Message generated for change (Comment added) made by digipigeon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3611677&group_id=232389

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: modules
Group: 1.9.x
Status: Open
Resolution: None
>Priority: 7
Private: No
Submitted By: Digipigeon (digipigeon)
Assigned to: Razvan Crainea (razvancrainea)
Summary: Offiline RTP Proxy causes system freeze

Initial Comment:
I have a setup running multiple RTP Proxy instances, when multiple rtp proxies fail, CPU utilisation approaches 100%, this does not usually happen with 1 or 2 failures, but when 4+ fail (this can even be 4 instances on an 8 core/instance machine), the issues arrise.

The severity of this event happening has caused the system to stop responding for short periods, observations notice intermittent periods of no logging information and substantial packet drops (due to high CPU usage).

Once RTP Proxy instances that have failed come back online, CPU utilisation drops right down and everything functions as expected.

----------------------------------------------------------------------

>Comment By: Digipigeon (digipigeon)
Date: 2013-05-02 03:57

Message:
I have updated my code in an attempt to circumvent the issue to include:
modparam("rtpproxy", "rtpproxy_disable_tout", -1)

My observations on the above modification are as follows:
1. CPU regularly runs ah a higher value than before modification.
2. The CPU still reaches 100% utilisation which then causes kernal packet
drops (SIP, just because of high load).
3. The 100% CPU usage (through my limited observations) does not appear
last as long in locked up state.

Further to this I can observe that multiple RTP servers fail successively,
however the chance that 5 RTP servers go offline at the time is unlikely,
however I am considering that the successive fails of multiple rtp servers
could be a effect of the high CPU utilisation then causing packet drops
rather than multiple.

However in addition this, I have setup a ICMP ping as well as a V ping via
UDP in the control channel, both with return negligible packet loss.

----------------------------------------------------------------------

Comment By: Digipigeon (digipigeon)
Date: 2013-04-24 10:40

Message:
Well this happened sooner than I was expecting, I managed to run the
commands on the processes that were taking up the CPU usage, however I cant
gurentee that I was able to run the script before the process completed its
high usage.

Please see files, Dump 1 and Dump 2



----------------------------------------------------------------------

Comment By: Digipigeon (digipigeon)
Date: 2013-04-24 03:40

Message:
Hi Răzvan,

Thank you, I will make a note of this, however it is my aim not to let my
system enter this state again and I have only been able to replicate with
live traffic, so I am not sure when I will have the opportunity to run this
again. I will keep you updated though.

Kind Regards Jonathan

----------------------------------------------------------------------

Comment By: Razvan Crainea (razvancrainea)
Date: 2013-04-24 03:31

Message:
Hi!

When opensips is in 100% CPU usage, can you run the following command:

gdb $OPENSIPS $PID -batch --eval-command="bt full"

Note you have to replace $OPENSIPS with OpenSIPS' executable path and $PID
with the value of the pid that holds the CPU in 100%.

Best regards,
Răzvan

----------------------------------------------------------------------

Comment By: Digipigeon (digipigeon)
Date: 2013-04-24 03:25

Message:
Hello Răzvan,

I would make the same assumption, which is expected, however I don't know
why this would lead to 100% CPU usage, surely probing only involved sending
and waiting for a udp packet.

I am using 32 processes on an 8 core machine.
I am not using rtpproxy_disable_tout parameter.
I have not tried to disable via MI command.

Regards Jonathan

----------------------------------------------------------------------

Comment By: Razvan Crainea (razvancrainea)
Date: 2013-04-24 00:41

Message:
Hello!

My assumption is that OpenSIPS is trying to probe the other RTPProxy
servers without any success. That's probably why the CPU goes that high.
How many OpenSIPS processes are you using? Are you using the
rtpproxy_disable_tout [1] parameter? If yes, what value are you setting?
Have you tried to permanently disable via MI commands [2 ]the RTPProxy
servers that failed?

[1] http://www.opensips.org/html/docs/modules/1.9.x/rtpproxy#id250074
[2] http://www.opensips.org/html/docs/modules/1.9.x/rtpproxy#id293384

Best regards,
Răzvan

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3611677&group_id=232389