
<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Perhaps your virtual machine simply cannot handle the load. The commands to close sessions may also be dropped or lost under such environment.<div class=""><br class=""></div><div class="">Adrian<br class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 16 Mar 2017, at 11:22, Daniel Zanutti <<a href="mailto:daniel.zanutti@gmail.com" class="">daniel.zanutti@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Dan<div class=""><br class=""></div><div class="">Looks like this problem is only happening on virtual machines, not on physical machines. And only while they are on high load. </div><div class=""><br class=""></div><div class="">But i'm not sure about the kernel rule, is there any way to check it? </div><div class=""><br class=""></div><div class="">Please take a look at this case, this Relay will never halt because there are more than 3k sessions that will never finish internally (the call has already hangup hours ago):</div><div class=""><br class=""></div><div class=""><table class="gmail-table_test" style="font-family: 'open sans', arial, helvetica, sans-serif; font-size: 12px; margin-top: 0px; margin-left: 0px; margin-right: 0px; width: 1893px; background-color: rgb(235, 239, 242); padding: 0px 5px; background-position: initial initial; background-repeat: initial initial;"><tbody class=""><tr class="gmail-cinza2" align="right" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(242,242,242);color:rgb(102,102,102);font-family:verdana;font-size:10px"><td class="">8</td><td class="">2.2.2.2</td><td class="">2.6.1</td><td class="">44h01'05"<br class=""></td><td class="">112.03kbps</td><td class="">3045</td><td valign="top" class=""><br class="">audio 3045</td><td class="">Halting</td></tr></tbody></table></div><div class=""><br class=""></div><div class="">Some of these calls:</div><div class=""><br class=""></div><div class=""><table class="gmail-table_test" style="font-family: 'open sans', arial, helvetica, sans-serif; font-size: 12px; margin-top: 0px; margin-left: 0px; margin-right: 0px; width: 1893px; background-color: rgb(235, 239, 242); padding: 0px 5px; background-position: initial initial; background-repeat: initial initial;"><tbody class=""><tr valign="top" class="gmail-cinza1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(230,230,230);color:rgb(102,102,102);font-family:verdana;font-size:10px"><td class="gmail-border" rowspan="1" align="center"></td><td class="gmail-border" rowspan="1" align="center"><br class=""></td><td class="gmail-border" align="left"><br class=""></td><td class="gmail-border" align="left"><br class=""></td><td class="gmail-border" align="left"><br class=""></td><td class="gmail-border" align="left"><br class=""></td><td class="gmail-border" align="center"><br class=""></td><td class="gmail-border" align="center"><br class=""></td><td class="gmail-border" align="center"><br class=""></td><td class="gmail-border" align="right"><br class=""></td><td class="gmail-border" align="right"><br class=""></td><td class="gmail-border" align="right"><br class=""></td></tr><tr valign="top" class="gmail-cinza2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(242,242,242);color:rgb(102,102,102);font-family:verdana;font-size:10px"><td class="gmail-border" rowspan="1">728</td><td class="gmail-border" rowspan="1"><span class=""><b class="">From:</b> 222222@</span>4.4.4.4<br class=""><span class=""><b class="">To:</b> <a href="mailto:33333333@sip.aaa.com.br" class="">33333333@sip.aaa.com.br</a></span><br class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="unknown agent" title="unknown agent" border="0" class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="HG4000/1.0" title="HG4000/1.0" border="0" class=""></td><td class="gmail-border" align="left"><a href="http://6.6.6.6:55632/" class="">6.6.6.6:55632</a></td><td class="gmail-border" align="left"><a href="http://2.2.2.2:46640/" class="">2.2.2.2:46640</a></td><td class="gmail-border" align="left"><a href="http://2.2.2.2:46866/" class="">2.2.2.2:46866</a></td><td class="gmail-border" align="left"><a href="http://7.7.7.7:4170/" class="">7.7.7.7:4170</a></td><td class="gmail-border" align="center"><span class="">active</span></td><td class="gmail-border" align="center">G729</td><td class="gmail-border" align="center">audio</td><td class="gmail-border" align="right">21h35'34"</td><td class="gmail-border" align="right">0</td><td class="gmail-border" align="right">0</td></tr><tr valign="top" class="gmail-cinza1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(230,230,230);color:rgb(102,102,102);font-family:verdana;font-size:10px"><td class="gmail-border" rowspan="1">729</td><td class="gmail-border" rowspan="1"><span class=""><b class="">From:</b> 2222222@</span>4.4.4.4<span class="">:5064</span><br class=""><span class=""><b class="">To:</b> 33333333@</span><a href="http://sip.aaa.com.br/" class="">sip.aaa.com.br</a><br class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="TS-v4.6.0-11eW" title="TS-v4.6.0-11eW" border="0" class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="Agitel GSM Bridge v2.0" title="Agitel GSM Bridge v2.0" border="0" class=""></td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">6.6.6.6</span>:34908</td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">2.2.2.2</span>:58158</td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">2.2.2.2</span>:54372</td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">7.7.7.7</span>:16846</td><td class="gmail-border" align="center"><span class="">active</span></td><td class="gmail-border" align="center">G729</td><td class="gmail-border" align="center">audio</td><td class="gmail-border" align="right">16h11'51"</td><td class="gmail-border" align="right">0</td><td class="gmail-border" align="right">0</td></tr><tr valign="top" class="gmail-cinza2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(242,242,242);color:rgb(102,102,102);font-family:verdana;font-size:10px"><td class="gmail-border" rowspan="1">730</td><td class="gmail-border" rowspan="1"><span class=""><b class="">From:</b> 22222222@</span>4.4.4.4<br class=""><span class=""><b class="">To:</b> 33333333@</span><a href="http://sip.aaa.com.br/" class="">sip.aaa.com.br</a><br class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="Mediant 2000/v.6.60A.328.003" title="Mediant 2000/v.6.60A.328.003" border="0" class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="unknown agent" title="unknown agent" border="0" class=""></td><td class="gmail-border" align="left"><a href="http://6.6.6.6:46324/" class="">6.6.6.6:46324</a></td><td class="gmail-border" align="left"><a href="http://2.2.2.2:50156/" class="">2.2.2.2:50156</a></td><td class="gmail-border" align="left"><a href="http://2.2.2.2:48182/" class="">2.2.2.2:48182</a></td><td class="gmail-border" align="left"><a href="http://7.7.7.7:18516/" class="">7.7.7.7:18516</a></td><td class="gmail-border" align="center"><span class="">active</span></td><td class="gmail-border" align="center">G729</td><td class="gmail-border" align="center">audio</td><td class="gmail-border" align="right">19h45'38"</td><td class="gmail-border" align="right">0</td><td class="gmail-border" align="right">0</td></tr><tr valign="top" class="gmail-cinza1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(230,230,230);color:rgb(102,102,102);font-family:verdana;font-size:10px"><td class="gmail-border" rowspan="1">731</td><td class="gmail-border" rowspan="1"><span class=""><b class="">From:</b> 222222@</span>4.4.4.4<span class="">:5061</span><br class=""><span class=""><b class="">To:</b> 33333333333@</span><a href="http://sip.aaa.com.br/" class="">sip.aaa.com.br</a><br class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="TS-v4.6.0-14b" title="TS-v4.6.0-14b" border="0" class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="gsm-gw-3.4.1" title="gsm-gw-3.4.1" border="0" class=""></td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">6.6.6.6</span>:54800</td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">2.2.2.2</span>:43998</td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">2.2.2.2</span>:46144</td><td class="gmail-border" align="left"><span style="background-color:rgb(242,242,242)" class="">7.7.7.7</span>:12360</td><td class="gmail-border" align="center"><span class="">active</span></td><td class="gmail-border" align="center">G729</td><td class="gmail-border" align="center">audio</td><td class="gmail-border" align="right">19h09'41"</td><td class="gmail-border" align="right">0</td><td class="gmail-border" align="right">0</td></tr><tr valign="top" class="gmail-cinza2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(242,242,242);color:rgb(102,102,102);font-family:verdana;font-size:10px"><td class="gmail-border" rowspan="1">732</td><td class="gmail-border" rowspan="1"><span class=""><b class="">From:</b> <a href="mailto:2222222@4.4.4.4" class="">2222222@4.4.4.4</a></span><br class=""><span class=""><b class="">To:</b> 333333333333@</span><a href="http://sip.aaa.com.br/" class="">sip.aaa.com.br</a><br class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="Trinit IVR" title="Trinit IVR" border="0" class=""></td><td class="gmail-border" rowspan="1" align="center"><img src="http://agitel.dazsoft.com/painel/cdr/images//30/unknown.png" alt="HG4000/1.0" title="HG4000/1.0" border="0" class=""></td><td class="gmail-border" align="left"><a href="http://6.6.6.6:18854/" class="">6.6.6.6:18854</a></td><td class="gmail-border" align="left"><a href="http://2.2.2.2:51924/" class="">2.2.2.2:51924</a></td><td class="gmail-border" align="left"><a href="http://2.2.2.2:40512/" class="">2.2.2.2:40512</a></td><td class="gmail-border" align="left"><a href="http://7.7.7.7:4200/" class="">7.7.7.7:4200</a></td><td class="gmail-border" align="center"><span class="">active</span></td><td class="gmail-border" align="center">G729</td><td class="gmail-border" align="center">audio</td><td class="gmail-border" align="right">19h37'59"</td><td class="gmail-border" align="right">0</td><td class="gmail-border" align="right">0</td></tr></tbody></table></div><div class=""><br class=""></div><div class="">Is there any way to drop these sessions? Maybe using the internal timeout system of mediaproxy?</div><div class=""><br class=""></div><div class="">If you could take a look personally, we could negotiate an hourly rate.</div><div class=""><br class=""></div><div class="">Thanks again</div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Mar 16, 2017 at 10:54 AM, Dan Pascu <span dir="ltr" class=""><<a href="mailto:dan@ag-projects.com" target="_blank" class="">dan@ag-projects.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="">

One thing came to mind. A case when the relay could get overloaded is if a lot of clients start sessions and only one endpoint sends media. That is the only case where the relay would have to deal with the media traffic itself and having hundreds of such sessions at the same time could overload the relay.<br class="">

<br class="">

The way the relay works is for each call it starts listening on 4 ports (2 for RTP and 2 for RTCP). Each endpoint will send 2 streams (1 RTP one RTCP) and initially the relay will just listen on these ports and when it receives data it learns the endpoint's address. After it learns both endpoint's addresses, it adds a conntrack rule in the kernel to allow the kernel to directly relay the media streams between the endpoints and it will never see a media packet from the endpoints again until the call ends. This allows for very efficient data forwarding because it's done entirely in the kernel with no data being transferred from kernel to user-space and back like traditional solutions. We have seen media relays handling hundreds of calls at a time with 0% CPU load on the relay.<br class="">

<br class="">

So the only thing I can think of causing something like what you describe (even though I'm still not sure what you meant by hanging up sessions), is that somehow this process didn't finish setting up completely and the relay directly receives media streams from hundreds of devices because only one endpoint sends data (or the other endpoint's data gets filtered at some firewall), and because it cannot learn both endpoint's addresses it cannot setup the kernel conntrack rule to move data forwarding to the kernel.<br class="">

<div class="HOEnZb"><div class="h5"><br class="">

On 14 Mar 2017, at 13:38, Dan Pascu wrote:<br class="">

<br class="">

><br class="">

> On 13 Mar 2017, at 18:58, Daniel Zanutti wrote:<br class="">

><br class="">

>> Hi guys<br class="">

>><br class="">

>> I sent this email a few days ago, anyone from Mediaproxy team could take a look? I could debug it, just need some directions on where to look.<br class="">

><br class="">

> We have never encountered this problem, so I', not sure what to suggest, even more considering that the description is not very clear. What do you mean when you say the relay starts to hang some sessions? That it timeouts on them not having traffic and initiates a BYE for those sessions? Because in the next paragraph you imply that they never timeout.<br class="">

><br class="">

>><br class="">

>> Thanks<br class="">

>><br class="">

>> On Tue, Mar 7, 2017 at 11:10 AM, Daniel Zanutti <<a href="mailto:daniel.zanutti@gmail.com" class="">daniel.zanutti@gmail.com</a>> wrote:<br class="">

>> I'm using mediaproxy on several installations and have noticed that when the machine is on high load (> 700 sessions), the media-relay process starts to hang some sessions.<br class="">

>><br class="">

>> These sessions doesn't have any RTP being sent/received anymore and they never hangup. After some hours of frozen sessions, the media-relay process doesn't connect to the dispatcher anymore, but keep using high CPU on the machine. Maybe it's on loop internally, not sure.<br class="">

>><br class="">

>> Is there any solution for this? Maybe a timer to cleanup old sessions (2 or 4+ hours old).<br class="">

>><br class="">

>> Thanks<br class="">

>><br class="">

>><br class="">

>><br class="">

>> ______________________________<wbr class="">_________________<br class="">

>> Users mailing list<br class="">

>> <a href="mailto:Users@lists.opensips.org" class="">Users@lists.opensips.org</a><br class="">

>> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.opensips.org/cgi-<wbr class="">bin/mailman/listinfo/users</a><br class="">

><br class="">

><br class="">

> --<br class="">

> Dan<br class="">

><br class="">

><br class="">

><br class="">

><br class="">

><br class="">

> ______________________________<wbr class="">_________________<br class="">

> Users mailing list<br class="">

> <a href="mailto:Users@lists.opensips.org" class="">Users@lists.opensips.org</a><br class="">

> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.opensips.org/cgi-<wbr class="">bin/mailman/listinfo/users</a><br class="">

<br class="">

<br class="">

--<br class="">

Dan<br class="">

<br class="">

<br class="">

<br class="">

<br class="">

<br class="">

______________________________<wbr class="">_________________<br class="">

Users mailing list<br class="">

<a href="mailto:Users@lists.opensips.org" class="">Users@lists.opensips.org</a><br class="">

<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.opensips.org/cgi-<wbr class="">bin/mailman/listinfo/users</a><br class="">

</div></div></blockquote></div><br class=""></div>

_______________________________________________<br class="">Users mailing list<br class=""><a href="mailto:Users@lists.opensips.org" class="">Users@lists.opensips.org</a><br class="">http://lists.opensips.org/cgi-bin/mailman/listinfo/users<br class=""></div></blockquote></div><br class=""></div></div></div></body></html>