<div dir="ltr">HI Saul,<div><br></div><div>The ports in use are 16384-32768. That provides for ~8000 session, yes? We were nowhere near that many calls. So, it's much more likely that ports were not being released for some reason.</div>
<div><br></div><div>There is a limited amount of T.38 traffic through these relays. The majority of the T.38-speaking devices are Adtran TA900-series, which, in general, behave quite well. That's on our side. On the partner side, usually Sonus GSX 9000-series. I bet it's the unusual client that is the problem.</div>
<div><br></div><div>A SIP capture isn't practical here. :( I'll try to correlate the exceptions with CDR records to determine if there is a particular trunk group on my network that needs its attitude adjusted.</div>
<div><br></div><div>Until we can update the entire call flow (Opensips, Mediaproxy, all deps, ...), or find a definitive misbehaving client, I'll restart the relays on a schedule. If I am able to reproduce the exceptions I'll provide debug data.</div>
<div><br></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div>- Jeff</div><div>
<br></div></div></div>
<br><br><div class="gmail_quote">On Wed, Jul 23, 2014 at 4:01 AM, Saúl Ibarra Corretgé <span dir="ltr"><<a href="mailto:saul@ag-projects.com" target="_blank">saul@ag-projects.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Jeff,<br>
<div class=""><br>
On 21 Jul 2014, at 21:55, Jeff Pyle <<a href="mailto:jpyle@fidelityvoice.com">jpyle@fidelityvoice.com</a>> wrote:<br>
<br>
> Hello,<br>
><br>
> This is on Opensips 1.6 with Mediaproxy 2.4.4. Yeah, they're old. I know.<br>
><br>
> We see this from time to time:<br>
><br>
> media-relay[10719]: Traceback (most recent call last):<br>
> media-relay[10719]: File "/usr/lib/python2.5/site-packages/twisted/internet/udp.py", line 126, in doRead<br>
> media-relay[10719]: self.protocol.datagramReceived(data, addr)<br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 130, in datagramReceived<br>
> media-relay[10719]: self.cb_func(host, port, data)<br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 226, in got_data<br>
> media-relay[10719]: self.substream.send_data(self, data, is_stun)<br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 311, in send_data<br>
> media-relay[10719]: dest.listener.protocol.send(data, is_stun)<br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 159, in send<br>
> media-relay[10719]: self.transport.write(data, (ip, port))<br>
> media-relay[10719]: File "/usr/lib/python2.5/site-packages/twisted/internet/udp.py", line 155, in write<br>
> media-relay[10719]: return self.socket.sendto(datagram, addr)<br>
> media-relay[10719]: error: (1, 'Operation not permitted')<br>
><br>
> There doesn't seem to be any pattern. Nor do there seem to be any complaints.<br>
><br>
> Today we had it happen about 10 times, far more than the logs indicate is normal. Then we had many lines of this:<br>
><br>
> media-relay[10719]: error: Could not reserve relay ports for session, all allocated ports are being used<br>
><br>
<br>
</div>What port range are you using? Any chance they were all exhausted?<br>
<div class=""><br>
> Then a few instances of:<br>
><br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/relay.py", line 175, in lineReceived<br>
> media-relay[10719]: response = self.factory.parent.got_command(self.factory.host, self.command, self.headers)<br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/relay.py", line 387, in got_command<br>
> media-relay[10719]: local_media = self.session_manager.update_session(dispatcher, **headers)<br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 754, in update_session<br>
> media-relay[10719]: session.update_media(cseq, to_tag, user_agent, media, is_downstream, is_caller_cseq)<br>
> media-relay[10719]: File "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 566, in update_media<br>
> media-relay[10719]: raise ValueError('Media types do not match: "%s" and "%s"' % (stream.media_type, media_type))<br>
</div>> media-relay[10719]: ValueError: Media types do not match: "audio" and “image”<br>
><br>
<br>
Hum, this could be a bug. Maybe some weird fax device? A SIP trace of such calls would be nice.<br>
<div class=""><br>
> Followed by lots (and LOTS) of these with various port combinations:<br>
><br>
> media-relay[10719]: warning: Cannot use port pair 28836/28837<br>
><br>
<br>
</div>It’s possible that due to the previous exception ports are not deallocated so they kept piling up until the range was exhausted.<br>
<div class=""><br>
> This happened on two relays at the same time. The dispatchers lost connectivity with the relays, and I had to kill -9 the relays to shake them loose. Upon a relay restart all seems normal.<br>
><br>
> Even though old, this media relay configuration has been rock solid for years. Today, not so much. I'm wondering if this is a known bug that hasn't bitten us until today? Or, something else?<br>
><br>
> They are scheduled for replacement with more current software, but until then, I'd like to learn what I can.<br>
><br>
<br>
</div>I browsed through the MediaProxy changelogs just in case memory was not serving well, and I don’t see anything related to this problem. At a first glance it looks like an issue potentially caused by a device that can do a=image, usually a fax. We do have some workarounds inplace for misbehaving fax devices, maybe there is one case we miss.<br>
<br>
<br>
Regards,<br>
<br>
--<br>
Saúl Ibarra Corretgé<br>
AG Projects<br>
<br>
<br>
<br>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div>