[OpenSIPS-Users] media-relay error - operation not permitted

[Jeff Pyle]

Hi Saúl,

On Fri, Feb 8, 2013 at 3:24 AM, Saúl Ibarra Corretgé
<saul at ag-projects.com>wrote:

> Hi Jeff,
>
> On Feb 7, 2013, at 10:52 PM, Jeff Pyle wrote:
>
> > Hello,
> >
> > On a media relay from Mediaproxy 2.4.4, we've noticed a lot of errors
> like the following today:
> >
> > media-relay[10351]: Traceback (most recent call last):
> > media-relay[10351]:   File
> "/usr/lib/python2.5/site-packages/twisted/internet/udp.py", line 126, in
> doRead
> > media-relay[10351]:     self.protocol.datagramReceived(data, addr)
> > media-relay[10351]:   File
> "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 130, in
> datagramReceived
> > media-relay[10351]:     self.cb_func(host, port, data)
> > media-relay[10351]:   File
> "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 226, in
> got_data
> > media-relay[10351]:     self.substream.send_data(self, data, is_stun)
> > media-relay[10351]:   File
> "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 311, in
> send_data
> > media-relay[10351]:     dest.listener.protocol.send(data, is_stun)
> > media-relay[10351]:   File
> "/usr/lib/pymodules/python2.5/mediaproxy/mediacontrol.py", line 159, in send
> > media-relay[10351]:     self.transport.write(data, (ip, port))
> > media-relay[10351]:   File
> "/usr/lib/python2.5/site-packages/twisted/internet/udp.py", line 155, in
> write
> > media-relay[10351]:     return self.socket.sendto(datagram, addr)
> > media-relay[10351]: error: (1, 'Operation not permitted')
> >
> > Is there a limit within the OS that might cause this if media sessions
> exceeded a certain threshold?  It seems to happen around 500 calls.
> >
>
> I have never seen this. The error you are getting is EPERM, which is kind
> of odd for UDP. Do you have any firewall on that machine?
>

Just iptables.  The OUTPUT chain's policy on all tables is ACCEPT, and we
have no OUTPUT rules (only INPUT).  OUTPUT is wide open.

We did find this in /usr/share/pyshared/twisted/internet/process.py:

   601          try:
   602              import resource
   603              maxfds = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
+ 1
   604              # OS-X reports 9223372036854775808. That's a lot of fds
to close
   605              if maxfds > 1024:
   606                  maxfds = 1024
   607          except:
   608              maxfds = 256

The 1024 was interesting because we seemed to be having the problem around
500 calls, possibly 512?  This is from python-twisted 8.1.0.  We tried to
adjust line 606 to maxfds = 4096, and make what seemed like appropriate
updates in /etc/security/limits.conf:

root soft nofile 1024
root hard nofile 16384         # was 1024

This is all new to us so it's very possible we're on the wrong track.

Today we split the traffic between two relays.  Each handles around 425
calls at peak usage.  We made these twisted+limits updates on one relay
only.  The quantity of the Traceback errors is lower at 22 so far today on
one relay, and 3 on the other.  The first apparence was with less than 100
calls on the relay.  The two relay machines have the same configurations,
except for IP address.  I don't have an explanation why they are higher on
one relay than the other.


- Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20130208/3b2486d3/attachment-0001.htm>