[OpenSIPS-Devel] [opensips] do not always retry write to pipe if we get blocking errors (#519)

[Eric Tamme]

If you have an rabbitmq event subscription the event_rabbitmq module will shm_alloc rmq events and write pointers of the structs to the pipe.

In the event that the node you have connected to goes down, the pipe will start to fill and when it reaches its max capacity (65535 bytes) or approximately 2700 events based on the 8 byte pointer, the write call with return EAGAIN causing the while loop to become an infinite loop until pointers start getting pulled off the pipe.  This causes massive CPU consumption, as well as blocking any process that generates an event to event_rabbitmq.

Attempts to publish to rabbit MQ timeout after 3 minutes based on a default system tcp timeout, so only one event every 3 minutes will be pulled from the pipe while the amqp node is down.

You can recreate this issue by setting up a proxy with an event_rabbitmq subscription, adding an iptables rule to block access to the amqp node and send traffic to the proxy that would generate an event till you hit the max pipe size ~2703 events pointers.

This commit changes the logic to simply retry the write 3 times, then abort.


You can view, comment on, or merge this pull request online at:

  https://github.com/OpenSIPS/opensips/pull/519

-- Commit Summary --

  * do not always retry write to pipe if we get blocking errors as this causes the proxy to lock up and consume cpu

-- File Changes --

    M modules/event_rabbitmq/rabbitmq_send.c (3)

-- Patch Links --

https://github.com/OpenSIPS/opensips/pull/519.patch
https://github.com/OpenSIPS/opensips/pull/519.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/pull/519
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20150521/b5811e5c/attachment.htm>