[OpenSIPS-Users] Low Latency Message Queue Recommendations

[Brett Nemeroff]

Hello Calvin,
I personally would recommend something RabbitMQ based. Rabbit is a great
holding area that makes offloading the request from OpenSIPs a breeze.
Making parallized workers to push that into something else is also
pretty easy to whip up.

That being said, if you are prone to run out of memory due to regular
failures, you have a larger issue. Main problems work out like this:
1. Gotta get it out of OpenSIPs fast to prevent blocking calls
2. Send it to another server that isn't going to be delayed "storing it"
Likely requires some sort of memory store on that end
3. Final resting spot sometimes has interruptions
4. Memory capacity is going to be a problem.

Rabbit has some mechanisms to allow you to writeback to disk for
persistency. However, once you start writing to disk, the writes ultimately
slow down and it also breaks down.

One thing I've looked at to resolve these kinds of problems is to have a
sort of "pressure release" so that if the final destination isn't going to
be available for some time (or we've reached some metric threshold) that we
write the data somewhere else as a temp measure and allow async/background
syncing once everything else comes back healthy. I'm not sure there is
another way to do it.

Let us know what you think.
-Brett


On Mon, Nov 14, 2022 at 12:02 PM Calvin Ellison <calvin.ellison at voxox.com>
wrote:

> What are the recommended solutions for posting data to an external system
> with the lowest latency? Today we're using a Redis list to push data from
> OpenSIPS and pull it from Redis using Logstash. These are call detail
> records that need to reach our OpenSearch index with minimum delay.
>
> We have Redis running in memory-only mode to reduce latency, largely
> because the underlying storage is a very busy SAN. The current pipeline
> works until there is a disruption to Logstash or OpenSearch. If the
> disruption is not corrected quickly enough, Redis runs out of memory and we
> lose data. If Redis crashes, OpenSIPS itself breaks down.
>
> Is there a more robust solution for getting things to OpenSearch? Latency
> is the goal, but we need to tolerate disruption without data loss.
>
>
>
> Calvin Ellison
>
> Systems Architect
>
> calvin.ellison at voxox.com
>
> +1 (213) 285-0555
>
> <http://voxox.com>
>
> <https://www.facebook.com/VOXOX/>
> <https://www.instagram.com/voxoxofficial/>
> <https://www.linkedin.com/company/3573541/admin/>
> <https://twitter.com/Voxox>
>
> The information contained herein is confidential and privileged
> information or work product intended only for the individual or entity to
> whom it is addressed. Any unauthorized use, distribution, or copying of
> this communication is strictly prohibited. If you have received this
> communication in error, please notify me immediately.
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20221114/d674e21c/attachment-0001.html>