<div dir="ltr">Hello Calvin,<div>I personally would recommend something RabbitMQ based. Rabbit is a great holding area that makes offloading the request from OpenSIPs a breeze. Making parallized workers to push that into something else is also pretty easy to whip up.</div><div><br></div><div>That being said, if you are prone to run out of memory due to regular failures, you have a larger issue. Main problems work out like this: </div><div>1. Gotta get it out of OpenSIPs fast to prevent blocking calls</div><div>2. Send it to another server that isn't going to be delayed "storing it" Likely requires some sort of memory store on that end</div><div>3. Final resting spot sometimes has interruptions</div><div>4. Memory capacity is going to be a problem. </div><div><br></div><div>Rabbit has some mechanisms to allow you to writeback to disk for persistency. However, once you start writing to disk, the writes ultimately slow down and it also breaks down. </div><div><br></div><div>One thing I've looked at to resolve these kinds of problems is to have a sort of "pressure release" so that if the final destination isn't going to be available for some time (or we've reached some metric threshold) that we write the data somewhere else as a temp measure and allow async/background syncing once everything else comes back healthy. I'm not sure there is another way to do it. </div><div><br></div><div>Let us know what you think.</div><div>-Brett</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 14, 2022 at 12:02 PM Calvin Ellison <<a href="mailto:calvin.ellison@voxox.com">calvin.ellison@voxox.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">What are the recommended solutions for posting data to an external system with the lowest latency? Today we're using a Redis list to push data from OpenSIPS and pull it from Redis using Logstash. These are call detail records that need to reach our OpenSearch index with minimum delay.<div><br></div><div>We have Redis running in memory-only mode to reduce latency, largely because the underlying storage is a very busy SAN. The current pipeline works until there is a disruption to Logstash or OpenSearch. If the disruption is not corrected quickly enough, Redis runs out of memory and we lose data. If Redis crashes, OpenSIPS itself breaks down.</div><div><br></div><div>Is there a more robust solution for getting things to OpenSearch? Latency is the goal, but we need to tolerate disruption without data loss.<br clear="all"><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Proxima Nova",sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:800;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Proxima Nova",sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:800;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Proxima Nova",sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:800;vertical-align:baseline;white-space:pre-wrap">Calvin Ellison</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Proxima Nova",sans-serif;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Systems Architect</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Proxima Nova",sans-serif;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="mailto:calvin.ellison@voxox.com" target="_blank">calvin.ellison@voxox.com</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Proxima Nova",sans-serif;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">+1 (213) 285-0555</span><span style="font-size:2.25pt;font-family:"Proxima Nova",sans-serif;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br><br></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><a href="http://voxox.com" target="_blank"><span style="font-size:10pt;font-family:"Proxima Nova",sans-serif;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span style="border:none;display:inline-block;overflow:hidden;width:139px;height:53px"><img src="https://lh5.googleusercontent.com/IpUfF3j5Y4T9UmEfLISsHXhgSenc131vheivxwdo6oyzyT1i5KfVnK8cftA1Qx-APliNpMTLx5PUOZ6k1lXHgMZvDXTXKkOWDpmXMlUcmsoKBZQIE_wip4sS8XMtdryakordMboW" width="139" height="53" style="margin-left: 0px; margin-top: 0px;"></span></span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><a href="https://www.facebook.com/VOXOX/" target="_blank"><span style="font-size:12pt;font-family:"Proxima Nova",sans-serif;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span style="border:none;display:inline-block;overflow:hidden;width:27px;height:27px"><img src="https://lh6.googleusercontent.com/qS_cwPUkKDN7UEAMffT5cjE1qN41OWxwS8HKu2DZi8R9OfYZCrdVxxPsyJy_SOaTDjWHjPAsBAzGnrgiSgrqDFWQKoi1HAdYUldZOtUQY_2EH0FC7M67fPko4xEmw7yfwiIpnRIF" width="27" height="27" style="margin-left: 0px; margin-top: 0px;"></span></span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><a href="https://www.instagram.com/voxoxofficial/" target="_blank"><span style="font-size:12pt;font-family:"Proxima Nova",sans-serif;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span style="border:none;display:inline-block;overflow:hidden;width:26px;height:26px"><img src="https://lh3.googleusercontent.com/AFGDGW3w1Yco24kAphoI6UZW170z44YjZpLZ0rHGRfCDNyctuD4DkIYooDbxEoJxrv0NQ4TswSWA4k0FPOD6w5v_ZFadQmeoXHhP1rgeZXdoLV7n-1PGxYhi31pbte1GHYwURny8" width="26" height="26" style="margin-left: 0px; margin-top: 0px;"></span></span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><a href="https://www.linkedin.com/company/3573541/admin/" target="_blank"><span style="font-size:12pt;font-family:"Proxima Nova",sans-serif;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span style="border:none;display:inline-block;overflow:hidden;width:26px;height:26px"><img src="https://lh6.googleusercontent.com/skcophrAnEg1jyCTXau3hMbh0GF9b1TeZ-SdROEZ0OxxHuTRKGNTS_lYIh2LIdgJ6rLTL1LMUVOlhxrHm7hxZ6p8XfYovu7xd-tXhvU1OEm4Hi6niWF6Trr7EfMFqGoLNMMeyYqk" width="26" height="26" style="margin-left: 0px; margin-top: 0px;"></span></span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><a href="https://twitter.com/Voxox" target="_blank"><span style="font-size:12pt;font-family:"Proxima Nova",sans-serif;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span style="border:none;display:inline-block;overflow:hidden;width:26px;height:26px"><img src="https://lh6.googleusercontent.com/-IAXOgfWybf32YI5OpWoYsxhr1Iq2lvdT8a7LDpiP9IIXDxWemvF2v-dQrQC7ZPDn6eQtNQav84fAm6w39EUMfUkEq9e-mebAWCw5F1g57C3UWrnxcLoJmnPxTsSec0vI-biI8J7" width="26" height="26" style="margin-left: 0px; margin-top: 0px;"></span></span></a></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:9pt;font-family:"Helvetica Neue",sans-serif;color:rgb(153,153,153);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">The information contained herein is confidential and privileged information or work product intended only for the individual or entity to whom it is addressed. Any unauthorized use, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify me immediately.</span></p></span></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote></div>