<div dir="ltr"><div>I built a similar functioning platform back in 2015 based on similar hardware
(Westmere Xeons, hyperthreading enabled)
running bare metal on Centos6. At some point we bumped it up to dual X5670s (cheap upgrade nowadays), but it was handling 12000 CPS peaks on 1 server with 3000-5000 CPS sustained for large parts of the day. I don't think you are too far off in hardware.<br></div><div>This was on version 1.9, so there was no Async. IIRC it was either 32 or 64 children. Async requires the TM module which adds additional overhead and memory allocation. <br></div><div>The LRN database was stored in mysql with a very simple table (TN, LRN) to keep memory usage down so that it could be pinned in memory (server had 48 or 72GB I think). MySQL was set to dump the innodb buffer cache to disk on restart so that the whole database would be back in memory on restart. Doing a full table scan would initially populate the MySQL cache.<br></div><div>Blacklists and other smaller datasets were stored in OpenSIPs using the userblacklist module.
There are better ways to do that in version 2 and onwards. Bigger list were stored in memcached. I prefer redis for this purpose now.<br></div><div><br></div><div>I would suggest simplifying testing by using a single MySQL server and bypassing the F5 to eliminate that as a source of connection problems or additional latencies.</div><div>In the OpenSIPs script, eliminate everything but 1 dip, probably just dip the LRN to start. <br></div><div>Performance test the stripped down scenario with sipp. Based on past experience, you should be able to hit or come close to your performance goal with only 1 dip in play. </div><div>
If you do hit your performance targets, keep adding more dips one by one until it breaks.
</div><div>If you can't reach your performance target with this stripped down scenario, then I'd suggest testing without the async and transactions enabled. I wouldn't think transactions would be a necessity in this scenario. I ran into CPS problems on that other open source SIP server when using async under heavy load. The transaction creation was chewing up CPU and memory. I'm not sure how different the implementation is here. </div><div>
I seem to start having problems with sipp when I hit a few thousand CPS due to it being single threaded. You probably will need to run multiple parallel sipp processes for your load test, if not already.
</div><div>If using an OS with systemd journald for logging, that will be a big bottleneck in of itself with even small amounts of logging.</div><div>In 1.9, I hacked together a module to create a timestamp string with ms for logging query latencies for diagnostic purposes. There may be a better out of the box way to do it now.</div><div>For children sizing, I would suggest benchmarking with at least 16 children and then doubling it to compare performance.<br></div><div>Watch the logs for out of memory or defragment messages and bump up shared memory or package memory if necessary. Package memory is probably going to be your problem, but it doesn't sound like it is a problem yet.</div><div><br></div><div>BR,<br></div><div>Jon Abrams<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 10, 2020 at 3:20 PM Calvin Ellison <<a href="mailto:calvin.ellison@voxox.com">calvin.ellison@voxox.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">We've checked our F5 BigIP configuration and added a second database server to the pool. Both DBs have been checked for max connections, open files, etc. Memcached has been moved to a dedicated server. Using a SIPp scenario for load testing from a separate host, things seem to fall apart on OpenSIPS around 3,000 CPS with every CPU core at or near 100% and no logs indicating fallback to sync/blocking mode. Both databases barely noticed the few hundred connections. Does this seem reasonable for a dual CPU server with 8 cores and 16 threads?<br><br><a href="https://ark.intel.com/content/www/us/en/ark/products/47925/intel-xeon-processor-e5620-12m-cache-2-40-ghz-5-86-gt-s-intel-qpi.html" target="_blank">https://ark.intel.com/content/www/us/en/ark/products/47925/intel-xeon-processor-e5620-12m-cache-2-40-ghz-5-86-gt-s-intel-qpi.html</a><br><br>What is the OpenSIPS opinion on Hyper-Threading?<br><br>Is there a way to estimate max CPS based on SPECrate, BogoMIPS, or some other metric?<br><br>I would love to know if my opensips.cfg has any mistakes, omissions, or inefficiencies. Is there a person or group who does sanity checks?<br><br>What should I be looking at within OpenSIPS during a load test to identify bottlenecks?<div><br></div><div>I'm still looking for guidance on the things below, especially children vs timer_partitions:<br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Is there an established method for fine-tuning these things?<br>shared memory<br>process memory<br>children<br>db_max_async_connections<br>listen=... use_children<br>modparam("tm", "timer_partitions", ?)</blockquote><br>What else is worth considering?<br><br>Regards,<br><br>Calvin Ellison<br>Senior Voice Operations Engineer<br><a href="mailto:calvin.ellison@voxox.com" target="_blank">calvin.ellison@voxox.com</a><br><br>On Thu, Jun 4, 2020 at 5:18 PM David Villasmil <<a href="mailto:david.villasmil.work@gmail.com" target="_blank">david.villasmil.work@gmail.com</a>> wrote:<br>><br>> Maybe you are hitting the max connections? How many connections are there when it starts to show those errors?</div></div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote></div>