[OpenSIPS-Users] Fine tuning high CPS and msyql queries

[Alex Balashov]

Perhaps a simpler way look at it: buffers. It's in the name - they 
buffer things.

If your "work queue" buffer (which is what the packet RecvQ 
fundamentally is) is chronically full, to the degree that it needs 
increased significantly, it means things aren't consistently leaving out 
at the other end at the same velocity that they're coming in.

In small amounts, this regulating capacity is acceptable and 
necessary--that's why buffers exist.

But increasing the depth of the queue by 78x (if I'm not mistaken, 
212992 is the default--at least, it is on all my CentOS 7.x and 8.x 
systems, which I guess also have "absolutely terrible sysctl defaults") 
is faker than a Ponzi scheme. In some other contexts, this would be 
called morally bankrupt and intellectually fraudulent. I guess here we 
call it "mad dialer CPS" or whatever.

-- Alex

On 6/12/20 6:50 PM, Alex Balashov wrote:
> There's no free lunch, but it seems like you and others want one. :-) 
> Increasing these values just increases the depth of the kernel's packet 
> queue for the sync processes to consume as able. It doesn't mean they're 
> able, and accordingly, request response time will go up.
> 
> A healthy system that is able to keep up with the load you're throwing 
> at it should show a receive queue at +/- 0 most of the time, maybe with 
> some ephemeral spikes but generally trending around 0. If packets are 
> stacking up in the RecvQ, it means the SIP worker processes aren't 
> available enough to consume them all in a timely fashion.
> 
> Leaning on async won't help here if the workload is largely CPU-bound. 
> If it's largely bound over waiting on network I/O from external 
> services, it merely deputises the problem of notifying you when there's 
> a response from those services to the kernel. But - vitally - it won't 
> get your requests processed faster, and setup latency is a very 
> important consideration in real-time communications, especially from the 
> perspective of interoperability with the synchronous/circuit-switched PSTN.
> 
> In short, async isn't magic, and neither is increasing the receive 
> queue. It's simple thermodynamics; there's only so much CPU available, 
> and depending on the nature of the workload, throughput becomes more a 
> linear function of available CPU hardware threads, or less, but slower, 
> if it's largely I/O-bound.
> 
> The metaphor of a balloon is appropriate. You're pushing the problem 
> around by squeezing one part of the balloon, causing another to enlarge. 
> Various parts of the balloon can be squeezed - async vs. sync, various 
> queues and buffers, etc. But the internal volume of air held by the 
> balloon is more or less the same. A little slack can be added into the 
> system through your rmem_max technique, as long as you're willing to 
> tolerate increased processing latency--and it will generate increased 
> latency; if it didn't, you wouldn't need to increase it--but ultimately, 
> you're just pushing the air around the balloon. A fixed amount of CPU 
> and memory is available to accommodate the large number of processes 
> that sleep on an external I/O-bound workload, and there are diminishing 
> returns from both internal OpenSIPS contention and context switching.
> 
> I'm not saying there aren't some local minima and maxima, but they 
> aren't as magnitudinal as folks think. It's not that Ubuntu Server is 
> mistuned, it's that you're abusing it. :-) You can't put the milk back 
> in the cow, although it's quite a spectacle ...
> 
> -- Alex
> 
> On 6/12/20 6:02 PM, Calvin Ellison wrote:
>> I noticed a way-too-small receive buffer value in the OpenSIPS startup 
>> messages and it turns out that a fresh Ubuntu 18 Server install has 
>> absolutely terrible sysctl defaults for high-performance networking. I 
>> got my 8-core lab from less than 2,000 CPS up to 14,000 CPS using a 
>> spread of all dips in non-async mode just by setting the following to 
>> match "maxbuffer=16777216":
>>
>> net.core.rmem_max = 16777216
>> net.core.wmem_max = 16777216
>>
>> Does OpenSIPS have guidelines for sysclt and other OS parameters?
>>
>>     Async requires the TM module which adds additional overhead and
>>     memory allocation.
>>
>>
>> According to with the docs:
>> "By requiring less processes to complete the same amount of work in the
>> same amount of time, process context switching is minimized and
>> overall CPU usage is improved. Less processes will also eat up less
>> system memory."
>>
>> So which is it? When should async be used, and when should async not 
>> be used? One can only invest so many hours in load testing 
>> combinations of sync/async, the number of children, timer_partitions, 
>> etc. Some fuzzy math based on CPU core count, SpecInt Rate, BogoMIPS, 
>> etc. would be a great starting point.
>>
>> Regards,
>>
>> *Calvin Ellison*
>> Senior Voice Operations Engineer
>> calvin.ellison at voxox.com <mailto:calvin.ellison at voxox.com>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
> 

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/