[OpenSIPS-Users] Fine tuning high CPS and msyql queries

[Alex Balashov]

Hi Calvin,

I'm really glad you were able to get things sorted out, and I apologise 
if the thread got testy. I do appreciate your follow-up, which I think 
will benefit readers looking for similar answers.

A few inline thoughts:

On 6/15/20 4:04 PM, Calvin Ellison wrote:

> I attempted to reproduce the original breakdown around 3000 CPS using 
> the default 212992 byte receive buffer and could not, which tells me I 
> broke a cardinal rule of load testing and changed more than one thing at 
> a time. Also, don't do load testing when tired. I suspect that I had 
> also made a change to the sipp scenario recv/sched loops, or I had 
> unknowingly broken something while checking out the tuned package.

In several decades of doing backend systems programming, I've not found 
tuning Linux kernel defaults to be generally fruitful for improving 
throughput to any non-trivial degree. The defaults are sensible for 
almost all use-cases, all the more so given modern hardware and 
multi-core processors and the rest.

This is in sharp contrast to the conservative defaults some applications 
(e.g. Apache, MySQL) ship with on many distributions. I think the idea 
behind such conservative settings is to constrain the application so 
that in the event of a DDoS or similar event, it does not take over all 
available hardware resources, which would impede response and resolution.

But on the kernel settings, the only impactful changes I have ever seen 
are minor adjustments to slightly improve very niche server load 
problems of a rather global nature (e.g. related to I/O scheduling, NIC 
issues, storage, etc). This wasn't that kind of scenario.

In most respects, it just follows from first principles and Occam's 
Razor, IMHO. There's no reason for kernels to ship tuned unnecessarily 
conservatively to deny average users something on the order of _several 
times'_ more performance from their hardware, and any effort to do that 
would be readily apparent and, it stands to reason, staunchly opposed. 
It therefore also stands to reason that there isn't some silver bullet 
or magic setting that unlocks multiplicative performance gains, if only 
one just knows the secret sauce or thinks to tweak it--for the simple 
reason that if such a tweak existed, it would be systemically 
rationalised away, absent a clear and persuasive basis for such an 
artificial and contrived limit to exist. I cannot conceive of what such 
a basis would look like, and I'd like to think that's not just a failure 
of imagination.

Or in other words, it goes with the commonsensical, "If it seems too 
good to be true, it is," intuition. The basic fundamentals of the 
application, and to a lesser but still very significant extent the 
hardware (in terms of its relative homogeneity nowadays), determine 
99.9% of the performance characteristics, and matter a thousand times 
more than literally anything one can tweak.

> I deeply appreciate Alex's instance that I was wrong and to keep 
> digging. I am happy to retract my claim regarding "absolutely terrible 
> sysctl defaults". Using synchronous/blocking DB queries, the 8-core 
> server reached 14,000 CPS, at which point I declared it fixed and went 
> to bed. It could probably go higher: there's only one DB query with a 
> <10ms response time, Memcache for the query response, and some logic to 
> decide how to respond. There's only a single non-200 final response, so 
> it's probably as minimalist as it gets.

I would agree that with such a minimal call processing loop, given a 
generous number of CPU cores you shouldn't be terribly limited.

> If anyone else is trying to tune their setup, I think Alex's advice to 
> "not run more than 2 * (CPU threads) [children]" is the best place to 
> start. I had inherited this project from someone else's work under 
> version 1.11 and they had used 128 children. They were using remote DB 
> servers with much higher latency than the local DBs we have today, so 
> that might have been the reason. Or they were just wrong to being with.

Aye. Barring a workload consisting of exceptionally latent blocking 
service queries, there's really not a valid reason to ever have that 
many child processes, and even if one does have such a workload, plenty 
of reasons to lean on the fundamental latency problem rather than 
working around it with more child processes.

With the proviso that I am not an expert in modern-day OpenSIPS 
concurrency innards, the common OpenSER heritage prescribes a preforked 
worker process pool with SysV shared memory for inter-process 
communication (IPC). Like any shared memory space, this requires mutex 
locking so that multiple threads (in this case, processes) don't 
access/modify the same data structures at the same time* in ways that 
step on the others. Because every process holds and waits on these 
locks, this model works well when there aren't very many processes and 
their path to execution is mostly clear and not especially volatile, and 
when as little data is shared as possible. If you add a lot of 
processes, then there's a lot of fighting among them for internal locks 
and for CPU time, even if the execution cycle per se is fairly 
efficient. If you have 16 cores and 128 child processes, those processes 
are going to be fighting for those cores if they execute efficiently, 
while suffering from some amount of internal concurrency gridlock if 
they are not executing efficiently. Thus, 128 is for almost all cases 
very far beyond the sweet spot.

By analogy, think of a large multi-lane highway where almost all cars 
travel at more or less a constant speed, and, vitally, almost always 
stay in their lane, only very seldom making a lane change. As anyone who 
has ever been stuck in traffic knows, small speed changes by individual 
actors or small groups of cars can set off huge compression waves that 
have impact for miles back, and lane changes also have accordion 
effects. It's not a perfect analogy by any means, but it kind of conveys 
some sense of the general problem of contention. You really want to keep 
the "lanes" clear and eliminate all possible sources of friction, 
variance, and overlap.

* For exclusion purposes; of course, there's no such thing as truly 
simultaneous execution.

> The Description for Asynchronous Statements is extremely tempting and 
> was what started me down that path; it might be missing a qualification 
> that Async can be an improvement for slow blocking operations, but the 
> additional overhead may be a disadvantage for very fast blocking 
> operations.

There is indeed a certain amount of overhead in pushing data around 
multiple threads, the locking of shared data structures involved in 
doing so, etc. For slow, blocking operations, there's nevertheless an 
advantage, but if the operations aren't especially blocking, in many 
cases all that "async" stuff is just extra overhead.

Asynchronous tricks which deputise notification of the availability of 
further work or I/O to the kernel can be pretty efficient, just because 
life in kernel space is pretty efficient. But async execution in 
user-space requires user-space contrivances that suffer from all the 
problems of user-space in turn, so the economics can be really 
different. Mileage of course greatly varies with the implementation details.

-- Alex

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/