[OpenSIPS-Users] Long reload time for mi tls_reload for 200 tls/ssl certs

[Denys Pozniak]

Hey Liviu and Bogdan,

Thanks for the help! There are intermediate test results:
- (-m 1024 -M 32 -a F_MALLOC) with sequentially tls reloads, the speed
situation does not change:

#opensips-cli -x mi get_statistics fragments && opensips-cli -x mi
get_statistics free_size && opensips-cli -x mi get_statistics
real_used_size && time opensips-cli -x mi tls_reload

{
    "shmem:fragments": 2211707
}
{
    "shmem:free_size": 803916912
}
{
    "shmem:real_used_size": 269797192
}
"OK"
real    1m25.763s

#opensips-cli -x mi get_statistics fragments && opensips-cli -x mi
get_statistics free_size && opensips-cli -x mi get_statistics
real_used_size && time opensips-cli -x mi tls_reload
    "shmem:fragments": 2212109
}
{
    "shmem:free_size": 803993112
}
{
    "shmem:real_used_size": 269775568
}
"OK"
real    1m20.740s

#opensips-cli -x mi get_statistics fragments && opensips-cli -x mi
get_statistics free_size && opensips-cli -x mi get_statistics
real_used_size && time opensips-cli -x mi tls_reload
{
    "shmem:fragments": 2212511
}
{
    "shmem:free_size": 803915424
}
{
    "shmem:real_used_size": 269799376
}
"OK"
real    1m17.780s

#opensips-cli -x mi get_statistics fragments && opensips-cli -x mi
get_statistics free_size && opensips-cli -x mi get_statistics
real_used_size && time opensips-cli -x mi tls_reload
    "shmem:fragments": 2212913
}
{
    "shmem:free_size": 803858880
}
{
    "shmem:real_used_size": 269855408
}
"OK"
real    1m13.682s

#opensips-cli -x mi get_statistics fragments && opensips-cli -x mi
get_statistics free_size && opensips-cli -x mi get_statistics
real_used_size && time opensips-cli -x mi tls_reload
{
    "shmem:fragments": 2213337
}
{
    "shmem:free_size": 803876200
}
{
    "shmem:real_used_size": 269865624
}


- (-m 1024 -M 32 -a HP_MALLOC) with this memory allocator the service does
not even start as systemd kills it by timeout:


*Jun 11 16:33:33 opensips-01.localhost systemd[1]: opensips.service: Failed
with result 'timeout'.*
*Jun 11 16:33:33 opensips-01.localhost systemd[1]: Stopped opensips.service
- OpenSIPS is a very fast and flexible SIP (RFC3261) server.*
*Jun 11 16:33:33 opensips-01.localhost systemd[1]: opensips.service:
Consumed 1min 52.462s CPU time.*

> What is the usage of the shm mem (use the shmem: stats class to see) ?
At the time of tls reload I can’t get any statistics (this time opensips
eats 100% of the CPU), only there is a possibility between reloads



пт, 31 мая 2024 г. в 11:51, Liviu Chircu <liviu at opensips.org>:

> Hi Denys,
>
> The report shows OpenSSL library doing small SHM allocations (4, 10,
> 608, 24... bytes), which seem to frequently take place inside the
> PEM_read_bio_X509() loop (as part of the load_certificate_db()
> function).  Such a sequence of allocations could be stress-testing the
> allocator in a way that could justify 250 ms per certificate in total,
> as it is fragmenting the memory.  The effect can be more pronounced the
> *less* stuff is going on in your OpenSIPS instance, as the process of
> breaking up the big memory chunk into smaller units may use up to
> hundreds of cycles on each allocation.  For example:  testing box with
> no SIP traffic, or 'tls_reload' after a fresh restart, etc.
>
> Please try the following:
>
> - still using F_MALLOC, try doing more 'tls_reload' operations in a
> row.  Does performance improve?
>
> - try using the "-a HP_MALLOC" allocator when booting your OpenSIPS -
> that one favors memory fragmentation a bit more, so subsequent reloads
> should be faster
>
> Best regards,
>
> Liviu Chircu
> www.twitter.com/liviuchircu | www.opensips-solutions.com
>
> On 14.05.2024 16:47, Denys Pozniak wrote:
> > I disabled logging and added some resources to the virtual machine.
> > On a working OpenSIPS, I reloaded the tls several times and in
> > parallel ran a trap.
> > #opensips-cli -x mi tls_reload
> > #opensips-cli -x trap
> >
> > If possible, please analyze it again, maybe you could find something
> > interesting:
> > https://github.com/denyspozniak/opensips_tls_debug/tree/main
> >
> > Thanks in advance!
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>


-- 

BR,
Denys Pozniak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20240611/1d80cf3f/attachment.html>