[OpenSIPS-Users] OpenSIPS CPU Overload, Blocking detected and Timer Warnings

Wed Feb 19 14:45:36 UTC 2025

Hello Bogdan,

Thank you for your response!

I have shared two files opensips1 and opensips2 gdb trap traces with you,
Both opensips are in Active - Active mode.

Kindly let me know if you require any further information.

Regards,
Devang Dhandhalya

On Fri, Feb 14, 2025 at 8:49 PM Bogdan-Andrei Iancu <bogdan at opensips.org>
wrote:

> Hi Devang,
>
> The blocked processes show some lock waiting in the usrloc module. But how
> is holding the lock, hard to tell. I would nee the backtrace (do the
> "trap") from all processes.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   https://www.opensips-solutions.com
>   https://www.siphub.com
>
> On 14.02.2025 12:19, Devang Dhandhalya wrote:
>
> Hello Bogdan,
>
> Thank you for your response!
> I captured a *GDB dump* using the opensips-cli trap command for a
> specific OpenSIPS process that was consuming *100% CPU*.
> Both opensips pod configuration: 4 core, 16 GB RAM each.
>
>
> *Here is the Trap dump: *https://pastebin.com/5RcfWFdq
>
> In this dump 4 process (140,143,144 and 158 pid ) dump given, if required
> full dump then also i can provide.
>
> Here is the OpenSIPS Warning and ERROR Logs:
> WARNING:core:timer_ticker: timer task <nh-timer> already scheduled 1010 ms
> ago (now 89373630 ms), delaying execution
> ERROR:freeswitch:handle_reconnects: failed to connect to FS sock
> '172.x.x.x:8021'
> ERROR:freeswitch:handle_io: oom
> WARNING:core:timer_ticker: timer task <ul-timer> already scheduled 60000
> ms ago (now 89461620 ms), delaying execution
>
> Here is the opensips configuration:
> https://pastebin.com/p8ZCyniy
>
> Here is 1 more issue, OpenSIPS with the load_balancer module to distribute
> calls among three FreeSWITCH servers. The FreeSWITCH servers are running in
> Kubernetes, so their IPs change when pods restart. When a FreeSWITCH pod
> restarts and gets a new IP, we update the load_balancer table in OpenSIPS
> and run the lb_reload command. However, OpenSIPS still tries to connect to
> the old FreeSWITCH instance instead of using the updated one?
>
> In the dump it looks like OpenSIPS has encountered a deadlock or
> performance issue in the usrloc (User Location) module.
> Any suggestions would be appreciated, kindly Let me know if you require
> any further information related to opensips configuration or kubernetes pod
> setup.
>
> Regards,
> Devang Dhandhalya
>
>
> On Thu, Dec 12, 2024 at 7:47 PM Bogdan-Andrei Iancu <bogdan at opensips.org>
> wrote:
>
>> Hi Devang,
>>
>> You get such CRITICAL errors when your opensips is blocked / stuck. The
>> way to debug is to do a "trap" via CLI - the backtraces should be
>> correlated with the reported blocked processes. Hopefully the backtrace
>> will provide some hints about the blockage.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   https://www.opensips-solutions.com
>>   https://www.siphub.com
>>
>> On 10.12.2024 16:43, Devang Dhandhalya wrote:
>>
>> Hello Everyone
>>
>> I am using OpenSIPS(3.4.9) kubernetes contarized with Active-Active HA
>> setup
>>
>> I am facing an issue where the CPU usage of OpenSIPS gradually increases,
>> and eventually, I am unable to use opensips-cli to check the process list
>> or retrieve statistics.
>> Below are the errors I am encountering:
>>
>> WARNING:core:timer_ticker: timer task <nh-timer> already scheduled
>> 117890260 ms ago (now 181015110 ms), delaying execution
>> WARNING:core:timer_ticker: timer task <ul-timer> already scheduled
>> 117893270 ms ago (now 181015120 ms), delaying execution
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 39  to proc id 7/94 [SIP receiver udp:172.50.59.6:5060]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 39  to proc id 7/94 [SIP receiver udp:172.50.59.6:5060]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 39  to proc id 7/94 [SIP receiver udp:172.50.59.6:5060]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 39  to proc id 7/94 [SIP receiver udp:172.50.59.6:5060]
>> DERROR:core:signal_pkg_status: failed to trigger pkg stats for process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 39  to proc id 7/94 [SIP receiver udp:172.50.59.6:5060]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 7
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 227  to proc id 54/141 [TCP receiver]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 54
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 227  to proc id 54/141 [TCP receiver]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 54
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job type
>> 0[RPC] on 227  to proc id 54/141 [TCP receiver]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for process 54
>> ERROR:core:handle_new_connect: maximum number of connections exceeded:
>> 2048/2048
>>
>> In opensips configuration we ae handling TLS and WSS protocols
>> We used Nathelper module for handling NAT and storing usrloc details in
>> mongoDB using *federation-cachedb-cluster* and pinging_mode is ownership
>> we are using auto scaling profiles and tag core parameter with socket and
>> this tag parameter we using in save function as ownership tag
>>
>> *Important*: I noticed that when any one of the opensips pod is
>> restarted after that sometime we are facing above warning of nh-timer and
>> ul-timer and after that cpu starts increasing.
>> After restart opensips pod private ip is changing so performing require
>> action in postgres db to remove old record and add new record and perform
>> clusterer reload on that opensips pod.
>>
>> So After performing the above actions why are we facing nh-timer and
>> ul-timer warnings? I think due to that our resources start using and after
>> this leads to OpenSIPS being unable to process calls or execute
>> opensips-cli commands effectively.
>>
>> Any suggestions would be appreciated, kindly Let me know if you require
>> any further information related to opensips configuration.
>>
>> Regards,
>> *Devang Dhandhalya*
>>
>>
>> *[image: https://www.ecosmob.com/itexpo-2025/]
>> <https://www.ecosmob.com/itexpo-2025/> *
>> *Disclaimer*
>> In addition to generic Disclaimer which you have agreed on our website,
>> any views or opinions presented in this email are solely those of the
>> originator and do not necessarily represent those of the Company or its
>> sister concerns. Any liability (in negligence, contract or otherwise)
>> arising from any third party taking any action, or refraining from taking
>> any action on the basis of any of the information contained in this email
>> is hereby excluded.
>>
>> *Confidentiality*
>> This communication (including any attachment/s) is intended only for the
>> use of the addressee(s) and contains information that is PRIVILEGED AND
>> CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or copying
>> of this communication is prohibited. Please inform originator if you have
>> received it in error.
>>
>> *Caution for viruses, malware etc.*
>> This communication, including any attachments, may not be free of
>> viruses, trojans, similar or new contaminants/malware, interceptions or
>> interference, and may not be compatible with your systems. You shall carry
>> out virus/malware scanning on your own before opening any attachment to
>> this e-mail. The sender of this e-mail and Company including its sister
>> concerns shall not be liable for any damage that may incur to you as a
>> result of viruses, incompleteness of this message, a delay in receipt of
>> this message or any other computer problems.
>>
>> _______________________________________________
>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>>
>
> --
> Regards,
> *Devang Dhandhalya*
>
> [image: Ecosmob Technologies Pvt. Ltd.] <https://www.ecosmob.com/>
>
> Ecosmob Technologies Pvt. Ltd.
> https://www.ecosmob.com
>
> VoIP | Web | Mobile | IoT | Big Data
>
> *Disclaimer*
> In addition to generic Disclaimer which you have agreed on our website,
> any views or opinions presented in this email are solely those of the
> originator and do not necessarily represent those of the Company or its
> sister concerns. Any liability (in negligence, contract or otherwise)
> arising from any third party taking any action, or refraining from taking
> any action on the basis of any of the information contained in this email
> is hereby excluded.
>
> *Confidentiality*
> This communication (including any attachment/s) is intended only for the
> use of the addressee(s) and contains information that is PRIVILEGED AND
> CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or copying
> of this communication is prohibited. Please inform originator if you have
> received it in error.
>
> *Caution for viruses, malware etc.*
> This communication, including any attachments, may not be free of viruses,
> trojans, similar or new contaminants/malware, interceptions or
> interference, and may not be compatible with your systems. You shall carry
> out virus/malware scanning on your own before opening any attachment to
> this e-mail. The sender of this e-mail and Company including its sister
> concerns shall not be liable for any damage that may incur to you as a
> result of viruses, incompleteness of this message, a delay in receipt of
> this message or any other computer problems.
>
>
>

-- 
Regards,
*Devang Dhandhalya*

[image: Ecosmob Technologies Pvt. Ltd.] <https://www.ecosmob.com/>

Ecosmob Technologies Pvt. Ltd.
https://www.ecosmob.com

VoIP | Web | Mobile | IoT | Big Data

-- 

*Disclaimer*
In addition to generic Disclaimer which you have agreed on 
our website, any views or opinions presented in this email are solely those 
of the originator and do not necessarily represent those of the Company or 
its sister concerns. Any liability (in negligence, contract or otherwise) 
arising from any third party taking any action, or refraining from taking 
any action on the basis of any of the information contained in this email 
is hereby excluded.

*Confidentiality*
This communication (including any 
attachment/s) is intended only for the use of the addressee(s) and contains 
information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, 
dissemination, distribution, or copying of this communication is 
prohibited. Please inform originator if you have received it in error.

*Caution for viruses, malware etc.*
This communication, including any 
attachments, may not be free of viruses, trojans, similar or new 
contaminants/malware, interceptions or interference, and may not be 
compatible with your systems. You shall carry out virus/malware scanning on 
your own before opening any attachment to this e-mail. The sender of this 
e-mail and Company including its sister concerns shall not be liable for 
any damage that may incur to you as a result of viruses, incompleteness of 
this message, a delay in receipt of this message or any other computer 
problems. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20250219/31920623/attachment-0001.html>