[OpenSIPS-Users] OpenSIPS CPU Overload, Blocking detected and Timer Warnings
Bogdan-Andrei Iancu
bogdan at opensips.org
Fri Feb 14 15:19:22 UTC 2025
Hi Devang,
The blocked processes show some lock waiting in the usrloc module. But
how is holding the lock, hard to tell. I would nee the backtrace (do the
"trap") from all processes.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
https://www.siphub.com
On 14.02.2025 12:19, Devang Dhandhalya wrote:
> Hello Bogdan,
>
> Thank you for your response!
> I captured a *GDB dump* using the |opensips-cli trap| command for a
> specific OpenSIPS process that was consuming *100% CPU*.
> Both opensips pod configuration: 4 core, 16 GB RAM each.
>
> *Here is the Trap dump:
> *https://pastebin.com/5RcfWFdq
>
> In this dump 4 process (140,143,144 and 158 pid ) dump given, if
> required full dump then also i can provide.
>
> Here is the OpenSIPS Warning and ERROR Logs:
> WARNING:core:timer_ticker: timer task <nh-timer> already scheduled
> 1010 ms ago (now 89373630 ms), delaying execution
> ERROR:freeswitch:handle_reconnects: failed to connect to FS sock
> '172.x.x.x:8021'
> ERROR:freeswitch:handle_io: oom
> WARNING:core:timer_ticker: timer task <ul-timer> already scheduled
> 60000 ms ago (now 89461620 ms), delaying execution
>
> Here is the opensips configuration:
> https://pastebin.com/p8ZCyniy
>
> Here is 1 more issue, OpenSIPS with the load_balancer module to
> distribute calls among three FreeSWITCH servers. The FreeSWITCH
> servers are running in Kubernetes, so their IPs change when pods
> restart. When a FreeSWITCH pod restarts and gets a new IP, we update
> the load_balancer table in OpenSIPS and run the lb_reload command.
> However, OpenSIPS still tries to connect to the old FreeSWITCH
> instance instead of using the updated one?
>
> In the dump it looks like OpenSIPS has encountered a deadlock or
> performance issue in the usrloc (User Location) module.
> Any suggestions would be appreciated, kindly Let me know if you
> require any further information related to opensips configuration or
> kubernetes pod setup.
>
> Regards,
> Devang Dhandhalya
>
>
> On Thu, Dec 12, 2024 at 7:47 PM Bogdan-Andrei Iancu
> <bogdan at opensips.org> wrote:
>
> Hi Devang,
>
> You get such CRITICAL errors when your opensips is blocked /
> stuck. The way to debug is to do a "trap" via CLI - the backtraces
> should be correlated with the reported blocked processes.
> Hopefully the backtrace will provide some hints about the blockage.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
> https://www.opensips-solutions.com
> https://www.siphub.com
>
> On 10.12.2024 16:43, Devang Dhandhalya wrote:
>> Hello Everyone
>>
>> I am using OpenSIPS(3.4.9) kubernetes contarized with
>> Active-Active HA setup
>>
>> I am facing an issue where the CPU usage of OpenSIPS gradually
>> increases, and eventually, I am unable to use opensips-cli to
>> check the process list or retrieve statistics.
>> Below are the errors I am encountering:
>>
>> WARNING:core:timer_ticker: timer task <nh-timer> already
>> scheduled 117890260 ms ago (now 181015110 ms), delaying execution
>> WARNING:core:timer_ticker: timer task <ul-timer> already
>> scheduled 117893270 ms ago (now 181015120 ms), delaying execution
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 39 to proc id 7/94 [SIP receiver
>> udp:172.50.59.6:5060 <http://172.50.59.6:5060>]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 39 to proc id 7/94 [SIP receiver
>> udp:172.50.59.6:5060 <http://172.50.59.6:5060>]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 39 to proc id 7/94 [SIP receiver
>> udp:172.50.59.6:5060 <http://172.50.59.6:5060>]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 39 to proc id 7/94 [SIP receiver
>> udp:172.50.59.6:5060 <http://172.50.59.6:5060>]
>> DERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 39 to proc id 7/94 [SIP receiver
>> udp:172.50.59.6:5060 <http://172.50.59.6:5060>]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 7
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 7
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 227 to proc id 54/141 [TCP receiver]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 54
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 227 to proc id 54/141 [TCP receiver]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 54
>> CRITICAL:core:__ipc_send_job: blocking detected while sending job
>> type 0[RPC] on 227 to proc id 54/141 [TCP receiver]
>> ERROR:core:signal_pkg_status: failed to trigger pkg stats for
>> process 54
>> ERROR:core:handle_new_connect: maximum number of connections
>> exceeded: 2048/2048
>>
>> In opensips configuration we ae handling TLS and WSS protocols
>> We used Nathelper module for handling NAT and storing usrloc
>> details in mongoDB using *federation-cachedb-cluster* and
>> pinging_mode is ownership
>> we are using auto scaling profiles and tag core parameter with
>> socket and this tag parameter we using in save function as
>> ownership tag
>>
>> *Important*: I noticed that when any one of the opensips pod is
>> restarted after that sometime we are facing above warning of
>> nh-timer and ul-timer and after that cpu starts increasing.
>> After restart opensips pod private ip is changing so performing
>> require action in postgres db to remove old record and add new
>> record and perform clusterer reload on that opensips pod.
>>
>> So After performing the above actions why are we facing nh-timer
>> and ul-timer warnings? I think due to that our resources start
>> using and after this leads to OpenSIPS being unable to process
>> calls or execute opensips-cli commands effectively.
>>
>> Any suggestions would be appreciated, kindly Let me know if you
>> require any further information related to opensips configuration.
>>
>> Regards,
>> *Devang Dhandhalya*
>>
>> *https://www.ecosmob.com/itexpo-2025/
>> <https://www.ecosmob.com/itexpo-2025/>
>> *
>> *Disclaimer*
>> In addition to generic Disclaimer which you have agreed on our
>> website, any views or opinions presented in this email are solely
>> those of the originator and do not necessarily represent those of
>> the Company or its sister concerns. Any liability (in negligence,
>> contract or otherwise) arising from any third party taking any
>> action, or refraining from taking any action on the basis of any
>> of the information contained in this email is hereby excluded.
>>
>> *Confidentiality*
>> This communication (including any attachment/s) is intended only
>> for the use of the addressee(s) and contains information that is
>> PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination,
>> distribution, or copying of this communication is prohibited.
>> Please inform originator if you have received it in error.
>>
>> *Caution for viruses, malware etc.*
>> This communication, including any attachments, may not be free of
>> viruses, trojans, similar or new contaminants/malware,
>> interceptions or interference, and may not be compatible with
>> your systems. You shall carry out virus/malware scanning on your
>> own before opening any attachment to this e-mail. The sender of
>> this e-mail and Company including its sister concerns shall not
>> be liable for any damage that may incur to you as a result of
>> viruses, incompleteness of this message, a delay in receipt of
>> this message or any other computer problems.
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> --
> Regards,
> *Devang Dhandhalya*
>
> Ecosmob Technologies Pvt. Ltd. <https://www.ecosmob.com/>
>
> Ecosmob Technologies Pvt. Ltd.
> https://www.ecosmob.com
>
> VoIP | Web | Mobile | IoT | Big Data
>
>
> *Disclaimer*
> In addition to generic Disclaimer which you have agreed on our
> website, any views or opinions presented in this email are solely
> those of the originator and do not necessarily represent those of the
> Company or its sister concerns. Any liability (in negligence, contract
> or otherwise) arising from any third party taking any action, or
> refraining from taking any action on the basis of any of the
> information contained in this email is hereby excluded.
>
> *Confidentiality*
> This communication (including any attachment/s) is intended only for
> the use of the addressee(s) and contains information that is
> PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination,
> distribution, or copying of this communication is prohibited. Please
> inform originator if you have received it in error.
>
> *Caution for viruses, malware etc.*
> This communication, including any attachments, may not be free of
> viruses, trojans, similar or new contaminants/malware, interceptions
> or interference, and may not be compatible with your systems. You
> shall carry out virus/malware scanning on your own before opening any
> attachment to this e-mail. The sender of this e-mail and Company
> including its sister concerns shall not be liable for any damage that
> may incur to you as a result of viruses, incompleteness of this
> message, a delay in receipt of this message or any other computer
> problems.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20250214/a8af7a79/attachment-0001.html>
More information about the Users
mailing list