<div dir="ltr">Hello Bogdan,<br><br>Thank you for your response!<br>I captured a <strong>GDB dump</strong> using the <code>opensips-cli trap</code> command for a specific OpenSIPS process that was consuming <strong>100% CPU</strong>. <br>Both opensips pod configuration: 4 core, 16 GB RAM each.<br><br><b>Here is the Trap dump:<br></b><a href="https://pastebin.com/5RcfWFdq">https://pastebin.com/5RcfWFdq</a><div><div><br>In this dump 4 process (140,143,144 and 158 pid ) dump given, if required full dump then also i can provide.<br><br>Here is the OpenSIPS Warning and ERROR Logs:<br>WARNING:core:timer_ticker: timer task <nh-timer> already scheduled 1010 ms ago (now 89373630 ms), delaying execution<br>ERROR:freeswitch:handle_reconnects: failed to connect to FS sock '172.x.x.x:8021'<br>ERROR:freeswitch:handle_io: oom<br>WARNING:core:timer_ticker: timer task <ul-timer> already scheduled 60000 ms ago (now 89461620 ms), delaying execution<br><br>Here is the opensips configuration:<br><a href="https://pastebin.com/p8ZCyniy">https://pastebin.com/p8ZCyniy</a></div><div><br>Here is 1 more issue, OpenSIPS with the load_balancer module to distribute calls among three FreeSWITCH servers. The FreeSWITCH servers are running in Kubernetes, so their IPs change when pods restart. When a FreeSWITCH pod restarts and gets a new IP, we update the load_balancer table in OpenSIPS and run the lb_reload command. However, OpenSIPS still tries to connect to the old FreeSWITCH instance instead of using the updated one?<br><br>In the dump it looks like OpenSIPS has encountered a deadlock or performance issue in the usrloc (User Location) module.<br>Any suggestions would be appreciated, kindly Let me know if you require any further information related to opensips configuration or kubernetes pod setup.<br> <br>Regards,<br>Devang Dhandhalya<br><br></div></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Thu, Dec 12, 2024 at 7:47 PM Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<font face="monospace">Hi Devang,<br>
<br>
You get such CRITICAL errors when your opensips is blocked /
stuck. The way to debug is to do a "trap" via CLI - the backtraces
should be correlated with the reported blocked processes.
Hopefully the backtrace will provide some hints about the
blockage.<br>
<br>
Regards,<br>
</font>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
<a href="https://www.siphub.com" target="_blank">https://www.siphub.com</a></pre>
<div>On 10.12.2024 16:43, Devang Dhandhalya
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Everyone<br>
<br>
I am using OpenSIPS(3.4.9) kubernetes contarized with
Active-Active HA setup<br>
<br>
I am facing an issue where the CPU usage of OpenSIPS gradually
increases, and eventually, I am unable to use opensips-cli to
check the process list or retrieve statistics.<br>
Below are the errors I am encountering:<br>
<br>
WARNING:core:timer_ticker: timer task <nh-timer> already
scheduled 117890260 ms ago (now 181015110 ms), delaying
execution</div>
WARNING:core:timer_ticker: timer task <ul-timer> already
scheduled 117893270 ms ago (now 181015120 ms), delaying
execution<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 39 to proc id 7/94 [SIP receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 39 to proc id 7/94 [SIP receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 39 to proc id 7/94 [SIP receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 39 to proc id 7/94 [SIP receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
DERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 39 to proc id 7/94 [SIP receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 7<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 227 to proc id 54/141 [TCP receiver]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 54<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 227 to proc id 54/141 [TCP receiver]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 54<br>
CRITICAL:core:__ipc_send_job: blocking detected while sending
job type 0[RPC] on 227 to proc id 54/141 [TCP receiver]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg stats for
process 54<br>
ERROR:core:handle_new_connect: maximum number of connections
exceeded: 2048/2048<br>
<div><br>
In opensips configuration we ae handling TLS and WSS protocols<br>
We used Nathelper module for handling NAT and storing usrloc
details in mongoDB using <b>federation-cachedb-cluster</b>
and pinging_mode is ownership<br>
we are using auto scaling profiles and tag core parameter with
socket and this tag parameter we using in save function as
ownership tag<br>
<br>
<b>Important</b>: I noticed that when any one of the opensips
pod is restarted after that sometime we are facing above
warning of nh-timer and ul-timer and after that cpu starts
increasing.<br>
After restart opensips pod private ip is changing so
performing require action in postgres db to remove old record
and add new record and perform clusterer reload on that
opensips pod.<br>
<br>
So After performing the above actions why are we facing
nh-timer and ul-timer warnings? I think due to that our
resources start using and after this leads to OpenSIPS being
unable to process calls or execute opensips-cli commands
effectively.<br>
<br>
Any suggestions would be appreciated, kindly Let me know if
you require any further information related to opensips
configuration.<br>
</div>
<div><br>
</div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr"><span style="color:rgb(34,34,34)">
<div style="font-size:12.8px">Regards,<br>
<b>Devang Dhandhalya</b></div>
</span></div>
</div>
</div>
<br>
<div><b><a href="https://www.ecosmob.com/itexpo-2025/" target="_blank"><img src="https://ecosmobnew.ecosmob.net/wp-content/uploads/2024/11/It-expo-25-Email-signature.jpg" alt="https://www.ecosmob.com/itexpo-2025/"></a><br>
</b></div>
<div><b>Disclaimer</b></div>
<div>
<div><span>In addition to generic Disclaimer which you have
agreed on our website, any views or opinions presented in
this email are solely those of the originator and do not
necessarily represent those of the Company or its sister
concerns. Any liability (in negligence, contract or
otherwise) arising from any third party taking any action,
or refraining from taking any action on the basis of any of
the information contained in this email is hereby excluded.</span></div>
</div>
<div><span><br>
</span></div>
<div><span><b>Confidentiality</b></span></div>
<div><span>This communication (including any attachment/s) is
intended only for the use of the addressee(s) and contains
information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized
reading, dissemination, distribution, or copying of this
communication is prohibited. Please inform originator if you
have received it in error.</span></div>
<div><span><br>
</span></div>
<div><span><b>Caution for viruses, malware etc.</b></span></div>
<div><span>This communication, including any attachments, may not
be free of viruses, trojans, similar or new
contaminants/malware, interceptions or interference, and may
not be compatible with your systems. You shall carry out
virus/malware scanning on your own before opening any
attachment to this e-mail. The sender of this e-mail and
Company including its sister concerns shall not be liable for
any damage that may incur to you as a result of viruses,
incompleteness of this message, a delay in receipt of this
message or any other computer problems. </span></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
</blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span style="color:rgb(34,34,34)"><div style="font-size:12.8px">Regards,<br><b>Devang Dhandhalya</b></div></span><div style="color:rgb(34,34,34)"><div dir="ltr"><div dir="ltr"><p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px"><a href="https://www.ecosmob.com/" style="color:rgb(17,85,204)" target="_blank"><img alt="Ecosmob Technologies Pvt. Ltd." src="https://htmlsigs.s3.amazonaws.com/logos/files/000/650/332/landscape/ecm_2048_1152.png" border="0" height="40" width="160"></a></p><p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px"><font size="2"><span style="font-weight:bold;color:rgb(33,33,33)">Ecosmob Technologies Pvt. Ltd.</span> <br><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><a href="https://www.ecosmob.com/" style="color:rgb(17,85,204)" target="_blank">https://www.ecosmob.com</a></font></p><p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px">VoIP | Web | Mobile | IoT | Big Data</p></div></div></div></div></div>
<br>
<div></div><div><b>Disclaimer</b></div><div><div><span>In addition to generic Disclaimer which you have agreed on our website, any views or opinions presented in this email are solely those of the originator and do not necessarily represent those of the Company or its sister concerns. Any liability (in negligence, contract or otherwise) arising from any third party taking any action, or refraining from taking any action on the basis of any of the information contained in this email is hereby excluded.</span></div></div><div><span><br></span></div><div><span><b>Confidentiality</b></span></div><div><span>This communication (including any attachment/s) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or copying of this communication is prohibited. Please inform originator if you have received it in error.</span></div><div><span><br></span></div><div><span><b>Caution for viruses, malware etc.</b></span></div><div><span>This communication, including any attachments, may not be free of viruses, trojans, similar or new contaminants/malware, interceptions or interference, and may not be compatible with your systems. You shall carry out virus/malware scanning on your own before opening any attachment to this e-mail. The sender of this e-mail and Company including its sister concerns shall not be liable for any damage that may incur to you as a result of viruses, incompleteness of this message, a delay in receipt of this message or any other computer problems. </span></div>