<div dir="ltr">Hello Bogdan,<br><br>Thank you for your response!<br><br>I have shared two files opensips1 and opensips2 gdb trap traces with you, Both opensips are in Active - Active mode.<br><br>Kindly let me know if you require any further information.<br><br>Regards,<br>Devang Dhandhalya</div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Fri, Feb 14, 2025 at 8:49 PM Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<font face="monospace">Hi Devang,<br>
<br>
The blocked processes show some lock waiting in the usrloc module.
But how is holding the lock, hard to tell. I would nee the
backtrace (do the "trap") from all processes.<br>
<br>
Regards,<br>
</font>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
<a href="https://www.siphub.com" target="_blank">https://www.siphub.com</a></pre>
<div>On 14.02.2025 12:19, Devang Dhandhalya
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello Bogdan,<br>
<br>
Thank you for your response!<br>
I captured a <strong>GDB dump</strong> using the <code>opensips-cli
trap</code> command for a specific OpenSIPS process that was
consuming <strong>100% CPU</strong>. <br>
Both opensips pod configuration: 4 core, 16 GB RAM each.<br>
<br>
<b>Here is the Trap dump:<br>
</b><a href="https://pastebin.com/5RcfWFdq" target="_blank">https://pastebin.com/5RcfWFdq</a>
<div>
<div><br>
In this dump 4 process (140,143,144 and 158 pid ) dump
given, if required full dump then also i can provide.<br>
<br>
Here is the OpenSIPS Warning and ERROR Logs:<br>
WARNING:core:timer_ticker: timer task <nh-timer>
already scheduled 1010 ms ago (now 89373630 ms), delaying
execution<br>
ERROR:freeswitch:handle_reconnects: failed to connect to FS
sock '172.x.x.x:8021'<br>
ERROR:freeswitch:handle_io: oom<br>
WARNING:core:timer_ticker: timer task <ul-timer>
already scheduled 60000 ms ago (now 89461620 ms), delaying
execution<br>
<br>
Here is the opensips configuration:<br>
<a href="https://pastebin.com/p8ZCyniy" target="_blank">https://pastebin.com/p8ZCyniy</a></div>
<div><br>
Here is 1 more issue, OpenSIPS with the load_balancer module
to distribute calls among three FreeSWITCH servers. The
FreeSWITCH servers are running in Kubernetes, so their IPs
change when pods restart. When a FreeSWITCH pod restarts and
gets a new IP, we update the load_balancer table in OpenSIPS
and run the lb_reload command. However, OpenSIPS still tries
to connect to the old FreeSWITCH instance instead of using
the updated one?<br>
<br>
In the dump it looks like OpenSIPS has encountered a
deadlock or performance issue in the usrloc (User Location)
module.<br>
Any suggestions would be appreciated, kindly Let me know if
you require any further information related to opensips
configuration or kubernetes pod setup.<br>
<br>
Regards,<br>
Devang Dhandhalya<br>
<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Dec 12, 2024 at
7:47 PM Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> <font face="monospace">Hi Devang,<br>
<br>
You get such CRITICAL errors when your opensips is blocked
/ stuck. The way to debug is to do a "trap" via CLI - the
backtraces should be correlated with the reported blocked
processes. Hopefully the backtrace will provide some hints
about the blockage.<br>
<br>
Regards,<br>
</font>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
<a href="https://www.siphub.com" target="_blank">https://www.siphub.com</a></pre>
<div>On 10.12.2024 16:43, Devang Dhandhalya wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Everyone<br>
<br>
I am using OpenSIPS(3.4.9) kubernetes contarized with
Active-Active HA setup<br>
<br>
I am facing an issue where the CPU usage of OpenSIPS
gradually increases, and eventually, I am unable to
use opensips-cli to check the process list or retrieve
statistics.<br>
Below are the errors I am encountering:<br>
<br>
WARNING:core:timer_ticker: timer task <nh-timer>
already scheduled 117890260 ms ago (now 181015110 ms),
delaying execution</div>
WARNING:core:timer_ticker: timer task <ul-timer>
already scheduled 117893270 ms ago (now 181015120 ms),
delaying execution<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 39 to proc id 7/94 [SIP
receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 39 to proc id 7/94 [SIP
receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 39 to proc id 7/94 [SIP
receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 39 to proc id 7/94 [SIP
receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
DERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 39 to proc id 7/94 [SIP
receiver udp:<a href="http://172.50.59.6:5060" target="_blank">172.50.59.6:5060</a>]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 7<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 7<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 227 to proc id 54/141 [TCP
receiver]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 54<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 227 to proc id 54/141 [TCP
receiver]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 54<br>
CRITICAL:core:__ipc_send_job: blocking detected while
sending job type 0[RPC] on 227 to proc id 54/141 [TCP
receiver]<br>
ERROR:core:signal_pkg_status: failed to trigger pkg
stats for process 54<br>
ERROR:core:handle_new_connect: maximum number of
connections exceeded: 2048/2048<br>
<div><br>
In opensips configuration we ae handling TLS and WSS
protocols<br>
We used Nathelper module for handling NAT and storing
usrloc details in mongoDB using <b>federation-cachedb-cluster</b>
and pinging_mode is ownership<br>
we are using auto scaling profiles and tag core
parameter with socket and this tag parameter we using
in save function as ownership tag<br>
<br>
<b>Important</b>: I noticed that when any one of the
opensips pod is restarted after that sometime we are
facing above warning of nh-timer and ul-timer and
after that cpu starts increasing.<br>
After restart opensips pod private ip is changing so
performing require action in postgres db to remove old
record and add new record and perform clusterer reload
on that opensips pod.<br>
<br>
So After performing the above actions why are we
facing nh-timer and ul-timer warnings? I think due to
that our resources start using and after this leads to
OpenSIPS being unable to process calls or execute
opensips-cli commands effectively.<br>
<br>
Any suggestions would be appreciated, kindly Let me
know if you require any further information related to
opensips configuration.<br>
</div>
<div><br>
</div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr"><span style="color:rgb(34,34,34)">
<div style="font-size:12.8px">Regards,<br>
<b>Devang Dhandhalya</b></div>
</span></div>
</div>
</div>
<br>
<div><b><a href="https://www.ecosmob.com/itexpo-2025/" target="_blank"><img src="https://ecosmobnew.ecosmob.net/wp-content/uploads/2024/11/It-expo-25-Email-signature.jpg" alt="https://www.ecosmob.com/itexpo-2025/"></a><br>
</b></div>
<div><b>Disclaimer</b></div>
<div>
<div><span>In addition to generic Disclaimer which you
have agreed on our website, any views or opinions
presented in this email are solely those of the
originator and do not necessarily represent those of
the Company or its sister concerns. Any liability
(in negligence, contract or otherwise) arising from
any third party taking any action, or refraining
from taking any action on the basis of any of the
information contained in this email is hereby
excluded.</span></div>
</div>
<div><span><br>
</span></div>
<div><span><b>Confidentiality</b></span></div>
<div><span>This communication (including any attachment/s)
is intended only for the use of the addressee(s) and
contains information that is PRIVILEGED AND
CONFIDENTIAL. Unauthorized reading, dissemination,
distribution, or copying of this communication is
prohibited. Please inform originator if you have
received it in error.</span></div>
<div><span><br>
</span></div>
<div><span><b>Caution for viruses, malware etc.</b></span></div>
<div><span>This communication, including any attachments,
may not be free of viruses, trojans, similar or new
contaminants/malware, interceptions or interference,
and may not be compatible with your systems. You shall
carry out virus/malware scanning on your own before
opening any attachment to this e-mail. The sender of
this e-mail and Company including its sister concerns
shall not be liable for any damage that may incur to
you as a result of viruses, incompleteness of this
message, a delay in receipt of this message or any
other computer problems. </span></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr"><span style="color:rgb(34,34,34)">
<div style="font-size:12.8px">Regards,<br>
<b>Devang Dhandhalya</b></div>
</span>
<div style="color:rgb(34,34,34)">
<div dir="ltr">
<div dir="ltr">
<p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px"><a href="https://www.ecosmob.com/" style="color:rgb(17,85,204)" target="_blank"><img alt="Ecosmob Technologies Pvt. Ltd." src="https://htmlsigs.s3.amazonaws.com/logos/files/000/650/332/landscape/ecm_2048_1152.png" width="160" height="40" border="0"></a></p>
<p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px"><font size="2"><span style="font-weight:bold;color:rgb(33,33,33)">Ecosmob
Technologies Pvt. Ltd.</span> <br>
<span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><a href="https://www.ecosmob.com/" style="color:rgb(17,85,204)" target="_blank">https://www.ecosmob.com</a></font></p>
<p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px">VoIP
| Web | Mobile | IoT | Big Data</p>
</div>
</div>
</div>
</div>
</div>
<br>
<div><b>Disclaimer</b></div>
<div>
<div><span>In addition to generic Disclaimer which you have
agreed on our website, any views or opinions presented in
this email are solely those of the originator and do not
necessarily represent those of the Company or its sister
concerns. Any liability (in negligence, contract or
otherwise) arising from any third party taking any action,
or refraining from taking any action on the basis of any of
the information contained in this email is hereby excluded.</span></div>
</div>
<div><span><br>
</span></div>
<div><span><b>Confidentiality</b></span></div>
<div><span>This communication (including any attachment/s) is
intended only for the use of the addressee(s) and contains
information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized
reading, dissemination, distribution, or copying of this
communication is prohibited. Please inform originator if you
have received it in error.</span></div>
<div><span><br>
</span></div>
<div><span><b>Caution for viruses, malware etc.</b></span></div>
<div><span>This communication, including any attachments, may not
be free of viruses, trojans, similar or new
contaminants/malware, interceptions or interference, and may
not be compatible with your systems. You shall carry out
virus/malware scanning on your own before opening any
attachment to this e-mail. The sender of this e-mail and
Company including its sister concerns shall not be liable for
any damage that may incur to you as a result of viruses,
incompleteness of this message, a delay in receipt of this
message or any other computer problems. </span></div>
</blockquote>
<br>
</div>
</blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span style="color:rgb(34,34,34)"><div style="font-size:12.8px">Regards,<br><b>Devang Dhandhalya</b></div></span><div style="color:rgb(34,34,34)"><div dir="ltr"><div dir="ltr"><p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px"><a href="https://www.ecosmob.com/" style="color:rgb(17,85,204)" target="_blank"><img alt="Ecosmob Technologies Pvt. Ltd." src="https://htmlsigs.s3.amazonaws.com/logos/files/000/650/332/landscape/ecm_2048_1152.png" border="0" height="40" width="160"></a></p><p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px"><font size="2"><span style="font-weight:bold;color:rgb(33,33,33)">Ecosmob Technologies Pvt. Ltd.</span> <br><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><span style="color:rgb(33,33,33)"></span><a href="https://www.ecosmob.com/" style="color:rgb(17,85,204)" target="_blank">https://www.ecosmob.com</a></font></p><p style="font-size:10px;font-family:helvetica,arial,sans-serif;line-height:12px;margin-bottom:10px">VoIP | Web | Mobile | IoT | Big Data</p></div></div></div></div></div>
<br>
<div></div><div><b>Disclaimer</b></div><div><div><span>In addition to generic Disclaimer which you have agreed on our website, any views or opinions presented in this email are solely those of the originator and do not necessarily represent those of the Company or its sister concerns. Any liability (in negligence, contract or otherwise) arising from any third party taking any action, or refraining from taking any action on the basis of any of the information contained in this email is hereby excluded.</span></div></div><div><span><br></span></div><div><span><b>Confidentiality</b></span></div><div><span>This communication (including any attachment/s) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL. Unauthorized reading, dissemination, distribution, or copying of this communication is prohibited. Please inform originator if you have received it in error.</span></div><div><span><br></span></div><div><span><b>Caution for viruses, malware etc.</b></span></div><div><span>This communication, including any attachments, may not be free of viruses, trojans, similar or new contaminants/malware, interceptions or interference, and may not be compatible with your systems. You shall carry out virus/malware scanning on your own before opening any attachment to this e-mail. The sender of this e-mail and Company including its sister concerns shall not be liable for any damage that may incur to you as a result of viruses, incompleteness of this message, a delay in receipt of this message or any other computer problems. </span></div>