<div dir="ltr">hello Bogdan<div><br></div><div>Thanks for your reply.</div><div><br></div><div>But after 100k calls almost all actions take too many time</div><div><br></div><div><div>Nov 21 14:00:36 OS /usr/local/sbin/opensips[7235]: WARNING:core:log_expiry: threshold exceeded : msg processing took too long - 61006 us.Source : SIP/2.0 200 OK#015#012Via: SIP/2.0/UDP xxx.xxx.x.226:5060;branch=z9hG4bKf59a.7f50e365.0;rport=5060#015#012From: <sip:50390909150@xxx.xxx.x.226>;tag=BFH7y5ycBmc4m#015#012To: <sip:50331152944@xxx.xxx.x.227>;tag=3D4vXraevcj6r#015#012Call-ID: 22b7aa0c-4967-1236-989f-a0369f590b64#015#012CSeq: 115300342 INVITE#015#012Contact: <sip:50331152944@xxx.xxx.x.227:5060;transport=udp>#015#012User-Agent: VOXSBCSWITCH#015#012Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REFER, NOTIFY#015#012Require: timer#015#012Supported: timer, path, replaces#015#012Allow-Events: talk, hold, conference, refer#015#012Session-Expires: 3600;refresher=uac#015#012Content-Type: application/sdp#015#012Content-Disposition: session#015#012Content-Length: 269#015#012X-FS-Support: update_display,send_info#015#012#015#012v=0#015#012o=FreeSWITCH 1511254322 1511254323 IN IP4 xxx.xxx.x.227#015#012s=FreeSWITCH#015#012c=IN IP4 xxx.xxx.x.227#015#012t=0 0#015#012a=sendrecv#015#012m=audio 18498 RTP/AVP 8 101#015#012a=rtpmap:8 PCMA/8000#015#012a=rtpmap:101 telephone-event/8000#015#012a=fmtp:101 0-16#015#012a=ptime:20#015#012a=rtcp:18499 IN IP4 xxx.xxx.x.227</div><div><div>Nov 21 14:00:36 OS /usr/local/sbin/opensips[7235]: WARNING:core:log_expiry: threshold exceeded : msg processing took too long - 61006 us.Source : SIP/2.0 200 OK#015#012Via: SIP/2.0/UDP xxx.xxx.x.226:5060;branch=z9hG4bKf59a.7f50e365.0;rport=5060#015#012From: <sip:50390909150@xxx.xxx.x.226>;tag=BFH7y5ycBmc4m#015#012To: <sip:50331152944@xxx.xxx.x.227>;tag=3D4vXraevcj6r#015#012Call-ID: 22b7aa0c-4967-1236-989f-a0369f590b64#015#012CSeq: 115300342 INVITE#015#012Contact: <sip:50331152944@xxx.xxx.x.227:5060;transport=udp>#015#012User-Agent: SBCSWITCH#015#012Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REFER, NOTIFY#015#012Require: timer#015#012Supported: timer, path, replaces#015#012Allow-Events: talk, hold, conference, refer#015#012Session-Expires: 3600;refresher=uac#015#012Content-Type: application/sdp#015#012Content-Disposition: session#015#012Content-Length: 269#015#012X-FS-Support: update_display,send_info#015#012#015#012v=0#015#012o=FreeSWITCH 1511254322 1511254323 IN IP4 xxx.xxx.x.227#015#012s=FreeSWITCH#015#012c=IN IP4 xxx.xxx.x.227#015#012t=0 0#015#012a=sendrecv#015#012m=audio 18498 RTP/AVP 8 101#015#012a=rtpmap:8 PCMA/8000#015#012a=rtpmap:101 telephone-event/8000#015#012a=fmtp:101 0-16#015#012a=ptime:20#015#012a=rtcp:18499 IN IP4 xxx.xxx.x.227</div><div><br></div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7233]: WARNING:core:log_expiry: threshold exceeded : msg processing took too long - 117018 us.Source : SIP/2.0 503 Service Unavailable#015#012Via: SIP/2.0/UDP xxx.xxx.x.226:5060;branch=z9hG4bKa7a3.2fc757f5.0;rport=5060#015#012Max-Forwards: 69#015#012From: <sip:50390909150@xxx.xxx.x.226>;tag=FQH9vNpjUD89p#015#012To: <sip:50331111944@xxx.xxx.x.227>;tag=F5m876094y75B#015#012Call-ID: 4e565b83-496e-1236-f5b6-a0369f590b64#015#012CSeq: 115301882 INVITE#015#012User-Agent: SBCSWITCH#015#012Accept: application/sdp#015#012Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REFER, NOTIFY#015#012Supported: timer, path, replaces#015#012Allow-Events: talk, hold, conference, refer#015#012Reason: Q.850;cause=41;text="NORMAL_TEMPORARY_FAILURE"#015#012Content-Length: 0</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7231]: WARNING:core:log_expiry: #1 is a module action : t_reply - 106282us - line 1112</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7238]: WARNING:core:log_expiry: #5 is a core action : 69 - 8us - line 987</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7228]: WARNING:core:log_expiry: #2 is a core action : 82 - 184us - line 696</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7235]: WARNING:core:log_expiry: #1 is a module action : t_reply - 116776us - line 1112</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7236]: WARNING:core:log_expiry: #2 is a module action : search - 26us - line 906</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7221]: WARNING:core:log_expiry: #1 is a module action : t_reply - 68668us - line 1112</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7233]: WARNING:core:log_expiry: #1 is a module action : t_reply - 116606us - line 1112</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7231]: WARNING:core:log_expiry: #2 is a module action : rtpproxy_engage - 2175us - line 811</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7228]: WARNING:core:log_expiry: #3 is a module action : search - 81us - line 906</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7235]: WARNING:core:log_expiry: #2 is a module action : search - 31us - line 906</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7236]: WARNING:core:log_expiry: #3 is a core action : 52 - 17us - line 957</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7221]: WARNING:core:log_expiry: #2 is a module action : t_check_status - 42us - line 879</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7233]: WARNING:core:log_expiry: #2 is a module action : search - 101us - line 906</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7231]: WARNING:core:log_expiry: #3 is a core action : 83 - 340us - line 764</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7228]: WARNING:core:log_expiry: #4 is a module action : t_check_status - 27us - line 956</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7235]: WARNING:core:log_expiry: #3 is a core action : 52 - 17us - line 957</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7236]: WARNING:core:log_expiry: #4 is a module action : t_check_status - 16us - line 879</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7221]: WARNING:core:log_expiry: #3 is a module action : search - 30us - line 906</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7233]: WARNING:core:log_expiry: #3 is a module action : t_check_status - 43us - line 956</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7231]: WARNING:core:log_expiry: #4 is a core action : 82 - 189us - line 696</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7228]: WARNING:core:log_expiry: #5 is a module action : t_check_status - 19us - line 1111</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7235]: WARNING:core:log_expiry: #4 is a module action : t_check_status - 14us - line 879</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7236]: WARNING:core:log_expiry: #5 is a core action : 69 - 8us - line 987</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7221]: WARNING:core:log_expiry: #4 is a module action : t_check_status - 19us - line 1111</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7233]: WARNING:core:log_expiry: #4 is a module action : t_check_status - 41us - line 1111</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7231]: WARNING:core:log_expiry: #5 is a module action : t_relay - 60us - line 843</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7235]: WARNING:core:log_expiry: #5 is a module action : t_check_status - 11us - line 956</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7221]: WARNING:core:log_expiry: #5 is a core action : 52 - 16us - line 957</div><div>Nov 21 14:51:33 OS /usr/local/sbin/opensips[7233]: WARNING:core:log_expiry: #5 is a core action : 69 - 18us - line 987</div><div>Nov 21 15:03:03 OS /usr/local/sbin/opensips[7216]: WARNING:core:handle_timer_job: utimer job <tm-utimer> has a 60000 us delay in execution</div><div><br></div></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><b>Diptesh Patel</b><div>Jr. Software Developer </div><div>Ecosmob Technologies Ltd</div><div>Ahmedabad</div><div>Mo:<b>+919898962659</b></div></div></div></div>
<br><div class="gmail_quote">On Tue, Nov 21, 2017 at 8:34 PM, Răzvan Crainea <span dir="ltr"><<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<tt>Hello!<br>
<br>
I see that you have configured different thresholds in your
script. Do you get any alerts from those thresholds?<br>
<br>
Also, how many concurrent calls are there when you are
experiencing this?<br>
<br>
Best regards,<br>
Răzvan<br>
</tt>
<pre class="m_6018427087198510989moz-signature" cols="72">Răzvan Crainea
OpenSIPS Developer
<a class="m_6018427087198510989moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre><div><div class="h5">
<div class="m_6018427087198510989moz-cite-prefix">On 11/21/2017 03:08 PM, Dipteshkumar
Patel wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<div dir="ltr">
<div>hello all</div>
<div><br>
</div>
<div>I am using opensips-2.3.2 running on 16 cores of cpu and
32GB of RAM</div>
<div><br>
</div>
<div>I am facing an isssue. After making 100k of calls, suddenly
opensips performance degrades and it takes much time to
process a packet. load average of cpu incresing
significantly.after restarting opensips it works fine for
another 100k calls then again problem appears.</div>
<div><br>
</div>
<div>It is an SBC server, using 6-7 sql queries with async with
database optimisation(using proper indexing), permission
module for auth, aaa_radius module for accounting and some
string operations.</div>
<div><br>
</div>
<div>Summary of operations</div>
<div>1. Sanity checks</div>
<div>2. Use IP authentication </div>
<div>3. Send Requet for accounting</div>
<div>4. Some string operations for accounting</div>
<div> </div>
<div>I am sending global parameters and module parameters of my
opensips config file</div>
<div><br>
</div>
<div>Please find here <a href="https://pastebin.com/B7Y5HRgH" target="_blank">https://pastebin.com/<wbr>B7Y5HRgH</a></div>
<div><br>
</div>
<div>Please suggest what is wrong and how to improve.</div>
<div><br>
</div>
<div>
<div class="m_6018427087198510989gmail_signature">
<div dir="ltr"><b>Diptesh Patel</b>
<div>Jr. Software Developer </div>
<div>Ecosmob Technologies Ltd</div>
<div>Ahmedabad</div>
<div>Mo:<b>+919898962659</b></div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_6018427087198510989mimeAttachmentHeader"></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
Users mailing list
<a class="m_6018427087198510989moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="m_6018427087198510989moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>