<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<font face="monospace">Somehow I think there is a confusion - the
drop_requests stat has nothing to do with the dropping on the
socket buffers (net level).<br>
<br>
Regards,<br>
</font>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
<a class="moz-txt-link-freetext" href="https://www.siphub.com">https://www.siphub.com</a></pre>
<div class="moz-cite-prefix">On 19.04.2024 11:13, johan wrote:<br>
</div>
<blockquote type="cite"
cite="mid:3aa2b48f-fa7d-4b66-ae4a-55fffc139409@democon.be">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Bogdan, <br>
</p>
<p>on the augmenting drop_requests, <br>
</p>
<p>drop is not used in that script. Neither is forward.
Everything is t_relay. There is no b2b. <br>
</p>
<p>The thing is that we observed drops on udp level. </p>
<p>We followed the recommendations of Ovidiu Sas's presentation
of last year in Houston: <br>
</p>
<p>- increase PKG mem <br>
</p>
<p>- increase SHM mem</p>
<p>- increase workers to 24 so that the queue empties faster. <br>
</p>
<p>- we checked the udp queues on linux level and we saw drops
there. <br>
</p>
<p> => we augmented them to 50 megs (sysctl -w
net.core.rmem_max=52428800 and sysctl -w
net.core.rmem_default=52428800) and the drops on OS level where
gone. <br>
</p>
<p>Also worker and memory load are max 30 %. <br>
</p>
<p><br>
</p>
<p>Hence we thought that we were okay, but still drops on opensips
level. Net result was that this node in the system lost all
connection with the destination of the loadbalancer although it
received keep alive options responses from the loadbalancer
destination on its NIC (we could see that in a continuously
running tcpdump). <br>
</p>
<p>=> hence it seems that it is opensips's receive buffer that
is too small (as I read the description : "<span
style="color: rgb(102, 102, 102); font-family: "Trebuchet MS", sans-serif, Verdana, Arial; font-size: 13.008px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Returns
the number of requests dropped even before entering the script
routing logic.</span>", I thought that this pointed to the
receive buffer of opensips). All of this is happening on a
physical machine on which two other opensips instances are
running also. Interestingly enough the problem is only observed
in the instance that handles registrations and invites (1600
REG/ s and 300 INV /s). <br>
</p>
<p><br>
</p>
<p>Therefore we dived a bit deeper and came on this <span
style="font-size:12pt">MAX_RECV_BUFFER_SIZE 262144 (which is
the default udp queue size setting on linux). Could this be
related somehow ? <br>
</span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><span style="font-size:12pt">Secondly, what would the
recommendation be for scaling a system like this ? <br>
</span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 18/04/2024 16:29, Bogdan-Andrei
Iancu wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b3437984-9f15-4937-8ce0-12aab465ee7b@opensips.org">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8">
The `<span style="font-size:12pt">drop_requests` statistic is
incremented when:<br>
* the request is dropped by a pre-script callback (like B2B
when there is no script execution for certain messages)<br>
* the stateless `forward()` core function failed to send out
something.<br>
<br>
Regards,<br>
</span>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext"
href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a>
<a class="moz-txt-link-freetext" href="https://www.siphub.com"
moz-do-not-send="true">https://www.siphub.com</a></pre>
<div class="moz-cite-prefix">On 18.04.2024 17:19, Johan De
Clercq wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEVWGm_SCK+E0pwc8DAo0eW9g6VjSEC_hRjtVFCSvjtBQhTyww@mail.gmail.com">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">No I don't.
<div>what I find strange is that MAX_RECV_BUFFER_SIZE
262144 is the default value of net.core.rmem_max and
net.core.rmem_default. </div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Op do 18 apr 2024 om
16:02 schreef Ben Newlin <<a
href="mailto:Ben.Newlin@genesys.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Ben.Newlin@genesys.com</a>>:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="EN-US">
<div>
<p class="MsoNormal"><span
style="font-size:11pt">Are you calling
drop() anywhere in your script?</span></p>
<p class="MsoNormal"><span
style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt"><a
href="https://www.opensips.org/Documentation/Script-CoreFunctions-3-4#toc13"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.opensips.org/Documentation/Script-CoreFunctions-3-4#toc13</a></span></p>
<p class="MsoNormal"><span
style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:black">Ben
Newlin</span><span style="font-size:11pt"></span></p>
<p class="MsoNormal"><span
style="font-size:11pt"> </span></p>
<div
id="m_5205456487139469700m_-318613669772535148mail-editor-reference-message-container">
<div>
<div
style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"
style="margin-bottom:12pt"><b><span
style="font-size:12pt;color:black">From:
</span></b><span
style="font-size:12pt;color:black">Users
<<a
href="mailto:users-bounces@lists.opensips.org" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">users-bounces@lists.opensips.org</a>>
on behalf of Johan De Clercq <<a
href="mailto:Johan@democon.be"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">Johan@democon.be</a>><br>
<b>Date: </b>Thursday, April 18,
2024 at 5:27 AM<br>
<b>To: </b>OpenSIPS users mailling
list <<a
href="mailto:users@lists.opensips.org" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">users@lists.opensips.org</a>><br>
<b>Subject: </b>Re:
[OpenSIPS-Users] question on core
statistics.</span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<table
style="border-collapse:collapse;border:none" cellspacing="0"
cellpadding="0" border="1">
<tbody>
<tr>
<td
style="border:1pt solid rgb(182,0,0);background:white;padding:0.75pt">
<p class="MsoNormal"><b><span
style="font-size:12pt;font-family:Calibri,sans-serif;color:rgb(182,0,0)"> EXTERNAL
EMAIL - Please use
caution with links
and attachments </span></b></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal"><span
style="font-size:12pt;font-family:Helvetica"> </span></p>
<div class="MsoNormal"
style="text-align:center"
align="center"><span
style="font-size:12pt;font-family:Helvetica">
<hr width="100%" size="0"
align="center"> </span></div>
</div>
<p class="MsoNormal"><span
style="font-size:12pt">would it
make sense to recompile with
other flags ? And how do I set
them (I don't find these of
menuconfig's compile options)? </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Currently
it has MAX_RECV_BUFFER_SIZE
262144 and BUF_SIZE 65535. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Can
somebody explain also what both
flags mean. </span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Op do 18
apr 2024 om 11:07 schreef Johan De
Clercq <<a
href="mailto:Johan@democon.be"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">Johan@democon.be</a>>:</span></p>
</div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">would
it make sense to recompile
with other flags ? </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Currently
it has MAX_RECV_BUFFER_SIZE
262144 and BUF_SIZE 65535. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Can
somebody explain also what
both flags mean. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">flags:
STATS: On,
DISABLE_NAGLE,
USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE
262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024,
BUF_SIZE 65535</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">poll
method support: poll,
epoll, sigio_rt, select.</span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Op do
18 apr 2024 om 10:32 schreef
Johan De Clercq <<a
href="mailto:Johan@democon.be" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Johan@democon.be</a>>:</span></p>
</div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Guys, </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">I
have an opensips
instance running with
24 worker children. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">The
worker load is very
low. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">UDP
queues are on 50
megs. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">when
i query via the OS </span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">cat /proc/net/udp</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> sl local_address rem_address st tx_queue
rx_queue tr
tm->when
retrnsmt uid
timeout inode ref
pointer drops
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 590: 03231D0A:13C4 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 413684019 2
ffff880074820bc0 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 591: 03231D0A:13C5 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 413766438 2
ffff880465e4a440 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 592: 03231D0A:13C6 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 412035865 2
ffff8803e5a56b80 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 934: 01231D0A:151C 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 26790 2
ffff88046c054840 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 935: 0201FFEF:151D 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 26787 2
ffff88046c054bc0 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 935: 01231D0A:151D 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 26791 2
ffff88046c0544c0 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 1972: 00000000:D92A 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 15506 2
ffff88046dce5040 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 5479: 00000000:E6DD 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 22811 2
ffff880465e4ab40 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">12075: AA0914AC:00A1 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 20572 2
ffff88086d020800 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">12075: 0100007F:00A1 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 20571 2
ffff88086d020b80 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">13320: 00000000:857E 00000000:0000 07
00000000:00000000
00:00000000
00000000 100
0 17515 2
ffff8800368ac780 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">15661: 00000000:CEA3 00000000:0000 07
00000000:00000000
00:00000000
00000000 0
0 15505 2
ffff8800368acb00 0 </span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">=>
no drops</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">what
worries me is that
there are drop
requests and they go
up when I query via
the mi interface </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">opensipsctl
fifo get_statistics
drop_requests</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">core:drop_requests::
198107</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">opensipsctl
fifo get_statistics
drop_requests</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">core:drop_requests::
199157</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">opensipsctl_reg
fifo get_statistics
drop_requests</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">core:drop_requests::
204116</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">I
don't see any memory
issue, also the
processload is low. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">so
3 questions:</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">-
what exactly is
drop_request. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">-
do I need to worry
about this </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">-
how can I make them go
lower. </span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Users@lists.opensips.org</a><br>
<a
href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:Users@lists.opensips.org" moz-do-not-send="true">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users"
moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
</body>
</html>