<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Bogdan, <br>
</p>
<p>on the augmenting drop_requests, <br>
</p>
<p>drop is not used in that script. Neither is forward. Everything
is t_relay. There is no b2b. <br>
</p>
<p>The thing is that we observed drops on udp level. </p>
<p>We followed the recommendations of Ovidiu Sas's presentation of
last year in Houston: <br>
</p>
<p>- increase PKG mem <br>
</p>
<p>- increase SHM mem</p>
<p>- increase workers to 24 so that the queue empties faster. <br>
</p>
<p>- we checked the udp queues on linux level and we saw drops
there. <br>
</p>
<p> => we augmented them to 50 megs (sysctl -w
net.core.rmem_max=52428800 and sysctl -w
net.core.rmem_default=52428800) and the drops on OS level where
gone. <br>
</p>
<p>Also worker and memory load are max 30 %. <br>
</p>
<p><br>
</p>
<p>Hence we thought that we were okay, but still drops on opensips
level. Net result was that this node in the system lost all
connection with the destination of the loadbalancer although it
received keep alive options responses from the loadbalancer
destination on its NIC (we could see that in a continuously
running tcpdump). <br>
</p>
<p>=> hence it seems that it is opensips's receive buffer that is
too small (as I read the description : "<span
style="color: rgb(102, 102, 102); font-family: "Trebuchet MS", sans-serif, Verdana, Arial; font-size: 13.008px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Returns
the number of requests dropped even before entering the script
routing logic.</span>", I thought that this pointed to the
receive buffer of opensips). All of this is happening on a
physical machine on which two other opensips instances are running
also. Interestingly enough the problem is only observed in the
instance that handles registrations and invites (1600 REG/ s and
300 INV /s). <br>
</p>
<p><br>
</p>
<p>Therefore we dived a bit deeper and came on this <span
style="font-size:12pt">MAX_RECV_BUFFER_SIZE 262144 (which is the
default udp queue size setting on linux). Could this be related
somehow ? <br>
</span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><span style="font-size:12pt">Secondly, what would the
recommendation be for scaling a system like this ? <br>
</span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 18/04/2024 16:29, Bogdan-Andrei
Iancu wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b3437984-9f15-4937-8ce0-12aab465ee7b@opensips.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
The `<span style="font-size:12pt">drop_requests` statistic is
incremented when:<br>
* the request is dropped by a pre-script callback (like B2B when
there is no script execution for certain messages)<br>
* the stateless `forward()` core function failed to send out
something.<br>
<br>
Regards,<br>
</span>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext"
href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a>
<a class="moz-txt-link-freetext" href="https://www.siphub.com"
moz-do-not-send="true">https://www.siphub.com</a></pre>
<div class="moz-cite-prefix">On 18.04.2024 17:19, Johan De Clercq
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEVWGm_SCK+E0pwc8DAo0eW9g6VjSEC_hRjtVFCSvjtBQhTyww@mail.gmail.com">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">No I don't.
<div>what I find strange is that MAX_RECV_BUFFER_SIZE
262144 is the default value of net.core.rmem_max and
net.core.rmem_default. </div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Op do 18 apr 2024 om
16:02 schreef Ben Newlin <<a
href="mailto:Ben.Newlin@genesys.com" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">Ben.Newlin@genesys.com</a>>:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="EN-US">
<div>
<p class="MsoNormal"><span
style="font-size:11pt">Are you calling
drop() anywhere in your script?</span></p>
<p class="MsoNormal"><span
style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt"><a
href="https://www.opensips.org/Documentation/Script-CoreFunctions-3-4#toc13"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.opensips.org/Documentation/Script-CoreFunctions-3-4#toc13</a></span></p>
<p class="MsoNormal"><span
style="font-size:11pt"> </span></p>
<p class="MsoNormal"><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:black">Ben
Newlin</span><span style="font-size:11pt"></span></p>
<p class="MsoNormal"><span
style="font-size:11pt"> </span></p>
<div
id="m_5205456487139469700m_-318613669772535148mail-editor-reference-message-container">
<div>
<div
style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"
style="margin-bottom:12pt"><b><span
style="font-size:12pt;color:black">From:
</span></b><span
style="font-size:12pt;color:black">Users
<<a
href="mailto:users-bounces@lists.opensips.org" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">users-bounces@lists.opensips.org</a>>
on behalf of Johan De Clercq <<a
href="mailto:Johan@democon.be"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">Johan@democon.be</a>><br>
<b>Date: </b>Thursday, April 18, 2024
at 5:27 AM<br>
<b>To: </b>OpenSIPS users mailling
list <<a
href="mailto:users@lists.opensips.org" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">users@lists.opensips.org</a>><br>
<b>Subject: </b>Re: [OpenSIPS-Users]
question on core statistics.</span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<table
style="border-collapse:collapse;border:none" cellspacing="0"
cellpadding="0" border="1">
<tbody>
<tr>
<td
style="border:1pt solid rgb(182,0,0);background:white;padding:0.75pt">
<p class="MsoNormal"><b><span
style="font-size:12pt;font-family:Calibri,sans-serif;color:rgb(182,0,0)"> EXTERNAL
EMAIL - Please use
caution with links
and attachments </span></b></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal"><span
style="font-size:12pt;font-family:Helvetica"> </span></p>
<div class="MsoNormal"
style="text-align:center"
align="center"><span
style="font-size:12pt;font-family:Helvetica">
<hr width="100%" size="0"
align="center"> </span></div>
</div>
<p class="MsoNormal"><span
style="font-size:12pt">would it
make sense to recompile with other
flags ? And how do I set them (I
don't find these of
menuconfig's compile options)? </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Currently
it has MAX_RECV_BUFFER_SIZE 262144
and BUF_SIZE 65535. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Can
somebody explain also what both
flags mean. </span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Op do 18 apr
2024 om 11:07 schreef Johan De
Clercq <<a
href="mailto:Johan@democon.be"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">Johan@democon.be</a>>:</span></p>
</div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">would
it make sense to recompile
with other flags ? </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Currently
it has MAX_RECV_BUFFER_SIZE
262144 and BUF_SIZE 65535. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Can
somebody explain also what
both flags mean. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">flags:
STATS: On, DISABLE_NAGLE,
USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE
262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024,
BUF_SIZE 65535</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">poll
method support: poll,
epoll, sigio_rt, select.</span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Op do 18
apr 2024 om 10:32 schreef Johan
De Clercq <<a
href="mailto:Johan@democon.be"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">Johan@democon.be</a>>:</span></p>
</div>
<blockquote
style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Guys, </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">I
have an opensips
instance running with 24
worker children. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">The
worker load is very
low. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">UDP
queues are on 50 megs. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">when
i query via the OS </span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">cat
/proc/net/udp</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">
sl local_address
rem_address st
tx_queue rx_queue tr
tm->when retrnsmt
uid timeout inode
ref pointer drops
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">
590: 03231D0A:13C4
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0
413684019 2
ffff880074820bc0 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">
591: 03231D0A:13C5
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0
413766438 2
ffff880465e4a440 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">
592: 03231D0A:13C6
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0
412035865 2
ffff8803e5a56b80 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">
934: 01231D0A:151C
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 26790 2
ffff88046c054840 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">
935: 0201FFEF:151D
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 26787 2
ffff88046c054bc0 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">
935: 01231D0A:151D
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 26791 2
ffff88046c0544c0 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 1972:
00000000:D92A
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 15506 2
ffff88046dce5040 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> 5479:
00000000:E6DD
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 22811 2
ffff880465e4ab40 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">12075:
AA0914AC:00A1
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 20572 2
ffff88086d020800 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">12075:
0100007F:00A1
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 20571 2
ffff88086d020b80 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">13320:
00000000:857E
00000000:0000 07
00000000:00000000
00:00000000 00000000
100 0 17515 2
ffff8800368ac780 0
</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">15661:
00000000:CEA3
00000000:0000 07
00000000:00000000
00:00000000 00000000
0 0 15505 2
ffff8800368acb00 0 </span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">=>
no drops</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">what
worries me is that there
are drop requests and
they go up when I query
via the mi interface </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">opensipsctl
fifo get_statistics
drop_requests</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">core:drop_requests::
198107</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">opensipsctl
fifo get_statistics
drop_requests</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">core:drop_requests::
199157</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">opensipsctl_reg
fifo get_statistics
drop_requests</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">core:drop_requests::
204116</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">I
don't see any memory
issue, also the
processload is low. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">so
3 questions:</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">-
what exactly is
drop_request. </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">-
do I need to worry about
this </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">-
how can I make them go
lower. </span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Users@lists.opensips.org</a><br>
<a
href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:Users@lists.opensips.org" moz-do-not-send="true">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users"
moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</blockquote>
</body>
</html>