<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hi, John!<br>
<br>
From the dump you sent, I don't see any leaks. Perhaps some of
those fragments increase over time. Can you make a memory dump
after the server runs some time, like after it gets 100 messages?<br>
<br>
Best regards,<br>
</tt>
<pre class="moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a class="moz-txt-link-abbreviated" href="http://www.opensips-solutions.com">www.opensips-solutions.com</a></pre>
<div class="moz-cite-prefix">On 03/06/2017 03:02 PM, John Nash
wrote:<br>
</div>
<blockquote
cite="mid:CAA4rGPxzhzDGtXcfecg_Pysr5rA2_Qm4iEgSdmGZ-c6R1DSA1w@mail.gmail.com"
type="cite">
<div dir="ltr">Here is the dump
<div><a moz-do-not-send="true"
href="http://pastebin.com/DTEHF5Vc">http://pastebin.com/DTEHF5Vc</a><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar 6, 2017 at 6:20 PM, Răzvan
Crainea <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <tt>None of the
"actions" you are talking about have big impact on
private memory, but the shared one. Better do the dump
and send it over to point out what is "eating" memory.<br>
<br>
Best regards,<br>
</tt><span class="">
<pre class="m_-8335070280687897772moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a moz-do-not-send="true" class="m_-8335070280687897772moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
</span>
<div>
<div class="h5">
<div class="m_-8335070280687897772moz-cite-prefix">On
03/06/2017 02:39 PM, John Nash wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">with every call attempt it decreases.
I tried some changes by rejecting invite before
drouting call (That means after auth , dispatcher)
and found memory is stable but when drouting sends
Invite to external gateway and external gateway
rejects it. Then this issue happens.
<div><br>
</div>
<div>Inuse transactions and active dialogs also 0.
Somthing wrong happening in handling of failure
replies. But apart from <span
style="background-color:rgb(254,254,238);color:rgb(0,0,0)">use_next_gw
and setting some avps for CDR not much going
on there.</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar 6, 2017 at
5:54 PM, Răzvan Crainea <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:razvan@opensips.org"
target="_blank">razvan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <tt>Ok,
so it is the first listener for the
private IP that leaks. Next, is the memory
stabilizing in time? Or it is continously
decreasing?<br>
Yes, that's how you should make the dump.<br>
<br>
Best regards,<br>
</tt><span>
<pre class="m_-8335070280687897772m_5518641707690872261moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a moz-do-not-send="true" class="m_-8335070280687897772m_5518641707690872261moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
</span>
<div>
<div class="m_-8335070280687897772h5">
<div
class="m_-8335070280687897772m_5518641707690872261moz-cite-prefix">On
03/06/2017 10:57 AM, John Nash wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dear Razvan,
<div><br>
</div>
<div>Below is the info on my
processes</div>
<div>
<div>Process:: ID=0 PID=17351
Type=attendant</div>
<div>Process:: ID=1 PID=17352
Type=MI FIFO</div>
<div>Process:: ID=2 PID=17353
Type=MI Datagram</div>
<div>Process:: ID=3 PID=17354
Type=time_keeper</div>
<div>Process:: ID=4 PID=17355
Type=timer</div>
<div>Process:: ID=5 PID=17356
Type=SIP receiver udp:<a
moz-do-not-send="true"
href="http://1.1.1.1:9094"
target="_blank">1.1.1.1:9094</a> </div>
<div>Process:: ID=6 PID=17357
Type=SIP receiver udp:<a
moz-do-not-send="true"
href="http://1.1.1.1:5060"
target="_blank">1.1.1.1:5060</a> </div>
<div>Process:: ID=7 PID=17358
Type=SIP receiver udp:<a
moz-do-not-send="true"
href="http://192.168.45.5:5064"
target="_blank">192.168.45.5:5064</a> </div>
<div>Process:: ID=8 PID=17359
Type=Timer handler</div>
</div>
<div><br>
</div>
<div>1.1.1.1 is public IP (I
changed). The decrease in memory I
see is for Process:: ID=7
PID=17358 mainly. My call flow is
as following </div>
<div><br>
</div>
<div>- New Invite hits the opensips
on <a moz-do-not-send="true"
href="http://1.1.1.1:9094"
target="_blank">1.1.1.1:9094</a></div>
<div>- Apart from message validity
checks I query DB to check if its
a valid user (Using local cache
also there)</div>
<div>- Create dialog,
Topology_hiding functions are
called along with some avp
population</div>
<div>- Using dispatcher
ds_select_domain Call sent to udp:<a
moz-do-not-send="true"
href="http://192.168.45.2:7060"
target="_blank">192.168.45.2:7060</a>
(using force socket). This <a
moz-do-not-send="true"
href="http://192.168.45.2:7060"
target="_blank">192.168.45.2:7060</a> is
actually freeswitch </div>
<div>- Call again comes back to
opensips on udp:<a
moz-do-not-send="true"
href="http://192.168.45.5:5064"
target="_blank">192.168.45.5:5064</a></div>
<div>- New dialog is created and
topology_hiding is called</div>
<div>- Drouting function do_routing
is called which tries one gateway
and fails<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Dump i need to create with
memlog=4 memdump=1 right?</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar
6, 2017 at 2:05 PM, Răzvan Crainea
<span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:razvan@opensips.org"
target="_blank">razvan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> <tt>Hi, John!<br>
<br>
Transactions are stored in
shared memory, not in the
private one. So the possible
leak you are facing its not
related to transactions.<br>
During runtime, OpenSIPS
might resize some internal
structures, which may lead
to increase memory usage.
However, after a while,
these allocations should
stabilize</tt><tt> .<br>
Can you post the output of
the kill -SIGUSR1 on
pastebin so we can take a
look? Also, what type of
process is the one you are
seeing the leak into? You
can find out using the
'opensipsctl ps' command.<br>
<br>
Best regards,<br>
</tt>
<pre class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a moz-do-not-send="true" class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
<div>
<div
class="m_-8335070280687897772m_5518641707690872261h5">
<div
class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-cite-prefix">On
03/06/2017 09:55 AM,
John Nash wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div
class="m_-8335070280687897772m_5518641707690872261h5">
<div dir="ltr">I am
using OpenSIPS (2.1.5
(x86_64/linux)) in
production. I observed
private memory is
decreasing constantly
for one process mainly
and ultimately leading
to memory errors and
crash.
<div><br>
</div>
<div>To debug this
issue I prepared a
test server and
compiled opensips as
per <a
moz-do-not-send="true"
href="https://www.opensips.org/Documentation/TroubleShooting-OutOfMem"
target="_blank">https://www.opensips.org/D<wbr>ocumentation/TroubleShooting-O<wbr>utOfMem</a></div>
<div><br>
</div>
<div>I made only one
single call (which
was rejected by
opensips as it was
not authorized user)
and I saw private
free memory
decreased. I was
hoping since
transaction is done
ideally it should
release memory and
should show me same
memory as startup
but it did not. I
verified this with
many call attempts
and i see free
memory is always
decreasing slowly.</div>
<div><br>
</div>
<div>I used kill
-SIGUSR1 <lowest
pid> to create
memory dump. But i
am unable to make
sense of it. It
shows log like ...</div>
<div><br>
</div>
<div>r 6 07:29:19
Server3021
opensips[13276]:
Memory status (pkg):</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
qm_status
(0x7f5b8ebba010):</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
heap size= 4194304</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
used= 346768,
used+overhead=848792,
free=3345512</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
max used
(+overhead)= 931920</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
dumping all
alloc'ed. fragments:</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
0. N
address=0x7f5b8ebef528
frag=0x7f5b8ebef4f8
size=40 used=1</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
alloc'd from
script_cb.c:
add_callback(60)</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
start
check=f0f0f0f0f0f0f0f0,
end check=
c0c0c0c0c0c0c0c0,
abcdefedabcdefed</div>
<div>Mar 6 07:29:19
Server3021
opensips[13276]:
1. N
address=0x7f5b8ebef5b0 <br>
</div>
<div><br>
</div>
<div>I pasted only few
lines in this mail.
What should be my
next step?...How can
i really trace what
is wrong in my
script or any other
memory leak?</div>
<div><br>
</div>
</div>
<br>
<fieldset
class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
Users mailing list
<a moz-do-not-send="true" class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a moz-do-not-send="true" class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
______________________________<wbr>_________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a moz-do-not-send="true" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</blockquote></div>
</div>
<fieldset class="m_-8335070280687897772m_5518641707690872261mimeAttachmentHeader"></fieldset>
<pre>______________________________<wbr>_________________
Users mailing list
<a moz-do-not-send="true" class="m_-8335070280687897772m_5518641707690872261moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a moz-do-not-send="true" class="m_-8335070280687897772m_5518641707690872261moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a moz-do-not-send="true" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</blockquote></div>
</div>
<fieldset class="m_-8335070280687897772mimeAttachmentHeader"></fieldset>
<pre>______________________________<wbr>_________________
Users mailing list
<a moz-do-not-send="true" class="m_-8335070280687897772moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a moz-do-not-send="true" class="m_-8335070280687897772moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a moz-do-not-send="true" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</blockquote></div>
</div>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</body></html>