<div dir="ltr">Here is the dump<div><a href="http://pastebin.com/DTEHF5Vc">http://pastebin.com/DTEHF5Vc</a><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <span dir="ltr"><<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<tt>None of the "actions" you are talking about have big impact on
private memory, but the shared one. Better do the dump and send it
over to point out what is "eating" memory.<br>
<br>
Best regards,<br>
</tt><span class="">
<pre class="m_-8335070280687897772moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a class="m_-8335070280687897772moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
</span><div><div class="h5"><div class="m_-8335070280687897772moz-cite-prefix">On 03/06/2017 02:39 PM, John Nash
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">with every call attempt it decreases. I tried some
changes by rejecting invite before drouting call (That means
after auth , dispatcher) and found memory is stable but when
drouting sends Invite to external gateway and external gateway
rejects it. Then this issue happens.
<div><br>
</div>
<div>Inuse transactions and active dialogs also 0. Somthing
wrong happening in handling of failure replies. But apart
from <span style="background-color:rgb(254,254,238);color:rgb(0,0,0)">use_next_gw
and setting some avps for CDR not much going on there.</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar 6, 2017 at 5:54 PM, Răzvan
Crainea <span dir="ltr"><<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <tt>Ok, so it is the
first listener for the private IP that leaks. Next, is
the memory stabilizing in time? Or it is continously
decreasing?<br>
Yes, that's how you should make the dump.<br>
<br>
Best regards,<br>
</tt><span>
<pre class="m_-8335070280687897772m_5518641707690872261moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a class="m_-8335070280687897772m_5518641707690872261moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
</span>
<div>
<div class="m_-8335070280687897772h5">
<div class="m_-8335070280687897772m_5518641707690872261moz-cite-prefix">On
03/06/2017 10:57 AM, John Nash wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dear Razvan,
<div><br>
</div>
<div>Below is the info on my processes</div>
<div>
<div>Process:: ID=0 PID=17351 Type=attendant</div>
<div>Process:: ID=1 PID=17352 Type=MI FIFO</div>
<div>Process:: ID=2 PID=17353 Type=MI Datagram</div>
<div>Process:: ID=3 PID=17354 Type=time_keeper</div>
<div>Process:: ID=4 PID=17355 Type=timer</div>
<div>Process:: ID=5 PID=17356 Type=SIP receiver
udp:<a href="http://1.1.1.1:9094" target="_blank">1.1.1.1:9094</a> </div>
<div>Process:: ID=6 PID=17357 Type=SIP receiver
udp:<a href="http://1.1.1.1:5060" target="_blank">1.1.1.1:5060</a> </div>
<div>Process:: ID=7 PID=17358 Type=SIP receiver
udp:<a href="http://192.168.45.5:5064" target="_blank">192.168.45.5:5064</a> </div>
<div>Process:: ID=8 PID=17359 Type=Timer
handler</div>
</div>
<div><br>
</div>
<div>1.1.1.1 is public IP (I changed). The
decrease in memory I see is for Process:: ID=7
PID=17358 mainly. My call flow is as following </div>
<div><br>
</div>
<div>- New Invite hits the opensips on <a href="http://1.1.1.1:9094" target="_blank">1.1.1.1:9094</a></div>
<div>- Apart from message validity checks I query
DB to check if its a valid user (Using local
cache also there)</div>
<div>- Create dialog, Topology_hiding functions
are called along with some avp population</div>
<div>- Using dispatcher ds_select_domain Call sent
to udp:<a href="http://192.168.45.2:7060" target="_blank">192.168.45.2:7060</a> (using
force socket). This <a href="http://192.168.45.2:7060" target="_blank">192.168.45.2:7060</a> is
actually freeswitch </div>
<div>- Call again comes back to opensips on udp:<a href="http://192.168.45.5:5064" target="_blank">192.168.45.5:5064</a></div>
<div>- New dialog is created and topology_hiding
is called</div>
<div>- Drouting function do_routing is called
which tries one gateway and fails<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Dump i need to create with memlog=4 memdump=1
right?</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar 6, 2017 at
2:05 PM, Răzvan Crainea <span dir="ltr"><<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <tt>Hi,
John!<br>
<br>
Transactions are stored in shared memory,
not in the private one. So the possible
leak you are facing its not related to
transactions.<br>
During runtime, OpenSIPS might resize some
internal structures, which may lead to
increase memory usage. However, after a
while, these allocations should stabilize</tt><tt>
.<br>
Can you post the output of the kill
-SIGUSR1 on pastebin so we can take a
look? Also, what type of process is the
one you are seeing the leak into? You can
find out using the 'opensipsctl ps'
command.<br>
<br>
Best regards,<br>
</tt>
<pre class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-signature" cols="72">Răzvan Crainea
OpenSIPS Solutions
<a class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
<div>
<div class="m_-8335070280687897772m_5518641707690872261h5">
<div class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-cite-prefix">On
03/06/2017 09:55 AM, John Nash wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="m_-8335070280687897772m_5518641707690872261h5">
<div dir="ltr">I am using OpenSIPS
(2.1.5 (x86_64/linux)) in
production. I observed private
memory is decreasing constantly for
one process mainly and ultimately
leading to memory errors and crash.
<div><br>
</div>
<div>To debug this issue I prepared
a test server and compiled
opensips as per <a href="https://www.opensips.org/Documentation/TroubleShooting-OutOfMem" target="_blank">https://www.opensips.org/D<wbr>ocumentation/TroubleShooting-O<wbr>utOfMem</a></div>
<div><br>
</div>
<div>I made only one single call
(which was rejected by opensips as
it was not authorized user) and I
saw private free memory decreased.
I was hoping since transaction is
done ideally it should release
memory and should show me same
memory as startup but it did not.
I verified this with many call
attempts and i see free memory is
always decreasing slowly.</div>
<div><br>
</div>
<div>I used kill -SIGUSR1 <lowest
pid> to create memory dump. But
i am unable to make sense of it.
It shows log like ...</div>
<div><br>
</div>
<div>r 6 07:29:19 Server3021
opensips[13276]: Memory status
(pkg):</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: qm_status
(0x7f5b8ebba010):</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: heap size=
4194304</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: used= 346768,
used+overhead=848792, free=3345512</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: max used
(+overhead)= 931920</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: dumping all
alloc'ed. fragments:</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: 0. N
address=0x7f5b8ebef528
frag=0x7f5b8ebef4f8 size=40 used=1</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]:
alloc'd from script_cb.c:
add_callback(60)</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: start
check=f0f0f0f0f0f0f0f0, end check=
c0c0c0c0c0c0c0c0, abcdefedabcdefed</div>
<div>Mar 6 07:29:19 Server3021
opensips[13276]: 1. N
address=0x7f5b8ebef5b0 <br>
</div>
<div><br>
</div>
<div>I pasted only few lines in this
mail. What should be my next
step?...How can i really trace
what is wrong in my script or any
other memory leak?</div>
<div><br>
</div>
</div>
<br>
<fieldset class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
Users mailing list
<a class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="m_-8335070280687897772m_5518641707690872261m_2485402866099392609moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
______________________________<wbr>_________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</blockquote></div>
</div>
<fieldset class="m_-8335070280687897772m_5518641707690872261mimeAttachmentHeader"></fieldset>
<pre>______________________________<wbr>_________________
Users mailing list
<a class="m_-8335070280687897772m_5518641707690872261moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="m_-8335070280687897772m_5518641707690872261moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</blockquote></div>
</div>
<fieldset class="m_-8335070280687897772mimeAttachmentHeader"></fieldset>
<pre>______________________________<wbr>_________________
Users mailing list
<a class="m_-8335070280687897772moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="m_-8335070280687897772moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div></div></div><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>