<div dir="ltr">Hi Razvan,<div><br></div><div>Here is what I've done.</div><div><br></div><div>I took a core dump of the attendant process. Then, I stopped OpenSIPS so that it frees allocated fragments, and at the end lists all fragments that are still allocated.</div><div><br></div><div>In this list of fragments, I can see a lot of AVPs.</div><div><br></div><div>I see some AVPs without data (avp->data == NULL, avp->flags == 0). But something is weird, it looks like that all AVPs that don't have data have the same id. It looks like duplicate AVPs (in different memory fragments).</div><div><br></div><div>Some AVPs do have data and have a format that looks valid.</div><div><br></div><div>Some AVPs looks corrupted. For example, I found an AVP with same ID as the AVPs without data, but avp->data == 0x8000 which looks completely wrong.</div><div><br></div><div>Thanks,</div><div>Mickael</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 18, 2015 at 10:11 AM, Mickael Marrache <span dir="ltr"><<a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Razvan,<div><br></div><div>I created a core dump for the attendant process. Is it enough or we also need core dumps for other processes? Note that the leak appears in shared memory.</div><div><br></div><div>We do use QM debug for this version, this is how I discovered the remaining AVPs at shutdown where the remaining allocated memory fragments are listed.</div><div><br></div><div>Do you know where I should look for the AVPs in the core dump?</div><div><br></div><div>Thanks,</div><div>Mickael </div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 16, 2015 at 5:11 PM, Răzvan Crainea <span dir="ltr"><<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<tt>Hi, Mickael!<br>
<br>
I don't know what exactly might cause the leak. What you can do is
to try to get a core dump (using tools like gcore) during low (or
unexisting) traffic and try to see what do the AVPs that are
leaking contain. Are you using QM debug?<br>
<br>
Best regards,<br>
</tt>
<pre cols="72">Răzvan Crainea
OpenSIPS Solutions
<a href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre><div><div>
<div>On 05/27/2015 12:37 PM, Mickael
Marrache wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Any idea? Is there something that may help finding
the leak cause?</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May 19, 2015 at 1:17 PM,
Mickael Marrache <span dir="ltr"><<a href="mailto:mickaelmarrache@gmail.com" target="_blank"></a><a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Any idea what we should do?
<div><br>
</div>
<div>I may be doing something wrong but I don't remember I
had to take care of memory management when dealing with
AVPs.</div>
<div><br>
</div>
<div>Am I right?</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, May 18, 2015 at 12:08
PM, Mickael Marrache <span dir="ltr"><<a href="mailto:mickaelmarrache@gmail.com" target="_blank"></a><a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>We use customized sources based on 1.11.2.</div>
<div><br>
</div>
<div>However, we never modified the sources
related to AVP.</div>
<span><font color="#888888">
<div><br>
</div>
<div>Mickael</div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, May 18,
2015 at 12:03 PM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <tt>Hi
Mickael,<br>
<br>
If you do not have any SIP message
or transaction under processing, and
still have AVPs in memory....well,
it looks like a leak to me.<br>
<br>
What OpenSIPS version do you have ?<br>
<br>
Regards,<br>
</tt><span>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
</span>
<div>
<div>
<div>On 17.05.2015 11:55, Mickael
Marrache wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>Here is what I've done:</div>
<div><br>
</div>
<div>1) Ensured OpenSIPS
doesn't receive any calls.</div>
<div>2) Waited for all calls
to terminate.</div>
<div>3) Checked that the <span style="font-family:monospace;font-size:10.4000005722046px">inuse</span><tt style="font-size:10.4000005722046px">_tra<tt>nsaction</tt></tt> is 0.</div>
<div>4) Stopped OpenSIPS.</div>
<div><br>
</div>
<div>I confirm there are still
a lot of AVPs in memory.</div>
<div><br>
</div>
<div>Who is supposed to free
these AVPs? Is it a memory
leak?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Mickael</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Fri, May 15, 2015 at 12:48
PM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank"></a><a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <tt>Hi
Michael,<br>
<br>
<tt>You can do
"opensipsctl fifo
get<tt>_statistics
tm:" and you have
the "inuse<tt>_tra<tt>nsaction"
statistic :<br>
<tt> <a href="http://www.opensips.org/html/docs/modules/1.11.x/tm.html#id296307" target="_blank"></a><a href="http://www.opensips.org/html/docs/modules/1.11.x/tm.html#id296307" target="_blank">http://www.opensips.org/html/docs/modules/1.11.x/tm.html#id296307</a><br>
<br>
<tt>Regards,</tt><br>
<br>
</tt></tt></tt></tt></tt></tt><span>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
</span>
<div>
<div>
<div>On 14.05.2015
09:45, Mickael
Marrache wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span style="color:#1f497d" lang="EN-US">I'm not sure there were no
transactions
in memory at
shutdown.</span></p>
<p class="MsoNormal"><span style="color:#1f497d" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="color:#1f497d" lang="EN-US">How can I check this? I didn't find
an MI command
for that.</span></p>
<p class="MsoNormal"><span style="color:#1f497d" lang="EN-US"> </span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
Bogdan-Andrei
Iancu [<a href="mailto:bogdan@opensips.org" target="_blank"></a><a href="mailto:bogdan@opensips.org" target="_blank">mailto:bogdan@opensips.org</a>]
<br>
<b>Sent:</b>
Wednesday, May
13, 2015 6:44
PM<br>
<b>To:</b>
OpenSIPS users
mailling list;
<a href="mailto:mickaelmarrache@gmail.com" target="_blank"></a><a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a><br>
<b>Subject:</b>
Re:
[OpenSIPS-Users]
Possible
memory leak
related to
AVPs</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Mickael,</span></tt><span><br>
<br>
<tt>What means
"a lot" :) ?
I'm asking as
if you have
100
transactions
in memory
(when doing
shutdown) and
you use ~10
AVPs per
INVITE in
script ->
you have 1000
AVPs in mem.</tt><br>
<br>
<tt>When doing
the shutdown,
are any
transaction in
memory ? (with
no active
transactions
in mem, there
are no AVPs in
mem).</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
</span></p>
<pre>Bogdan-Andrei Iancu</pre>
<pre>OpenSIPS Founder and Developer</pre>
<pre><a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
<div>
<p class="MsoNormal">On
13.05.2015
17:53, Mickael
Marrache
wrote:</p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span lang="EN-US">We
have a memory
leak and are
trying to find
the cause.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">We
compiled the
call with
memory
debugging and
at shutdown,
we see a lot
of allocated
AVPs that are
not released.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">opensips[27560]:
277067. N
address=0x2aae957a0330
frag=0x2aae957a0300
size=24 used=1</span></p>
<p class="MsoNormal"><span lang="EN-US">opensips[27560]:
alloc'd from
usr_avp.c:
new_avp(119)</span></p>
<p class="MsoNormal"><span lang="EN-US">opensips[27560]:
start
check=f0f0f0f0f0f0f0f0,
end check=
c0c0c0c0c0c0c0c0,
abcdefedabcdefed</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Is
it possible to
have a memory
leak related
to AVPs? IIRC
OpenSIPS
handles it
internally.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks,</span></p>
<p class="MsoNormal"><span lang="EN-US">Mickael</span></p>
<p class="MsoNormal"><span><br>
<br>
<br>
</span></p>
<pre>_______________________________________________</pre>
<pre>Users mailing list</pre>
<pre><a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a></pre>
</blockquote>
<p class="MsoNormal"><span> </span></p>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>