<div dir="ltr">Hi Razvan,<div><br></div><div>Here is what I&#39;ve done.</div><div><br></div><div>I took a core dump of the attendant process. Then, I stopped OpenSIPS so that it frees allocated fragments, and at the end lists all fragments that are still allocated.</div><div><br></div><div>In this list of fragments, I can see a lot of AVPs.</div><div><br></div><div>I see some AVPs without data (avp-&gt;data == NULL, avp-&gt;flags == 0). But something is weird, it looks like that all AVPs that don&#39;t have data have the same id. It looks like duplicate AVPs (in different memory fragments).</div><div><br></div><div>Some AVPs do have data and have a format that looks valid.</div><div><br></div><div>Some AVPs looks corrupted. For example, I found an AVP with same ID as the AVPs without data, but avp-&gt;data == 0x8000 which looks completely wrong.</div><div><br></div><div>Thanks,</div><div>Mickael</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 18, 2015 at 10:11 AM, Mickael Marrache <span dir="ltr">&lt;<a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Razvan,<div><br></div><div>I created a core dump for the attendant process. Is it enough or we also need core dumps for other processes? Note that the leak appears in shared memory.</div><div><br></div><div>We do use QM debug for this version, this is how I discovered the remaining AVPs at shutdown where the remaining allocated memory fragments are listed.</div><div><br></div><div>Do you know where I should look for the AVPs in the core dump?</div><div><br></div><div>Thanks,</div><div>Mickael </div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 16, 2015 at 5:11 PM, Răzvan Crainea <span dir="ltr">&lt;<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <tt>Hi, Mickael!<br>
      <br>
      I don&#39;t know what exactly might cause the leak. What you can do is
      to try to get a core dump (using tools like gcore) during low (or
      unexisting) traffic and try to see what do the AVPs that are
      leaking contain. Are you using QM debug?<br>
      <br>
      Best regards,<br>
    </tt>
    <pre cols="72">Răzvan Crainea
OpenSIPS Solutions
<a href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre><div><div>
    <div>On 05/27/2015 12:37 PM, Mickael
      Marrache wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">Any idea? Is there something that may help finding
        the leak cause?</div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Tue, May 19, 2015 at 1:17 PM,
          Mickael Marrache <span dir="ltr">&lt;<a href="mailto:mickaelmarrache@gmail.com" target="_blank"></a><a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">Any idea what we should do?
              <div><br>
              </div>
              <div>I may be doing something wrong but I don&#39;t remember I
                had to take care of memory management when dealing with
                AVPs.</div>
              <div><br>
              </div>
              <div>Am I right?</div>
            </div>
            <div>
              <div>
                <div class="gmail_extra"><br>
                  <div class="gmail_quote">On Mon, May 18, 2015 at 12:08
                    PM, Mickael Marrache <span dir="ltr">&lt;<a href="mailto:mickaelmarrache@gmail.com" target="_blank"></a><a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div dir="ltr">Hi,
                        <div><br>
                        </div>
                        <div>We use customized sources based on 1.11.2.</div>
                        <div><br>
                        </div>
                        <div>However, we never modified the sources
                          related to AVP.</div>
                        <span><font color="#888888">
                            <div><br>
                            </div>
                            <div>Mickael</div>
                          </font></span></div>
                      <div>
                        <div>
                          <div class="gmail_extra"><br>
                            <div class="gmail_quote">On Mon, May 18,
                              2015 at 12:03 PM, Bogdan-Andrei Iancu <span dir="ltr">&lt;<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>&gt;</span>
                              wrote:<br>
                              <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                <div text="#000000" bgcolor="#FFFFFF"> <tt>Hi
                                    Mickael,<br>
                                    <br>
                                    If you do not have any SIP message
                                    or transaction under processing, and
                                    still have AVPs in memory....well,
                                    it looks like a leak to me.<br>
                                    <br>
                                    What OpenSIPS version do you have ?<br>
                                    <br>
                                    Regards,<br>
                                  </tt><span>
                                    <pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
                                  </span>
                                  <div>
                                    <div>
                                      <div>On 17.05.2015 11:55, Mickael
                                        Marrache wrote:<br>
                                      </div>
                                      <blockquote type="cite">
                                        <div dir="ltr">Hi,
                                          <div><br>
                                          </div>
                                          <div>Here is what I&#39;ve done:</div>
                                          <div><br>
                                          </div>
                                          <div>1) Ensured OpenSIPS
                                            doesn&#39;t receive any calls.</div>
                                          <div>2) Waited for all calls
                                            to terminate.</div>
                                          <div>3) Checked that the <span style="font-family:monospace;font-size:10.4000005722046px">inuse</span><tt style="font-size:10.4000005722046px">_tra<tt>nsaction</tt></tt> is 0.</div>
                                          <div>4) Stopped OpenSIPS.</div>
                                          <div><br>
                                          </div>
                                          <div>I confirm there are still
                                            a lot of AVPs in memory.</div>
                                          <div><br>
                                          </div>
                                          <div>Who is supposed to free
                                            these AVPs? Is it a memory
                                            leak?</div>
                                          <div><br>
                                          </div>
                                          <div>Thanks,</div>
                                          <div>Mickael</div>
                                        </div>
                                        <div class="gmail_extra"><br>
                                          <div class="gmail_quote">On
                                            Fri, May 15, 2015 at 12:48
                                            PM, Bogdan-Andrei Iancu <span dir="ltr">&lt;<a href="mailto:bogdan@opensips.org" target="_blank"></a><a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>&gt;</span>
                                            wrote:<br>
                                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                              <div text="#000000" bgcolor="#FFFFFF"> <tt>Hi
                                                  Michael,<br>
                                                  <br>
                                                  <tt>You can do
                                                    &quot;opensipsctl fifo
                                                    get<tt>_statistics
                                                      tm:&quot; and you have
                                                      the &quot;inuse<tt>_tra<tt>nsaction&quot;

                                                          statistic :<br>
                                                          <tt>    <a href="http://www.opensips.org/html/docs/modules/1.11.x/tm.html#id296307" target="_blank"></a><a href="http://www.opensips.org/html/docs/modules/1.11.x/tm.html#id296307" target="_blank">http://www.opensips.org/html/docs/modules/1.11.x/tm.html#id296307</a><br>
                                                          <br>
                                                          <tt>Regards,</tt><br>
                                                          <br>
                                                          </tt></tt></tt></tt></tt></tt><span>
                                                  <pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
                                                </span>
                                                <div>
                                                  <div>
                                                    <div>On 14.05.2015
                                                      09:45, Mickael
                                                      Marrache wrote:<br>
                                                    </div>
                                                    <blockquote type="cite">
                                                      <div>
                                                        <p class="MsoNormal"><span style="color:#1f497d" lang="EN-US">I&#39;m not sure there were no
                                                          transactions
                                                          in memory at
                                                          shutdown.</span></p>
                                                        <p class="MsoNormal"><span style="color:#1f497d" lang="EN-US"> </span></p>
                                                        <p class="MsoNormal"><span style="color:#1f497d" lang="EN-US">How can I check this? I didn&#39;t find
                                                          an MI command
                                                          for that.</span></p>
                                                        <p class="MsoNormal"><span style="color:#1f497d" lang="EN-US"> </span></p>
                                                        <div>
                                                          <div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
                                                          <p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;;color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;;color:windowtext">
                                                          Bogdan-Andrei
                                                          Iancu [<a href="mailto:bogdan@opensips.org" target="_blank"></a><a href="mailto:bogdan@opensips.org" target="_blank">mailto:bogdan@opensips.org</a>]
                                                          <br>
                                                          <b>Sent:</b>
                                                          Wednesday, May
                                                          13, 2015 6:44
                                                          PM<br>
                                                          <b>To:</b>
                                                          OpenSIPS users
                                                          mailling list;
                                                          <a href="mailto:mickaelmarrache@gmail.com" target="_blank"></a><a href="mailto:mickaelmarrache@gmail.com" target="_blank">mickaelmarrache@gmail.com</a><br>
                                                          <b>Subject:</b>
                                                          Re:
                                                          [OpenSIPS-Users]
                                                          Possible
                                                          memory leak
                                                          related to
                                                          AVPs</span></p>
                                                          </div>
                                                        </div>
                                                        <p class="MsoNormal"> </p>
                                                        <p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi Mickael,</span></tt><span><br>
                                                          <br>
                                                          <tt>What means
                                                          &quot;a lot&quot;  :) ?
                                                          I&#39;m asking as
                                                          if you have
                                                          100
                                                          transactions
                                                          in memory
                                                          (when doing
                                                          shutdown) and
                                                          you use ~10
                                                          AVPs per
                                                          INVITE in
                                                          script -&gt;
                                                          you have 1000
                                                          AVPs in mem.</tt><br>
                                                          <br>
                                                          <tt>When doing
                                                          the shutdown,
                                                          are any
                                                          transaction in
                                                          memory ? (with
                                                          no active
                                                          transactions
                                                          in mem, there
                                                          are no AVPs in
                                                          mem).</tt><br>
                                                          <br>
                                                          <tt>Regards,</tt><br>
                                                          <br>
                                                          </span></p>
                                                        <pre>Bogdan-Andrei Iancu</pre>
                                                        <pre>OpenSIPS Founder and Developer</pre>
                                                        <pre><a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
                                                        <div>
                                                          <p class="MsoNormal">On
                                                          13.05.2015
                                                          17:53, Mickael
                                                          Marrache
                                                          wrote:</p>
                                                        </div>
                                                        <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
                                                          <p class="MsoNormal">Hi,</p>
                                                          <p class="MsoNormal"> </p>
                                                          <p class="MsoNormal"><span lang="EN-US">We
                                                          have a memory
                                                          leak and are
                                                          trying to find
                                                          the cause.</span></p>
                                                          <p class="MsoNormal"><span lang="EN-US"> </span></p>
                                                          <p class="MsoNormal"><span lang="EN-US">We

                                                          compiled the
                                                          call with
                                                          memory
                                                          debugging and
                                                          at shutdown,
                                                          we see a lot
                                                          of allocated
                                                          AVPs that are
                                                          not released.</span></p>
                                                          <p class="MsoNormal"><span lang="EN-US"> </span></p>
                                                          <p class="MsoNormal"><span lang="EN-US">opensips[27560]:    


                                                          277067. N 
                                                          address=0x2aae957a0330
                                                          frag=0x2aae957a0300
                                                          size=24 used=1</span></p>
                                                          <p class="MsoNormal"><span lang="EN-US">opensips[27560]:            


                                                          alloc&#39;d from
                                                          usr_avp.c:
                                                          new_avp(119)</span></p>
                                                          <p class="MsoNormal"><span lang="EN-US">opensips[27560]:        


                                                          start
                                                          check=f0f0f0f0f0f0f0f0,
                                                          end check=
                                                          c0c0c0c0c0c0c0c0,
abcdefedabcdefed</span></p>
                                                          <p class="MsoNormal"><span lang="EN-US"> </span></p>
                                                          <p class="MsoNormal"><span lang="EN-US">Is
                                                          it possible to
                                                          have a memory
                                                          leak related
                                                          to AVPs? IIRC
                                                          OpenSIPS
                                                          handles it
                                                          internally.</span></p>
                                                          <p class="MsoNormal"><span lang="EN-US"> </span></p>
                                                          <p class="MsoNormal"><span lang="EN-US">Thanks,</span></p>
                                                          <p class="MsoNormal"><span lang="EN-US">Mickael</span></p>
                                                          <p class="MsoNormal"><span><br>
                                                          <br>
                                                          <br>
                                                          </span></p>
                                                          <pre>_______________________________________________</pre>
                                                          <pre>Users mailing list</pre>
                                                          <pre><a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a></pre>
                                                          <pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a></pre>
                                                        </blockquote>
                                                        <p class="MsoNormal"><span> </span></p>
                                                      </div>
                                                    </blockquote>
                                                    <br>
                                                  </div>
                                                </div>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </blockquote>
                                      <br>
                                    </div>
                                  </div>
                                </div>
                              </blockquote>
                            </div>
                            <br>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      <pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
    </blockquote>
    <br>
  </div></div></div>

<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>