<div dir="ltr"><div>Sure, here you go:</div><div><br></div><div>version: opensips 1.9.0-tls (x86_64/linux)</div><div>flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</div>
<div>ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535</div><div>poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.</div><div>svnrevision: unknown</div><div>
@(#) $Id: main.c 9790 2013-02-15 10:14:34Z bogdan_iancu $</div><div>main.c compiled on 12:39:26 May 22 2013 with gcc 4.4.7</div><div><br></div><div style>It also has the couple patches that I&#39;ve previously submitted (and you&#39;ve since merged) in the build (the three related to GRUU bugs). They aren&#39;t a factor, as this particular instance is not acting as a registrar at all.</div>
<div><br></div><div style>Thanks,</div><div style>Tolga</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, May 24, 2013 at 9:33 AM, Bogdan-Andrei Iancu <span dir="ltr">&lt;<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>

  
    
  
  <div text="#000000" bgcolor="#ffffff">
    <tt>Hi Tolga,<br>
      <br>
      Thanks for the info.<br>
      <br>
      What exact OpenSIPs version/revision are you using ? I need to
      correlate logs with sources.<br>
      <br>
      Regards,<br>
    </tt><div class="im">
    <pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
    <br></div><div><div class="h5">
    On 05/22/2013 11:02 PM, Tolga Tarhan wrote:
    <blockquote type="cite">
      <div dir="ltr">Sorry for the self-reply -- here&#39;s the (same)
        stacktraces with line numbers and params:
        <div><br>
        </div>
        <div>
          <div><font face="courier new, monospace">#0
               0x0000003564c328a5 in raise () from /lib64/libc.so.6</font></div>
          <div><font face="courier new, monospace">#1
               0x0000003564c34085 in abort () from /lib64/libc.so.6</font></div>
          <div><font face="courier new, monospace">#2
               0x000000000049d370 in qm_free (qm=&lt;value optimized
              out&gt;, p=&lt;value optimized out&gt;,
              file=0x7ffccb25df64 &quot;logic.c&quot;, func=&lt;value optimized
              out&gt;, line=755) at mem/q_malloc.c:450</font></div>
          <div><font face="courier new, monospace">#3
               0x00007ffccb253804 in process_bridge_200OK
              (msg=0x7ffcccdd2600, extra_headers=0xd5, body=&lt;value
              optimized out&gt;, tuple=0x7ffcc9518de0, entity=&lt;value
              optimized out&gt;) at logic.c:755</font></div>
          <div><font face="courier new, monospace">#4
               0x00007ffccb254ba2 in b2b_logic_notify_reply
              (src=&lt;value optimized out&gt;, msg=0x7ffcccdd2600,
              key=&lt;value optimized out&gt;, body=0x7fffc2ce15d0,
              extra_headers=0x7fffc2ce15c0, b2bl_key=0x7fffc2ce23f0,
              hash_index=649, local_index=1)</font></div>
          <div><font face="courier new, monospace">    at logic.c:1133</font></div>
          <div><font face="courier new, monospace">#5
               0x00007ffccb2565e1 in b2b_logic_notify (src=1,
              msg=0x7ffcccdd2600, key=0x7ffcc9525e40, type=1,
              param=0x7fffc2ce23f0) at logic.c:2040</font></div>
          <div><font face="courier new, monospace">#6
               0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110,
              htable=0x7ffcc94f38d0, ps=&lt;value optimized out&gt;) at
              dlg.c:2678</font></div>
          <div><font face="courier new, monospace">#7
               0x00007ffccc744b71 in run_trans_callbacks (type=256,
              trans=0x7ffcc951c110, req=&lt;value optimized out&gt;,
              rpl=&lt;value optimized out&gt;, code=&lt;value optimized
              out&gt;) at t_hooks.c:212</font></div>
          <div><font face="courier new, monospace">#8
               0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110,
              p_msg=&lt;value optimized out&gt;, branch=&lt;value
              optimized out&gt;, msg_status=&lt;value optimized out&gt;,
              cancel_bitmap=&lt;value optimized out&gt;) at
              t_reply.c:1391</font></div>
          <div><font face="courier new, monospace">#9
               0x00007ffccc750cc5 in reply_received
              (p_msg=0x7ffcccdd2600) at t_reply.c:1540</font></div>
          <div><font face="courier new, monospace">#10
              0x00000000004266da in forward_reply (msg=0x7ffcccdd2600)
              at forward.c:574</font></div>
          <div><font face="courier new, monospace">#11
              0x0000000000452ad8 in receive_msg (buf=&lt;value optimized
              out&gt;, len=&lt;value optimized out&gt;,
              rcv_info=0x7fffc2ce28c0) at receive.c:207</font></div>
          <div><font face="courier new, monospace">#12
              0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424</font></div>
          <div><font face="courier new, monospace">#13
              0x000000000042d763 in main_loop (argc=&lt;value optimized
              out&gt;, argv=&lt;value optimized out&gt;) at main.c:884</font></div>
          <div><font face="courier new, monospace">#14 main
              (argc=&lt;value optimized out&gt;, argv=&lt;value
              optimized out&gt;) at main.c:1557</font></div>
        </div>
        <div><font face="courier new, monospace"><br>
          </font></div>
        <div>
          <div><font face="courier new, monospace">#0
               0x0000003564c328a5 in raise () from /lib64/libc.so.6</font></div>
          <div><font face="courier new, monospace">#1
               0x0000003564c34085 in abort () from /lib64/libc.so.6</font></div>
          <div><font face="courier new, monospace">#2
               0x000000000049d370 in qm_free (qm=&lt;value optimized
              out&gt;, p=&lt;value optimized out&gt;,
              file=0x7ffccb262d75 &quot;records.c&quot;, func=&lt;value optimized
              out&gt;, line=595) at mem/q_malloc.c:450</font></div>
          <div><font face="courier new, monospace">#3
               0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0,
              hash_index=&lt;value optimized out&gt;, not_del_b2be=1) at
              records.c:595</font></div>
          <div><font face="courier new, monospace">#4
               0x00007ffccb25a3d5 in destroy_b2bl_htable () at
              records.c:705</font></div>
          <div><font face="courier new, monospace">#5
               0x000000000046dbf2 in destroy_modules () at
              sr_module.c:371</font></div>
          <div><font face="courier new, monospace">#6
               0x00000000004298a1 in cleanup (show_status=1) at
              main.c:348</font></div>
          <div><font face="courier new, monospace">#7
               0x000000000042a360 in handle_sigs () at main.c:549</font></div>
          <div><font face="courier new, monospace">#8
               0x000000000042db66 in main_loop (argc=&lt;value optimized
              out&gt;, argv=&lt;value optimized out&gt;) at main.c:1024</font></div>
          <div><font face="courier new, monospace">#9  main
              (argc=&lt;value optimized out&gt;, argv=&lt;value
              optimized out&gt;) at main.c:1557</font></div>
        </div>
        <div><br>
        </div>
        <div>Thanks,</div>
        <div>Tolga</div>
      </div>
      <div class="gmail_extra">
        <br>
        <br>
        <div class="gmail_quote">On Wed, May 22, 2013 at 1:00 PM, Tolga
          Tarhan <span dir="ltr">&lt;<a href="mailto:tolga@netbrains.com" target="_blank">tolga@netbrains.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            <div dir="ltr">Thank you -- I&#39;ve recompiled and enabled the
              memory debug. I have the log file from the whole
              experience available here:
              <div><br>
              </div>
              <div><a href="http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log" target="_blank">http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log</a><br>
              </div>
              <div><br>
              </div>
              <div>(note - real phone numbers and domain names in the
                log have been replaced with placeholders)</div>
              <div><br>
              </div>
              <div>The key item seems to be:</div>
              <div><br>
              </div>
              <div>CRITICAL:core:qm_free: freeing already freed pointer,
                first free: logic.c: process_bridge_200OK(740) -
                aborting<br>
              </div>
              <div><br>
              </div>
              <div>Although this appears to be after we&#39;ve already
                decided we&#39;re going to crash, as I see
                &quot;INFO:core:cleanup: cleanup&quot; and &quot;NFO:core:handle_sigs:
                child process 24788 exited by a signal 6&quot; above this
                part of the log.</div>
              <div><br>
              </div>
              <div>Also worth noting is the existance of
                &quot;CRITICAL:b2b_logic:b2bl_drop_entity: we should never
                end up here&quot; throughout the log.</div>
              <div>
                <div><br>
                </div>
                <div>Also, here&#39;s the stack trace at crash time. Note
                  that there were two core files generated for the same
                  crash, so this is the backtrace from each:</div>
                <div><br>
                </div>
                <div>
                  <div><font face="courier new, monospace">#0
                       0x0000003564c328a5 in raise () from
                      /lib64/libc.so.6</font></div>
                  <div><font face="courier new, monospace">#1
                       0x0000003564c34085 in abort () from
                      /lib64/libc.so.6</font></div>
                  <div><font face="courier new, monospace">#2
                       0x000000000049d370 in qm_free ()</font></div>
                  <div><font face="courier new, monospace">#3
                       0x00007ffccb253804 in process_bridge_200OK ()
                      from /usr/lib64/opensips/modules/b2b_logic.so</font></div>
                  <div><font face="courier new, monospace">#4
                       0x00007ffccb254ba2 in b2b_logic_notify_reply ()
                      from /usr/lib64/opensips/modules/b2b_logic.so</font></div>
                  <div><font face="courier new, monospace">#5
                       0x00007ffccb2565e1 in b2b_logic_notify () from
                      /usr/lib64/opensips/modules/b2b_logic.so</font></div>
                  <div><font face="courier new, monospace">#6
                       0x00007ffccb47a7a7 in b2b_tm_cback () from
                      /usr/lib64/opensips/modules/b2b_entities.so</font></div>
                  <div><font face="courier new, monospace">#7
                       0x00007ffccc744b71 in run_trans_callbacks () from
                      /usr/lib64/opensips/modules/tm.so</font></div>
                  <div><font face="courier new, monospace">#8
                       0x00007ffccc74fa12 in local_reply () from
                      /usr/lib64/opensips/modules/tm.so</font></div>
                  <div><font face="courier new, monospace">#9
                       0x00007ffccc750cc5 in reply_received () from
                      /usr/lib64/opensips/modules/tm.so</font></div>
                  <div><font face="courier new, monospace">#10
                      0x00000000004266da in forward_reply ()</font></div>
                  <div><font face="courier new, monospace">#11
                      0x0000000000452ad8 in receive_msg ()</font></div>
                  <div><font face="courier new, monospace">#12
                      0x0000000000496c95 in udp_rcv_loop ()</font></div>
                  <div><font face="courier new, monospace">#13
                      0x000000000042d763 in main ()</font></div>
                  <div><font face="courier new, monospace"><br>
                    </font></div>
                  <div>
                    <div><font face="courier new, monospace">#0
                         0x0000003564c328a5 in raise () from
                        /lib64/libc.so.6</font></div>
                    <div><font face="courier new, monospace">#1
                         0x0000003564c34085 in abort () from
                        /lib64/libc.so.6</font></div>
                    <div><font face="courier new, monospace">#2
                         0x000000000049d370 in qm_free ()</font></div>
                    <div><font face="courier new, monospace">#3
                         0x00007ffccb25a003 in b2bl_delete () from
                        /usr/lib64/opensips/modules/b2b_logic.so</font></div>
                    <div><font face="courier new, monospace">#4
                         0x00007ffccb25a3d5 in destroy_b2bl_htable ()
                        from /usr/lib64/opensips/modules/b2b_logic.so</font></div>
                    <div><font face="courier new, monospace">#5
                         0x000000000046dbf2 in destroy_modules ()</font></div>
                    <div><font face="courier new, monospace">#6
                         0x00000000004298a1 in cleanup ()</font></div>
                    <div><font face="courier new, monospace">#7
                         0x000000000042a360 in handle_sigs ()</font></div>
                    <div><font face="courier new, monospace">#8
                         0x000000000042db66 in main ()</font></div>
                  </div>
                  <div><br>
                  </div>
                  <div>I am unfamiliar with this codebase. Can anyone
                    garner anything useful from the logs?</div>
                  <div><br>
                  </div>
                  <div>Thanks,</div>
                  <div>Tolga</div>
                  <div><br>
                  </div>
                </div>
              </div>
            </div>
            <div>
              <div>
                <div class="gmail_extra"><br>
                  <br>
                  <div class="gmail_quote">On Wed, May 22, 2013 at 9:02
                    AM, Bogdan-Andrei Iancu <span dir="ltr">&lt;<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                      <div text="#000000" bgcolor="#ffffff"> <tt>Hello
                          Tolga,<br>
                          <br>
                          The crash seems to be in the memory manager,
                          most probably because of memory corruption. To
                          troubleshoot such issues you need to
                          compile-in the memory debugger - see <a href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem" target="_blank">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a>
                          .<br>
                          <br>
                          Using memlog=1 + memdump=10 you should get a
                          lot of logs related to mem ops, including a
                          final report + abort() when the corruption is
                          detected.<br>
                          <br>
                          Regards,<br>
                        </tt>
                        <pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
                        <div>
                          <div> <br>
                            On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
                          </div>
                        </div>
                        <blockquote type="cite">
                          <div>
                            <div>
                              <div dir="ltr">Hello,
                                <div><br>
                                </div>
                                <div>While using the B2BUA module in
                                  OpenSIPS 1.9.0, we&#39;ve encountered a
                                  consistent segfault. We are using a
                                  refer scenario just like the one in
                                  the B2BUA sample docs, and after
                                  several REFERs for the same call (to
                                  different destinations), OpenSIPS
                                  crashes with a segfault. The core file
                                  indicates the following backtrace:</div>
                                <div><br>
                                </div>
                                <div>
                                  <div><font face="courier new,
                                      monospace">#0  0x000000000049a334
                                      in fm_malloc ()</font></div>
                                  <div><font face="courier new,
                                      monospace">#1  0x00007fdaecd96230
                                      in shm_malloc_unsafe
                                      (type=B2B_CLIENT,
                                      entity_id=0x7fdaee8ec750,
                                      to_uri=0x7fff2d346360,
                                      from_uri=0x7fff2d346320,
                                      from_dname=0x0, ssid=&lt;value
                                      optimized out&gt;, msg=0x0) at
                                      ../../mem/shm_mem.h:248</font></div>
                                  <div><font face="courier new,
                                      monospace">#2  shm_malloc
                                      (type=B2B_CLIENT,
                                      entity_id=0x7fdaee8ec750,
                                      to_uri=0x7fff2d346360,
                                      from_uri=0x7fff2d346320,
                                      from_dname=0x0, ssid=&lt;value
                                      optimized out&gt;, msg=0x0) at
                                      ../../mem/shm_mem.h:258</font></div>
                                  <div><font face="courier new,
                                      monospace">#3
                                       b2bl_create_new_entity
                                      (type=B2B_CLIENT,
                                      entity_id=0x7fdaee8ec750,
                                      to_uri=0x7fff2d346360,
                                      from_uri=0x7fff2d346320,
                                      from_dname=0x0, ssid=&lt;value
                                      optimized out&gt;, msg=0x0) at
                                      logic.c:293</font></div>
                                  <div><font face="courier new,
                                      monospace">#4  0x00007fdaecd96882
                                      in b2bl_new_client
                                      (to_uri=&lt;value optimized
                                      out&gt;, from_uri=&lt;value
                                      optimized out&gt;, tuple=&lt;value
                                      optimized out&gt;,
                                      ssid=0x7fdaeb026c00, msg=&lt;value
                                      optimized out&gt;) at logic.c:607</font></div>
                                  <div><font face="courier new,
                                      monospace">#5  0x00007fdaecda3579
                                      in process_bridge_200OK
                                      (msg=0x7fdaee8e8b30,
                                      extra_headers=0x7fdaeb03d578,
                                      body=&lt;value optimized out&gt;,
                                      tuple=0x7fdaeb01ada8,
                                      entity=&lt;value optimized
                                      out&gt;) at logic.c:816</font></div>
                                  <div><font face="courier new,
                                      monospace">#6  0x00007fdaecda46c2
                                      in b2b_logic_notify_reply
                                      (src=&lt;value optimized out&gt;,
                                      msg=0x7fdaee8e8b30, key=&lt;value
                                      optimized out&gt;,
                                      body=0x7fff2d3468b0,
                                      extra_headers=0x7fff2d3468a0,
                                      b2bl_key=0x7fff2d3476d0,
                                      hash_index=649, local_index=0)</font></div>
                                  <div><font face="courier new,
                                      monospace">    at logic.c:1133</font></div>
                                  <div><font face="courier new,
                                      monospace">#7  0x00007fdaecda6081
                                      in b2b_logic_notify (src=1,
                                      msg=0x7fdaee8e8b30,
                                      key=0x7fdaeb03d500, type=1,
                                      param=0x7fff2d3476d0) at
                                      logic.c:2040</font></div>
                                  <div><font face="courier new,
                                      monospace">#8  0x00007fdaecfca7ad
                                      in b2b_tm_cback (t=0x7fdaeb054118,
                                      htable=0x7fdaeb014630,
                                      ps=&lt;value optimized out&gt;) at
                                      dlg.c:2678</font></div>
                                  <div><font face="courier new,
                                      monospace">#9  0x00007fdaee291441
                                      in run_trans_callbacks (type=256,
                                      trans=0x7fdaeb054118,
                                      req=&lt;value optimized out&gt;,
                                      rpl=&lt;value optimized out&gt;,
                                      code=&lt;value optimized out&gt;)
                                      at t_hooks.c:212</font></div>
                                  <div><font face="courier new,
                                      monospace">#10 0x00007fdaee29c0e2
                                      in local_reply (t=0x7fdaeb054118,
                                      p_msg=&lt;value optimized out&gt;,
                                      branch=&lt;value optimized
                                      out&gt;, msg_status=&lt;value
                                      optimized out&gt;,
                                      cancel_bitmap=&lt;value optimized
                                      out&gt;) at t_reply.c:1391</font></div>
                                  <div><font face="courier new,
                                      monospace">#11 0x00007fdaee29d31d
                                      in reply_received
                                      (p_msg=0x7fdaee8e8b30) at
                                      t_reply.c:1540</font></div>
                                  <div><font face="courier new,
                                      monospace">#12 0x000000000042625a
                                      in forward_reply ()</font></div>
                                  <div><font face="courier new,
                                      monospace">#13 0x0000000000451c28
                                      in receive_msg ()</font></div>
                                  <div><font face="courier new,
                                      monospace">#14 0x0000000000494e45
                                      in udp_rcv_loop ()</font></div>
                                  <div><font face="courier new,
                                      monospace">#15 0x000000000042d1a3
                                      in main ()</font></div>
                                  <div><br>
                                  </div>
                                  <div>I&#39;m not really sure how to
                                    diagnose this one. Any
                                    hints/fixes/suggestions would be
                                    very appreciated.</div>
                                  <div><br>
                                  </div>
                                  <div>Thanks,</div>
                                  <div>Tolga</div>
                                </div>
                              </div>
                            </div>
                          </div>
                          <pre><fieldset></fieldset>
_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
                        </blockquote>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
  </div></div></div>

</blockquote></div><br></div>