<div dir="ltr">Sorry for the self-reply -- here's the (same) stacktraces with line numbers and params:<div><br></div><div><div><font face="courier new, monospace">#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#1 0x0000003564c34085 in abort () from /lib64/libc.so.6</font></div><div><font face="courier new, monospace">#2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value optimized out>, file=0x7ffccb25df64 "logic.c", func=<value optimized out>, line=755) at mem/q_malloc.c:450</font></div>
<div><font face="courier new, monospace">#3 0x00007ffccb253804 in process_bridge_200OK (msg=0x7ffcccdd2600, extra_headers=0xd5, body=<value optimized out>, tuple=0x7ffcc9518de0, entity=<value optimized out>) at logic.c:755</font></div>
<div><font face="courier new, monospace">#4 0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value optimized out>, msg=0x7ffcccdd2600, key=<value optimized out>, body=0x7fffc2ce15d0, extra_headers=0x7fffc2ce15c0, b2bl_key=0x7fffc2ce23f0, hash_index=649, local_index=1)</font></div>
<div><font face="courier new, monospace"> at logic.c:1133</font></div><div><font face="courier new, monospace">#5 0x00007ffccb2565e1 in b2b_logic_notify (src=1, msg=0x7ffcccdd2600, key=0x7ffcc9525e40, type=1, param=0x7fffc2ce23f0) at logic.c:2040</font></div>
<div><font face="courier new, monospace">#6 0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110, htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678</font></div><div><font face="courier new, monospace">#7 0x00007ffccc744b71 in run_trans_callbacks (type=256, trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value optimized out>, code=<value optimized out>) at t_hooks.c:212</font></div>
<div><font face="courier new, monospace">#8 0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110, p_msg=<value optimized out>, branch=<value optimized out>, msg_status=<value optimized out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391</font></div>
<div><font face="courier new, monospace">#9 0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600) at t_reply.c:1540</font></div><div><font face="courier new, monospace">#10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at forward.c:574</font></div>
<div><font face="courier new, monospace">#11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>, len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207</font></div><div><font face="courier new, monospace">#12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424</font></div>
<div><font face="courier new, monospace">#13 0x000000000042d763 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:884</font></div><div><font face="courier new, monospace">#14 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1557</font></div>
</div><div><font face="courier new, monospace"><br></font></div><div><div><font face="courier new, monospace">#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6</font></div><div><font face="courier new, monospace">#1 0x0000003564c34085 in abort () from /lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value optimized out>, file=0x7ffccb262d75 "records.c", func=<value optimized out>, line=595) at mem/q_malloc.c:450</font></div>
<div><font face="courier new, monospace">#3 0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0, hash_index=<value optimized out>, not_del_b2be=1) at records.c:595</font></div><div><font face="courier new, monospace">#4 0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705</font></div>
<div><font face="courier new, monospace">#5 0x000000000046dbf2 in destroy_modules () at sr_module.c:371</font></div><div><font face="courier new, monospace">#6 0x00000000004298a1 in cleanup (show_status=1) at main.c:348</font></div>
<div><font face="courier new, monospace">#7 0x000000000042a360 in handle_sigs () at main.c:549</font></div><div><font face="courier new, monospace">#8 0x000000000042db66 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1024</font></div>
<div><font face="courier new, monospace">#9 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1557</font></div></div><div><br></div><div style>Thanks,</div><div style>Tolga</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan <span dir="ltr"><<a href="mailto:tolga@netbrains.com" target="_blank">tolga@netbrains.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thank you -- I've recompiled and enabled the memory debug. I have the log file from the whole experience available here:<div><br></div><div><a href="http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log" target="_blank">http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log</a><br>
</div><div><br></div><div>(note - real phone numbers and domain names in the log have been replaced with placeholders)</div><div><br></div><div>The key item seems to be:</div><div><br></div><div>CRITICAL:core:qm_free: freeing already freed pointer, first free: logic.c: process_bridge_200OK(740) - aborting<br>
</div><div><br></div><div>Although this appears to be after we've already decided we're going to crash, as I see "INFO:core:cleanup: cleanup" and "NFO:core:handle_sigs: child process 24788 exited by a signal 6" above this part of the log.</div>
<div><br></div><div>Also worth noting is the existance of "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here" throughout the log.</div><div><div><br></div><div>Also, here's the stack trace at crash time. Note that there were two core files generated for the same crash, so this is the backtrace from each:</div>
<div><br></div><div><div><font face="courier new, monospace">#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6</font></div><div><font face="courier new, monospace">#1 0x0000003564c34085 in abort () from /lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#2 0x000000000049d370 in qm_free ()</font></div><div><font face="courier new, monospace">#3 0x00007ffccb253804 in process_bridge_200OK () from /usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new, monospace">#4 0x00007ffccb254ba2 in b2b_logic_notify_reply () from /usr/lib64/opensips/modules/b2b_logic.so</font></div><div><font face="courier new, monospace">#5 0x00007ffccb2565e1 in b2b_logic_notify () from /usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new, monospace">#6 0x00007ffccb47a7a7 in b2b_tm_cback () from /usr/lib64/opensips/modules/b2b_entities.so</font></div><div><font face="courier new, monospace">#7 0x00007ffccc744b71 in run_trans_callbacks () from /usr/lib64/opensips/modules/tm.so</font></div>
<div><font face="courier new, monospace">#8 0x00007ffccc74fa12 in local_reply () from /usr/lib64/opensips/modules/tm.so</font></div><div><font face="courier new, monospace">#9 0x00007ffccc750cc5 in reply_received () from /usr/lib64/opensips/modules/tm.so</font></div>
<div><font face="courier new, monospace">#10 0x00000000004266da in forward_reply ()</font></div><div><font face="courier new, monospace">#11 0x0000000000452ad8 in receive_msg ()</font></div><div><font face="courier new, monospace">#12 0x0000000000496c95 in udp_rcv_loop ()</font></div>
<div><font face="courier new, monospace">#13 0x000000000042d763 in main ()</font></div><div><font face="courier new, monospace"><br></font></div><div><div><font face="courier new, monospace">#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#1 0x0000003564c34085 in abort () from /lib64/libc.so.6</font></div><div><font face="courier new, monospace">#2 0x000000000049d370 in qm_free ()</font></div><div><font face="courier new, monospace">#3 0x00007ffccb25a003 in b2bl_delete () from /usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new, monospace">#4 0x00007ffccb25a3d5 in destroy_b2bl_htable () from /usr/lib64/opensips/modules/b2b_logic.so</font></div><div><font face="courier new, monospace">#5 0x000000000046dbf2 in destroy_modules ()</font></div>
<div><font face="courier new, monospace">#6 0x00000000004298a1 in cleanup ()</font></div><div><font face="courier new, monospace">#7 0x000000000042a360 in handle_sigs ()</font></div><div><font face="courier new, monospace">#8 0x000000000042db66 in main ()</font></div>
</div><div><br></div><div>I am unfamiliar with this codebase. Can anyone garner anything useful from the logs?</div><div><br></div><div>Thanks,</div><div>Tolga</div><div><br></div></div></div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div text="#000000" bgcolor="#ffffff">
<tt>Hello Tolga,<br>
<br>
The crash seems to be in the memory manager, most probably because
of memory corruption. To troubleshoot such issues you need to
compile-in the memory debugger - see
<a href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem" target="_blank">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a> .<br>
<br>
Using memlog=1 + memdump=10 you should get a lot of logs related
to mem ops, including a final report + abort() when the corruption
is detected.<br>
<br>
Regards,<br>
</tt>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre><div><div>
<br>
On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
</div></div><blockquote type="cite"><div><div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>While using the B2BUA module in OpenSIPS 1.9.0,
we've encountered a consistent segfault. We are using a refer
scenario just like the one in the B2BUA sample docs, and after
several REFERs for the same call (to different destinations),
OpenSIPS crashes with a segfault. The core file indicates the
following backtrace:</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">#0
0x000000000049a334 in fm_malloc ()</font></div>
<div><font face="courier new, monospace">#1
0x00007fdaecd96230 in shm_malloc_unsafe (type=B2B_CLIENT,
entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360,
from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
optimized out>, msg=0x0) at ../../mem/shm_mem.h:248</font></div>
<div><font face="courier new, monospace">#2 shm_malloc
(type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
to_uri=0x7fff2d346360, from_uri=0x7fff2d346320,
from_dname=0x0, ssid=<value optimized out>, msg=0x0)
at ../../mem/shm_mem.h:258</font></div>
<div><font face="courier new, monospace">#3
b2bl_create_new_entity (type=B2B_CLIENT,
entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360,
from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
optimized out>, msg=0x0) at logic.c:293</font></div>
<div><font face="courier new, monospace">#4
0x00007fdaecd96882 in b2bl_new_client (to_uri=<value
optimized out>, from_uri=<value optimized out>,
tuple=<value optimized out>, ssid=0x7fdaeb026c00,
msg=<value optimized out>) at logic.c:607</font></div>
<div><font face="courier new, monospace">#5
0x00007fdaecda3579 in process_bridge_200OK
(msg=0x7fdaee8e8b30, extra_headers=0x7fdaeb03d578,
body=<value optimized out>, tuple=0x7fdaeb01ada8,
entity=<value optimized out>) at logic.c:816</font></div>
<div><font face="courier new, monospace">#6
0x00007fdaecda46c2 in b2b_logic_notify_reply
(src=<value optimized out>, msg=0x7fdaee8e8b30,
key=<value optimized out>, body=0x7fff2d3468b0,
extra_headers=0x7fff2d3468a0, b2bl_key=0x7fff2d3476d0,
hash_index=649, local_index=0)</font></div>
<div><font face="courier new, monospace"> at logic.c:1133</font></div>
<div><font face="courier new, monospace">#7
0x00007fdaecda6081 in b2b_logic_notify (src=1,
msg=0x7fdaee8e8b30, key=0x7fdaeb03d500, type=1,
param=0x7fff2d3476d0) at logic.c:2040</font></div>
<div><font face="courier new, monospace">#8
0x00007fdaecfca7ad in b2b_tm_cback (t=0x7fdaeb054118,
htable=0x7fdaeb014630, ps=<value optimized out>) at
dlg.c:2678</font></div>
<div><font face="courier new, monospace">#9
0x00007fdaee291441 in run_trans_callbacks (type=256,
trans=0x7fdaeb054118, req=<value optimized out>,
rpl=<value optimized out>, code=<value optimized
out>) at t_hooks.c:212</font></div>
<div><font face="courier new, monospace">#10
0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118,
p_msg=<value optimized out>, branch=<value
optimized out>, msg_status=<value optimized out>,
cancel_bitmap=<value optimized out>) at
t_reply.c:1391</font></div>
<div><font face="courier new, monospace">#11
0x00007fdaee29d31d in reply_received
(p_msg=0x7fdaee8e8b30) at t_reply.c:1540</font></div>
<div><font face="courier new, monospace">#12
0x000000000042625a in forward_reply ()</font></div>
<div><font face="courier new, monospace">#13
0x0000000000451c28 in receive_msg ()</font></div>
<div><font face="courier new, monospace">#14
0x0000000000494e45 in udp_rcv_loop ()</font></div>
<div><font face="courier new, monospace">#15
0x000000000042d1a3 in main ()</font></div>
<div><br>
</div>
<div>I'm not really sure how to diagnose this one.
Any hints/fixes/suggestions would be very appreciated.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tolga</div>
</div>
</div>
</div></div><pre><fieldset></fieldset>
_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>