[OpenSIPS-Users] B2BUA Segfault

Wed May 22 22:00:12 CEST 2013

Thank you -- I've recompiled and enabled the memory debug. I have the log
file from the whole experience available here:

http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log

(note - real phone numbers and domain names in the log have been replaced
with placeholders)

The key item seems to be:

CRITICAL:core:qm_free: freeing already freed pointer, first free: logic.c:
process_bridge_200OK(740) - aborting

Although this appears to be after we've already decided we're going to
crash, as I see "INFO:core:cleanup: cleanup" and "NFO:core:handle_sigs:
child process 24788 exited by a signal 6" above this part of the log.

Also worth noting is the existance of "CRITICAL:b2b_logic:b2bl_drop_entity:
we should never end up here" throughout the log.

Also, here's the stack trace at crash time. Note that there were two core
files generated for the same crash, so this is the backtrace from each:

#0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1  0x0000003564c34085 in abort () from /lib64/libc.so.6
#2  0x000000000049d370 in qm_free ()
#3  0x00007ffccb253804 in process_bridge_200OK () from
/usr/lib64/opensips/modules/b2b_logic.so
#4  0x00007ffccb254ba2 in b2b_logic_notify_reply () from
/usr/lib64/opensips/modules/b2b_logic.so
#5  0x00007ffccb2565e1 in b2b_logic_notify () from
/usr/lib64/opensips/modules/b2b_logic.so
#6  0x00007ffccb47a7a7 in b2b_tm_cback () from
/usr/lib64/opensips/modules/b2b_entities.so
#7  0x00007ffccc744b71 in run_trans_callbacks () from
/usr/lib64/opensips/modules/tm.so
#8  0x00007ffccc74fa12 in local_reply () from
/usr/lib64/opensips/modules/tm.so
#9  0x00007ffccc750cc5 in reply_received () from
/usr/lib64/opensips/modules/tm.so
#10 0x00000000004266da in forward_reply ()
#11 0x0000000000452ad8 in receive_msg ()
#12 0x0000000000496c95 in udp_rcv_loop ()
#13 0x000000000042d763 in main ()

#0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1  0x0000003564c34085 in abort () from /lib64/libc.so.6
#2  0x000000000049d370 in qm_free ()
#3  0x00007ffccb25a003 in b2bl_delete () from
/usr/lib64/opensips/modules/b2b_logic.so
#4  0x00007ffccb25a3d5 in destroy_b2bl_htable () from
/usr/lib64/opensips/modules/b2b_logic.so
#5  0x000000000046dbf2 in destroy_modules ()
#6  0x00000000004298a1 in cleanup ()
#7  0x000000000042a360 in handle_sigs ()
#8  0x000000000042db66 in main ()

I am unfamiliar with this codebase. Can anyone garner anything useful from
the logs?

Thanks,
Tolga

On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu <bogdan at opensips.org>wrote:

> **
> Hello Tolga,
>
> The crash seems to be in the memory manager, most probably because of
> memory corruption. To troubleshoot such issues you need to compile-in the
> memory debugger - see
> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem .
>
> Using memlog=1 + memdump=10 you should get a lot of logs related to mem
> ops, including a final report + abort() when the corruption is detected.
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>
>
> On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
>
> Hello,
>
>  While using the B2BUA module in OpenSIPS 1.9.0, we've encountered a
> consistent segfault. We are using a refer scenario just like the one in the
> B2BUA sample docs, and after several REFERs for the same call (to different
> destinations), OpenSIPS crashes with a segfault. The core file indicates
> the following backtrace:
>
>  #0  0x000000000049a334 in fm_malloc ()
> #1  0x00007fdaecd96230 in shm_malloc_unsafe (type=B2B_CLIENT,
> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320,
> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at
> ../../mem/shm_mem.h:248
> #2  shm_malloc (type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
> optimized out>, msg=0x0) at ../../mem/shm_mem.h:258
> #3  b2bl_create_new_entity (type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
> optimized out>, msg=0x0) at logic.c:293
> #4  0x00007fdaecd96882 in b2bl_new_client (to_uri=<value optimized out>,
> from_uri=<value optimized out>, tuple=<value optimized out>,
> ssid=0x7fdaeb026c00, msg=<value optimized out>) at logic.c:607
> #5  0x00007fdaecda3579 in process_bridge_200OK (msg=0x7fdaee8e8b30,
> extra_headers=0x7fdaeb03d578, body=<value optimized out>,
> tuple=0x7fdaeb01ada8, entity=<value optimized out>) at logic.c:816
> #6  0x00007fdaecda46c2 in b2b_logic_notify_reply (src=<value optimized
> out>, msg=0x7fdaee8e8b30, key=<value optimized out>, body=0x7fff2d3468b0,
> extra_headers=0x7fff2d3468a0, b2bl_key=0x7fff2d3476d0, hash_index=649,
> local_index=0)
>     at logic.c:1133
> #7  0x00007fdaecda6081 in b2b_logic_notify (src=1, msg=0x7fdaee8e8b30,
> key=0x7fdaeb03d500, type=1, param=0x7fff2d3476d0) at logic.c:2040
> #8  0x00007fdaecfca7ad in b2b_tm_cback (t=0x7fdaeb054118,
> htable=0x7fdaeb014630, ps=<value optimized out>) at dlg.c:2678
> #9  0x00007fdaee291441 in run_trans_callbacks (type=256,
> trans=0x7fdaeb054118, req=<value optimized out>, rpl=<value optimized out>,
> code=<value optimized out>) at t_hooks.c:212
> #10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118, p_msg=<value
> optimized out>, branch=<value optimized out>, msg_status=<value optimized
> out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391
> #11 0x00007fdaee29d31d in reply_received (p_msg=0x7fdaee8e8b30) at
> t_reply.c:1540
> #12 0x000000000042625a in forward_reply ()
> #13 0x0000000000451c28 in receive_msg ()
> #14 0x0000000000494e45 in udp_rcv_loop ()
> #15 0x000000000042d1a3 in main ()
>
>  I'm not really sure how to diagnose this one. Any
> hints/fixes/suggestions would be very appreciated.
>
>  Thanks,
> Tolga
>
>
> _______________________________________________
> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20130522/e2e4316c/attachment-0001.htm>