[OpenSIPS-Users] B2BUA Segfault

Wed May 22 22:02:12 CEST 2013

Sorry for the self-reply -- here's the (same) stacktraces with line numbers
and params:

#0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1  0x0000003564c34085 in abort () from /lib64/libc.so.6
#2  0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value
optimized out>, file=0x7ffccb25df64 "logic.c", func=<value optimized out>,
line=755) at mem/q_malloc.c:450
#3  0x00007ffccb253804 in process_bridge_200OK (msg=0x7ffcccdd2600,
extra_headers=0xd5, body=<value optimized out>, tuple=0x7ffcc9518de0,
entity=<value optimized out>) at logic.c:755
#4  0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value optimized
out>, msg=0x7ffcccdd2600, key=<value optimized out>, body=0x7fffc2ce15d0,
extra_headers=0x7fffc2ce15c0, b2bl_key=0x7fffc2ce23f0, hash_index=649,
local_index=1)
    at logic.c:1133
#5  0x00007ffccb2565e1 in b2b_logic_notify (src=1, msg=0x7ffcccdd2600,
key=0x7ffcc9525e40, type=1, param=0x7fffc2ce23f0) at logic.c:2040
#6  0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110,
htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678
#7  0x00007ffccc744b71 in run_trans_callbacks (type=256,
trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value optimized out>,
code=<value optimized out>) at t_hooks.c:212
#8  0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110, p_msg=<value
optimized out>, branch=<value optimized out>, msg_status=<value optimized
out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391
#9  0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600) at
t_reply.c:1540
#10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at
forward.c:574
#11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>,
len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207
#12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424
#13 0x000000000042d763 in main_loop (argc=<value optimized out>,
argv=<value optimized out>) at main.c:884
#14 main (argc=<value optimized out>, argv=<value optimized out>) at
main.c:1557

#0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1  0x0000003564c34085 in abort () from /lib64/libc.so.6
#2  0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value
optimized out>, file=0x7ffccb262d75 "records.c", func=<value optimized
out>, line=595) at mem/q_malloc.c:450
#3  0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0,
hash_index=<value optimized out>, not_del_b2be=1) at records.c:595
#4  0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705
#5  0x000000000046dbf2 in destroy_modules () at sr_module.c:371
#6  0x00000000004298a1 in cleanup (show_status=1) at main.c:348
#7  0x000000000042a360 in handle_sigs () at main.c:549
#8  0x000000000042db66 in main_loop (argc=<value optimized out>,
argv=<value optimized out>) at main.c:1024
#9  main (argc=<value optimized out>, argv=<value optimized out>) at
main.c:1557

Thanks,
Tolga

On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan <tolga at netbrains.com> wrote:

> Thank you -- I've recompiled and enabled the memory debug. I have the log
> file from the whole experience available here:
>
> http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log
>
> (note - real phone numbers and domain names in the log have been replaced
> with placeholders)
>
> The key item seems to be:
>
> CRITICAL:core:qm_free: freeing already freed pointer, first free: logic.c:
> process_bridge_200OK(740) - aborting
>
> Although this appears to be after we've already decided we're going to
> crash, as I see "INFO:core:cleanup: cleanup" and "NFO:core:handle_sigs:
> child process 24788 exited by a signal 6" above this part of the log.
>
> Also worth noting is the existance of
> "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here"
> throughout the log.
>
> Also, here's the stack trace at crash time. Note that there were two core
> files generated for the same crash, so this is the backtrace from each:
>
> #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
> #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
> #2  0x000000000049d370 in qm_free ()
> #3  0x00007ffccb253804 in process_bridge_200OK () from
> /usr/lib64/opensips/modules/b2b_logic.so
> #4  0x00007ffccb254ba2 in b2b_logic_notify_reply () from
> /usr/lib64/opensips/modules/b2b_logic.so
> #5  0x00007ffccb2565e1 in b2b_logic_notify () from
> /usr/lib64/opensips/modules/b2b_logic.so
> #6  0x00007ffccb47a7a7 in b2b_tm_cback () from
> /usr/lib64/opensips/modules/b2b_entities.so
> #7  0x00007ffccc744b71 in run_trans_callbacks () from
> /usr/lib64/opensips/modules/tm.so
> #8  0x00007ffccc74fa12 in local_reply () from
> /usr/lib64/opensips/modules/tm.so
> #9  0x00007ffccc750cc5 in reply_received () from
> /usr/lib64/opensips/modules/tm.so
> #10 0x00000000004266da in forward_reply ()
> #11 0x0000000000452ad8 in receive_msg ()
> #12 0x0000000000496c95 in udp_rcv_loop ()
> #13 0x000000000042d763 in main ()
>
> #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
> #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
> #2  0x000000000049d370 in qm_free ()
> #3  0x00007ffccb25a003 in b2bl_delete () from
> /usr/lib64/opensips/modules/b2b_logic.so
> #4  0x00007ffccb25a3d5 in destroy_b2bl_htable () from
> /usr/lib64/opensips/modules/b2b_logic.so
> #5  0x000000000046dbf2 in destroy_modules ()
> #6  0x00000000004298a1 in cleanup ()
> #7  0x000000000042a360 in handle_sigs ()
> #8  0x000000000042db66 in main ()
>
> I am unfamiliar with this codebase. Can anyone garner anything useful from
> the logs?
>
> Thanks,
> Tolga
>
>
>
> On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu <bogdan at opensips.org>wrote:
>
>> **
>> Hello Tolga,
>>
>> The crash seems to be in the memory manager, most probably because of
>> memory corruption. To troubleshoot such issues you need to compile-in the
>> memory debugger - see
>> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem .
>>
>> Using memlog=1 + memdump=10 you should get a lot of logs related to mem
>> ops, including a final report + abort() when the corruption is detected.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>
>>
>> On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
>>
>> Hello,
>>
>>  While using the B2BUA module in OpenSIPS 1.9.0, we've encountered a
>> consistent segfault. We are using a refer scenario just like the one in the
>> B2BUA sample docs, and after several REFERs for the same call (to different
>> destinations), OpenSIPS crashes with a segfault. The core file indicates
>> the following backtrace:
>>
>>  #0  0x000000000049a334 in fm_malloc ()
>> #1  0x00007fdaecd96230 in shm_malloc_unsafe (type=B2B_CLIENT,
>> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320,
>> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at
>> ../../mem/shm_mem.h:248
>> #2  shm_malloc (type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
>> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
>> optimized out>, msg=0x0) at ../../mem/shm_mem.h:258
>> #3  b2bl_create_new_entity (type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
>> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
>> optimized out>, msg=0x0) at logic.c:293
>> #4  0x00007fdaecd96882 in b2bl_new_client (to_uri=<value optimized out>,
>> from_uri=<value optimized out>, tuple=<value optimized out>,
>> ssid=0x7fdaeb026c00, msg=<value optimized out>) at logic.c:607
>> #5  0x00007fdaecda3579 in process_bridge_200OK (msg=0x7fdaee8e8b30,
>> extra_headers=0x7fdaeb03d578, body=<value optimized out>,
>> tuple=0x7fdaeb01ada8, entity=<value optimized out>) at logic.c:816
>> #6  0x00007fdaecda46c2 in b2b_logic_notify_reply (src=<value optimized
>> out>, msg=0x7fdaee8e8b30, key=<value optimized out>, body=0x7fff2d3468b0,
>> extra_headers=0x7fff2d3468a0, b2bl_key=0x7fff2d3476d0, hash_index=649,
>> local_index=0)
>>     at logic.c:1133
>> #7  0x00007fdaecda6081 in b2b_logic_notify (src=1, msg=0x7fdaee8e8b30,
>> key=0x7fdaeb03d500, type=1, param=0x7fff2d3476d0) at logic.c:2040
>> #8  0x00007fdaecfca7ad in b2b_tm_cback (t=0x7fdaeb054118,
>> htable=0x7fdaeb014630, ps=<value optimized out>) at dlg.c:2678
>> #9  0x00007fdaee291441 in run_trans_callbacks (type=256,
>> trans=0x7fdaeb054118, req=<value optimized out>, rpl=<value optimized out>,
>> code=<value optimized out>) at t_hooks.c:212
>> #10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118, p_msg=<value
>> optimized out>, branch=<value optimized out>, msg_status=<value optimized
>> out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391
>> #11 0x00007fdaee29d31d in reply_received (p_msg=0x7fdaee8e8b30) at
>> t_reply.c:1540
>> #12 0x000000000042625a in forward_reply ()
>> #13 0x0000000000451c28 in receive_msg ()
>> #14 0x0000000000494e45 in udp_rcv_loop ()
>> #15 0x000000000042d1a3 in main ()
>>
>>  I'm not really sure how to diagnose this one. Any
>> hints/fixes/suggestions would be very appreciated.
>>
>>  Thanks,
>> Tolga
>>
>>
>> _______________________________________________
>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20130522/b23bd624/attachment.htm>