[OpenSIPS-Users] B2BUA Segfault
Bogdan-Andrei Iancu
bogdan at opensips.org
Mon May 27 15:27:17 CEST 2013
Hello Tolga,
Please test the attached patch.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
On 05/24/2013 08:33 PM, Tolga Tarhan wrote:
> Sure, here you go:
>
> version: opensips 1.9.0-tls (x86_64/linux)
> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, DISABLE_NAGLE,
> USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC,
> FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> svnrevision: unknown
> @(#) $Id: main.c 9790 2013-02-15 10:14:34Z bogdan_iancu $
> main.c compiled on 12:39:26 May 22 2013 with gcc 4.4.7
>
> It also has the couple patches that I've previously submitted (and
> you've since merged) in the build (the three related to GRUU bugs).
> They aren't a factor, as this particular instance is not acting as a
> registrar at all.
>
> Thanks,
> Tolga
>
>
>
> On Fri, May 24, 2013 at 9:33 AM, Bogdan-Andrei Iancu
> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
>
> Hi Tolga,
>
> Thanks for the info.
>
> What exact OpenSIPs version/revision are you using ? I need to
> correlate logs with sources.
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
>
>
> On 05/22/2013 11:02 PM, Tolga Tarhan wrote:
>> Sorry for the self-reply -- here's the (same) stacktraces with
>> line numbers and params:
>>
>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6
>> #2 0x000000000049d370 in qm_free (qm=<value optimized out>,
>> p=<value optimized out>, file=0x7ffccb25df64 "logic.c",
>> func=<value optimized out>, line=755) at mem/q_malloc.c:450
>> #3 0x00007ffccb253804 in process_bridge_200OK
>> (msg=0x7ffcccdd2600, extra_headers=0xd5, body=<value optimized
>> out>, tuple=0x7ffcc9518de0, entity=<value optimized out>) at
>> logic.c:755
>> #4 0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value
>> optimized out>, msg=0x7ffcccdd2600, key=<value optimized out>,
>> body=0x7fffc2ce15d0, extra_headers=0x7fffc2ce15c0,
>> b2bl_key=0x7fffc2ce23f0, hash_index=649, local_index=1)
>> at logic.c:1133
>> #5 0x00007ffccb2565e1 in b2b_logic_notify (src=1,
>> msg=0x7ffcccdd2600, key=0x7ffcc9525e40, type=1,
>> param=0x7fffc2ce23f0) at logic.c:2040
>> #6 0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110,
>> htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678
>> #7 0x00007ffccc744b71 in run_trans_callbacks (type=256,
>> trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value
>> optimized out>, code=<value optimized out>) at t_hooks.c:212
>> #8 0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110,
>> p_msg=<value optimized out>, branch=<value optimized out>,
>> msg_status=<value optimized out>, cancel_bitmap=<value optimized
>> out>) at t_reply.c:1391
>> #9 0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600)
>> at t_reply.c:1540
>> #10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at
>> forward.c:574
>> #11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>,
>> len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207
>> #12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424
>> #13 0x000000000042d763 in main_loop (argc=<value optimized out>,
>> argv=<value optimized out>) at main.c:884
>> #14 main (argc=<value optimized out>, argv=<value optimized out>)
>> at main.c:1557
>>
>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6
>> #2 0x000000000049d370 in qm_free (qm=<value optimized out>,
>> p=<value optimized out>, file=0x7ffccb262d75 "records.c",
>> func=<value optimized out>, line=595) at mem/q_malloc.c:450
>> #3 0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0,
>> hash_index=<value optimized out>, not_del_b2be=1) at records.c:595
>> #4 0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705
>> #5 0x000000000046dbf2 in destroy_modules () at sr_module.c:371
>> #6 0x00000000004298a1 in cleanup (show_status=1) at main.c:348
>> #7 0x000000000042a360 in handle_sigs () at main.c:549
>> #8 0x000000000042db66 in main_loop (argc=<value optimized out>,
>> argv=<value optimized out>) at main.c:1024
>> #9 main (argc=<value optimized out>, argv=<value optimized out>)
>> at main.c:1557
>>
>> Thanks,
>> Tolga
>>
>>
>> On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan
>> <tolga at netbrains.com <mailto:tolga at netbrains.com>> wrote:
>>
>> Thank you -- I've recompiled and enabled the memory debug. I
>> have the log file from the whole experience available here:
>>
>> http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log
>>
>> (note - real phone numbers and domain names in the log have
>> been replaced with placeholders)
>>
>> The key item seems to be:
>>
>> CRITICAL:core:qm_free: freeing already freed pointer, first
>> free: logic.c: process_bridge_200OK(740) - aborting
>>
>> Although this appears to be after we've already decided we're
>> going to crash, as I see "INFO:core:cleanup: cleanup" and
>> "NFO:core:handle_sigs: child process 24788 exited by a signal
>> 6" above this part of the log.
>>
>> Also worth noting is the existance of
>> "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up
>> here" throughout the log.
>>
>> Also, here's the stack trace at crash time. Note that there
>> were two core files generated for the same crash, so this is
>> the backtrace from each:
>>
>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6
>> #2 0x000000000049d370 in qm_free ()
>> #3 0x00007ffccb253804 in process_bridge_200OK () from
>> /usr/lib64/opensips/modules/b2b_logic.so
>> #4 0x00007ffccb254ba2 in b2b_logic_notify_reply () from
>> /usr/lib64/opensips/modules/b2b_logic.so
>> #5 0x00007ffccb2565e1 in b2b_logic_notify () from
>> /usr/lib64/opensips/modules/b2b_logic.so
>> #6 0x00007ffccb47a7a7 in b2b_tm_cback () from
>> /usr/lib64/opensips/modules/b2b_entities.so
>> #7 0x00007ffccc744b71 in run_trans_callbacks () from
>> /usr/lib64/opensips/modules/tm.so
>> #8 0x00007ffccc74fa12 in local_reply () from
>> /usr/lib64/opensips/modules/tm.so
>> #9 0x00007ffccc750cc5 in reply_received () from
>> /usr/lib64/opensips/modules/tm.so
>> #10 0x00000000004266da in forward_reply ()
>> #11 0x0000000000452ad8 in receive_msg ()
>> #12 0x0000000000496c95 in udp_rcv_loop ()
>> #13 0x000000000042d763 in main ()
>>
>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6
>> #2 0x000000000049d370 in qm_free ()
>> #3 0x00007ffccb25a003 in b2bl_delete () from
>> /usr/lib64/opensips/modules/b2b_logic.so
>> #4 0x00007ffccb25a3d5 in destroy_b2bl_htable () from
>> /usr/lib64/opensips/modules/b2b_logic.so
>> #5 0x000000000046dbf2 in destroy_modules ()
>> #6 0x00000000004298a1 in cleanup ()
>> #7 0x000000000042a360 in handle_sigs ()
>> #8 0x000000000042db66 in main ()
>>
>> I am unfamiliar with this codebase. Can anyone garner
>> anything useful from the logs?
>>
>> Thanks,
>> Tolga
>>
>>
>>
>> On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu
>> <bogdan at opensips.org <mailto:bogdan at opensips.org>> wrote:
>>
>> Hello Tolga,
>>
>> The crash seems to be in the memory manager, most
>> probably because of memory corruption. To troubleshoot
>> such issues you need to compile-in the memory debugger -
>> see
>> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem
>> .
>>
>> Using memlog=1 + memdump=10 you should get a lot of logs
>> related to mem ops, including a final report + abort()
>> when the corruption is detected.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com
>>
>>
>> On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
>>> Hello,
>>>
>>> While using the B2BUA module in OpenSIPS 1.9.0,
>>> we've encountered a consistent segfault. We are using a
>>> refer scenario just like the one in the B2BUA sample
>>> docs, and after several REFERs for the same call (to
>>> different destinations), OpenSIPS crashes with a
>>> segfault. The core file indicates the following backtrace:
>>>
>>> #0 0x000000000049a334 in fm_malloc ()
>>> #1 0x00007fdaecd96230 in shm_malloc_unsafe
>>> (type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
>>> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320,
>>> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at
>>> ../../mem/shm_mem.h:248
>>> #2 shm_malloc (type=B2B_CLIENT,
>>> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360,
>>> from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
>>> optimized out>, msg=0x0) at ../../mem/shm_mem.h:258
>>> #3 b2bl_create_new_entity (type=B2B_CLIENT,
>>> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360,
>>> from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
>>> optimized out>, msg=0x0) at logic.c:293
>>> #4 0x00007fdaecd96882 in b2bl_new_client (to_uri=<value
>>> optimized out>, from_uri=<value optimized out>,
>>> tuple=<value optimized out>, ssid=0x7fdaeb026c00,
>>> msg=<value optimized out>) at logic.c:607
>>> #5 0x00007fdaecda3579 in process_bridge_200OK
>>> (msg=0x7fdaee8e8b30, extra_headers=0x7fdaeb03d578,
>>> body=<value optimized out>, tuple=0x7fdaeb01ada8,
>>> entity=<value optimized out>) at logic.c:816
>>> #6 0x00007fdaecda46c2 in b2b_logic_notify_reply
>>> (src=<value optimized out>, msg=0x7fdaee8e8b30,
>>> key=<value optimized out>, body=0x7fff2d3468b0,
>>> extra_headers=0x7fff2d3468a0, b2bl_key=0x7fff2d3476d0,
>>> hash_index=649, local_index=0)
>>> at logic.c:1133
>>> #7 0x00007fdaecda6081 in b2b_logic_notify (src=1,
>>> msg=0x7fdaee8e8b30, key=0x7fdaeb03d500, type=1,
>>> param=0x7fff2d3476d0) at logic.c:2040
>>> #8 0x00007fdaecfca7ad in b2b_tm_cback
>>> (t=0x7fdaeb054118, htable=0x7fdaeb014630, ps=<value
>>> optimized out>) at dlg.c:2678
>>> #9 0x00007fdaee291441 in run_trans_callbacks (type=256,
>>> trans=0x7fdaeb054118, req=<value optimized out>,
>>> rpl=<value optimized out>, code=<value optimized out>)
>>> at t_hooks.c:212
>>> #10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118,
>>> p_msg=<value optimized out>, branch=<value optimized
>>> out>, msg_status=<value optimized out>,
>>> cancel_bitmap=<value optimized out>) at t_reply.c:1391
>>> #11 0x00007fdaee29d31d in reply_received
>>> (p_msg=0x7fdaee8e8b30) at t_reply.c:1540
>>> #12 0x000000000042625a in forward_reply ()
>>> #13 0x0000000000451c28 in receive_msg ()
>>> #14 0x0000000000494e45 in udp_rcv_loop ()
>>> #15 0x000000000042d1a3 in main ()
>>>
>>> I'm not really sure how to diagnose this one. Any
>>> hints/fixes/suggestions would be very appreciated.
>>>
>>> Thanks,
>>> Tolga
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20130527/6383f46d/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: b2b_logic_2free.patch
Type: text/x-patch
Size: 422 bytes
Desc: not available
URL: <http://lists.opensips.org/pipermail/users/attachments/20130527/6383f46d/attachment-0001.bin>
More information about the Users
mailing list