[OpenSIPS-Users] B2BUA Segfault

Tolga Tarhan tolga at netbrains.com
Tue May 28 00:35:15 CEST 2013 

On a related note, however, is this of any concern:

May 27 15:35:02 b2b-dev1 /usr/sbin/opensips[19455]:
CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here

Or for that matter, any of these:

May 27 15:35:22 b2b-dev1 /usr/sbin/opensips[19455]:
WARNING:b2b_logic:b2bl_delete_entity: entity [0x7f8840c171d8]->[] not found
for tuple [383.0]
May 27 15:35:19 b2b-dev1 /usr/sbin/opensips[19458]:
ERROR:b2b_logic:process_bridge_negreply: unexpected entity_no [2] for tuple
[0x7f8840c19378]
May 27 15:35:19 b2b-dev1 /usr/sbin/opensips[19458]:
ERROR:b2b_logic:b2b_logic_notify_reply: Failed to process negative reply
while in bridging state


--
Tolga


On Mon, May 27, 2013 at 3:32 PM, Tolga Tarhan <tolga at netbrains.com> wrote:

> Bogdan,
>
> We've tried out the patch and it seems to resolve the issue. We will
> continue to test to see if we can reproduce the problem, but for now, it
> appears fixed.
>
> Thanks so much for your help on this.
>
> --
> Tolga
>
>
> On Mon, May 27, 2013 at 2:31 PM, Bogdan-Andrei Iancu <bogdan at opensips.org>wrote:
>
>> Hi Tolga,
>>
>> Yes the patch is correct, please give it a try.
>>
>> Thanks and regards,
>> Bogdan
>>
>>
>> Sent from Samsung Mobile
>>
>>
>>
>> -------- Original message --------
>> From: Tolga Tarhan <tolga at netbrains.com>
>> Date:
>> To: Bogdan-Andrei Iancu <bogdan at opensips.org>
>> Cc: OpenSIPS users mailling list <users at lists.opensips.org>
>> Subject: Re: [OpenSIPS-Users] B2BUA Segfault
>>
>>
>> Bogdan,
>>
>> The patch doesn't build:
>>
>> Compiling logic.c
>> logic.c: In function 'process_bridge_200OK':
>> logic.c:741: error: invalid operands to binary == (have 'char *' and
>> 'str')
>> logic.c:742: error: incompatible types when assigning to type 'str' from
>> type 'int'
>> make[1]: *** [logic.o] Error 1
>>
>> I've attached what I believe is a corrected patch. Let me know if this is
>> what you intended.
>>
>> I'm trying the (corrected) patch now to see if it resolves the original
>> issue. I'll let you know on my findings shortly.
>>
>> Thanks,
>> Tolga
>>
>>
>>
>> On Mon, May 27, 2013 at 6:27 AM, Bogdan-Andrei Iancu <bogdan at opensips.org
>> > wrote:
>>
>>> **
>>> Hello Tolga,
>>>
>>> Please test the attached patch.
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>
>>>
>>> On 05/24/2013 08:33 PM, Tolga Tarhan wrote:
>>>
>>>  Sure, here you go:
>>>
>>>  version: opensips 1.9.0-tls (x86_64/linux)
>>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, DISABLE_NAGLE, USE_MCAST,
>>> SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>> svnrevision: unknown
>>>  @(#) $Id: main.c 9790 2013-02-15 10:14:34Z bogdan_iancu $
>>> main.c compiled on 12:39:26 May 22 2013 with gcc 4.4.7
>>>
>>>  It also has the couple patches that I've previously submitted (and
>>> you've since merged) in the build (the three related to GRUU bugs). They
>>> aren't a factor, as this particular instance is not acting as a registrar
>>> at all.
>>>
>>>  Thanks,
>>> Tolga
>>>
>>>
>>>
>>> On Fri, May 24, 2013 at 9:33 AM, Bogdan-Andrei Iancu <
>>> bogdan at opensips.org> wrote:
>>>
>>>>  Hi Tolga,
>>>>
>>>> Thanks for the info.
>>>>
>>>> What exact OpenSIPs version/revision are you using ? I need to
>>>> correlate logs with sources.
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>
>>>>
>>>>   On 05/22/2013 11:02 PM, Tolga Tarhan wrote:
>>>>
>>>> Sorry for the self-reply -- here's the (same) stacktraces with line
>>>> numbers and params:
>>>>
>>>>  #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>>> #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>>> #2  0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value
>>>> optimized out>, file=0x7ffccb25df64 "logic.c", func=<value optimized out>,
>>>> line=755) at mem/q_malloc.c:450
>>>> #3  0x00007ffccb253804 in process_bridge_200OK (msg=0x7ffcccdd2600,
>>>> extra_headers=0xd5, body=<value optimized out>, tuple=0x7ffcc9518de0,
>>>> entity=<value optimized out>) at logic.c:755
>>>> #4  0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value optimized
>>>> out>, msg=0x7ffcccdd2600, key=<value optimized out>, body=0x7fffc2ce15d0,
>>>> extra_headers=0x7fffc2ce15c0, b2bl_key=0x7fffc2ce23f0, hash_index=649,
>>>> local_index=1)
>>>>     at logic.c:1133
>>>> #5  0x00007ffccb2565e1 in b2b_logic_notify (src=1, msg=0x7ffcccdd2600,
>>>> key=0x7ffcc9525e40, type=1, param=0x7fffc2ce23f0) at logic.c:2040
>>>> #6  0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110,
>>>> htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678
>>>> #7  0x00007ffccc744b71 in run_trans_callbacks (type=256,
>>>> trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value optimized out>,
>>>> code=<value optimized out>) at t_hooks.c:212
>>>> #8  0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110, p_msg=<value
>>>> optimized out>, branch=<value optimized out>, msg_status=<value optimized
>>>> out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391
>>>> #9  0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600) at
>>>> t_reply.c:1540
>>>> #10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at
>>>> forward.c:574
>>>> #11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>,
>>>> len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207
>>>> #12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424
>>>> #13 0x000000000042d763 in main_loop (argc=<value optimized out>,
>>>> argv=<value optimized out>) at main.c:884
>>>> #14 main (argc=<value optimized out>, argv=<value optimized out>) at
>>>> main.c:1557
>>>>
>>>>  #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>>> #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>>> #2  0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value
>>>> optimized out>, file=0x7ffccb262d75 "records.c", func=<value optimized
>>>> out>, line=595) at mem/q_malloc.c:450
>>>> #3  0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0,
>>>> hash_index=<value optimized out>, not_del_b2be=1) at records.c:595
>>>> #4  0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705
>>>> #5  0x000000000046dbf2 in destroy_modules () at sr_module.c:371
>>>> #6  0x00000000004298a1 in cleanup (show_status=1) at main.c:348
>>>> #7  0x000000000042a360 in handle_sigs () at main.c:549
>>>> #8  0x000000000042db66 in main_loop (argc=<value optimized out>,
>>>> argv=<value optimized out>) at main.c:1024
>>>> #9  main (argc=<value optimized out>, argv=<value optimized out>) at
>>>> main.c:1557
>>>>
>>>>  Thanks,
>>>> Tolga
>>>>
>>>>
>>>> On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan <tolga at netbrains.com>wrote:
>>>>
>>>>> Thank you -- I've recompiled and enabled the memory debug. I have the
>>>>> log file from the whole experience available here:
>>>>>
>>>>>  http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log
>>>>>
>>>>>  (note - real phone numbers and domain names in the log have been
>>>>> replaced with placeholders)
>>>>>
>>>>>  The key item seems to be:
>>>>>
>>>>>  CRITICAL:core:qm_free: freeing already freed pointer, first free:
>>>>> logic.c: process_bridge_200OK(740) - aborting
>>>>>
>>>>>  Although this appears to be after we've already decided we're going
>>>>> to crash, as I see "INFO:core:cleanup: cleanup" and "NFO:core:handle_sigs:
>>>>> child process 24788 exited by a signal 6" above this part of the log.
>>>>>
>>>>>  Also worth noting is the existance of
>>>>> "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here"
>>>>> throughout the log.
>>>>>
>>>>>  Also, here's the stack trace at crash time. Note that there were two
>>>>> core files generated for the same crash, so this is the backtrace from each:
>>>>>
>>>>>  #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>>>> #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>>>> #2  0x000000000049d370 in qm_free ()
>>>>> #3  0x00007ffccb253804 in process_bridge_200OK () from
>>>>> /usr/lib64/opensips/modules/b2b_logic.so
>>>>> #4  0x00007ffccb254ba2 in b2b_logic_notify_reply () from
>>>>> /usr/lib64/opensips/modules/b2b_logic.so
>>>>> #5  0x00007ffccb2565e1 in b2b_logic_notify () from
>>>>> /usr/lib64/opensips/modules/b2b_logic.so
>>>>> #6  0x00007ffccb47a7a7 in b2b_tm_cback () from
>>>>> /usr/lib64/opensips/modules/b2b_entities.so
>>>>> #7  0x00007ffccc744b71 in run_trans_callbacks () from
>>>>> /usr/lib64/opensips/modules/tm.so
>>>>> #8  0x00007ffccc74fa12 in local_reply () from
>>>>> /usr/lib64/opensips/modules/tm.so
>>>>> #9  0x00007ffccc750cc5 in reply_received () from
>>>>> /usr/lib64/opensips/modules/tm.so
>>>>> #10 0x00000000004266da in forward_reply ()
>>>>> #11 0x0000000000452ad8 in receive_msg ()
>>>>> #12 0x0000000000496c95 in udp_rcv_loop ()
>>>>> #13 0x000000000042d763 in main ()
>>>>>
>>>>>  #0  0x0000003564c328a5 in raise () from /lib64/libc.so.6
>>>>> #1  0x0000003564c34085 in abort () from /lib64/libc.so.6
>>>>> #2  0x000000000049d370 in qm_free ()
>>>>> #3  0x00007ffccb25a003 in b2bl_delete () from
>>>>> /usr/lib64/opensips/modules/b2b_logic.so
>>>>> #4  0x00007ffccb25a3d5 in destroy_b2bl_htable () from
>>>>> /usr/lib64/opensips/modules/b2b_logic.so
>>>>> #5  0x000000000046dbf2 in destroy_modules ()
>>>>> #6  0x00000000004298a1 in cleanup ()
>>>>> #7  0x000000000042a360 in handle_sigs ()
>>>>> #8  0x000000000042db66 in main ()
>>>>>
>>>>>  I am unfamiliar with this codebase. Can anyone garner anything
>>>>> useful from the logs?
>>>>>
>>>>>  Thanks,
>>>>> Tolga
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu <
>>>>> bogdan at opensips.org> wrote:
>>>>>
>>>>>>  Hello Tolga,
>>>>>>
>>>>>> The crash seems to be in the memory manager, most probably because of
>>>>>> memory corruption. To troubleshoot such issues you need to compile-in the
>>>>>> memory debugger - see
>>>>>> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem .
>>>>>>
>>>>>> Using memlog=1 + memdump=10 you should get a lot of logs related to
>>>>>> mem ops, including a final report + abort() when the corruption is detected.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>
>>>>>>
>>>>>> On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
>>>>>>
>>>>>>  Hello,
>>>>>>
>>>>>>  While using the B2BUA module in OpenSIPS 1.9.0, we've encountered a
>>>>>> consistent segfault. We are using a refer scenario just like the one in the
>>>>>> B2BUA sample docs, and after several REFERs for the same call (to different
>>>>>> destinations), OpenSIPS crashes with a segfault. The core file indicates
>>>>>> the following backtrace:
>>>>>>
>>>>>>  #0  0x000000000049a334 in fm_malloc ()
>>>>>> #1  0x00007fdaecd96230 in shm_malloc_unsafe (type=B2B_CLIENT,
>>>>>> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320,
>>>>>> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at
>>>>>> ../../mem/shm_mem.h:248
>>>>>> #2  shm_malloc (type=B2B_CLIENT, entity_id=0x7fdaee8ec750,
>>>>>> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value
>>>>>> optimized out>, msg=0x0) at ../../mem/shm_mem.h:258
>>>>>> #3  b2bl_create_new_entity (type=B2B_CLIENT,
>>>>>> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320,
>>>>>> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at logic.c:293
>>>>>> #4  0x00007fdaecd96882 in b2bl_new_client (to_uri=<value optimized
>>>>>> out>, from_uri=<value optimized out>, tuple=<value optimized out>,
>>>>>> ssid=0x7fdaeb026c00, msg=<value optimized out>) at logic.c:607
>>>>>> #5  0x00007fdaecda3579 in process_bridge_200OK (msg=0x7fdaee8e8b30,
>>>>>> extra_headers=0x7fdaeb03d578, body=<value optimized out>,
>>>>>> tuple=0x7fdaeb01ada8, entity=<value optimized out>) at logic.c:816
>>>>>> #6  0x00007fdaecda46c2 in b2b_logic_notify_reply (src=<value
>>>>>> optimized out>, msg=0x7fdaee8e8b30, key=<value optimized out>,
>>>>>> body=0x7fff2d3468b0, extra_headers=0x7fff2d3468a0, b2bl_key=0x7fff2d3476d0,
>>>>>> hash_index=649, local_index=0)
>>>>>>     at logic.c:1133
>>>>>> #7  0x00007fdaecda6081 in b2b_logic_notify (src=1,
>>>>>> msg=0x7fdaee8e8b30, key=0x7fdaeb03d500, type=1, param=0x7fff2d3476d0) at
>>>>>> logic.c:2040
>>>>>> #8  0x00007fdaecfca7ad in b2b_tm_cback (t=0x7fdaeb054118,
>>>>>> htable=0x7fdaeb014630, ps=<value optimized out>) at dlg.c:2678
>>>>>> #9  0x00007fdaee291441 in run_trans_callbacks (type=256,
>>>>>> trans=0x7fdaeb054118, req=<value optimized out>, rpl=<value optimized out>,
>>>>>> code=<value optimized out>) at t_hooks.c:212
>>>>>> #10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118, p_msg=<value
>>>>>> optimized out>, branch=<value optimized out>, msg_status=<value optimized
>>>>>> out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391
>>>>>> #11 0x00007fdaee29d31d in reply_received (p_msg=0x7fdaee8e8b30) at
>>>>>> t_reply.c:1540
>>>>>> #12 0x000000000042625a in forward_reply ()
>>>>>> #13 0x0000000000451c28 in receive_msg ()
>>>>>> #14 0x0000000000494e45 in udp_rcv_loop ()
>>>>>> #15 0x000000000042d1a3 in main ()
>>>>>>
>>>>>>  I'm not really sure how to diagnose this one. Any
>>>>>> hints/fixes/suggestions would be very appreciated.
>>>>>>
>>>>>>  Thanks,
>>>>>> Tolga
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20130527/3ff12c47/attachment-0001.htm>

More information about the Users mailing list