[OpenSIPS-Users] B2BUA Segfault

Mon May 27 23:31:54 CEST 2013

Hi Tolga,

Yes the patch is correct, please give it a try.

Thanks and regards,
Bogdan

Sent from Samsung Mobile

-------- Original message --------
From: Tolga Tarhan <tolga at netbrains.com> 
Date:  
To: Bogdan-Andrei Iancu <bogdan at opensips.org> 
Cc: OpenSIPS users mailling list <users at lists.opensips.org> 
Subject: Re: [OpenSIPS-Users] B2BUA Segfault 

Bogdan,

The patch doesn't build:

Compiling logic.c
logic.c: In function 'process_bridge_200OK':
logic.c:741: error: invalid operands to binary == (have 'char *' and 'str')
logic.c:742: error: incompatible types when assigning to type 'str' from type 'int'
make[1]: *** [logic.o] Error 1

I've attached what I believe is a corrected patch. Let me know if this is what you intended.

I'm trying the (corrected) patch now to see if it resolves the original issue. I'll let you know on my findings shortly.

Thanks,
Tolga

On Mon, May 27, 2013 at 6:27 AM, Bogdan-Andrei Iancu <bogdan at opensips.org> wrote:
Hello Tolga,

Please test the attached patch.

Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 05/24/2013 08:33 PM, Tolga Tarhan wrote:
Sure, here you go:

version: opensips 1.9.0-tls (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
svnrevision: unknown
@(#) $Id: main.c 9790 2013-02-15 10:14:34Z bogdan_iancu $
main.c compiled on 12:39:26 May 22 2013 with gcc 4.4.7

It also has the couple patches that I've previously submitted (and you've since merged) in the build (the three related to GRUU bugs). They aren't a factor, as this particular instance is not acting as a registrar at all.

Thanks,
Tolga

On Fri, May 24, 2013 at 9:33 AM, Bogdan-Andrei Iancu <bogdan at opensips.org> wrote:
Hi Tolga,

Thanks for the info.

What exact OpenSIPs version/revision are you using ? I need to correlate logs with sources.

Regards,
 Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 05/22/2013 11:02 PM, Tolga Tarhan wrote:
Sorry for the self-reply -- here's the (same) stacktraces with line numbers and                       params:                      

#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1 0x0000003564c34085 in abort () from /lib64/libc.so.6
#2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value optimized out>, file=0x7ffccb25df64 "logic.c", func=<value optimized out>, line=755) at mem/q_malloc.c:450
#3 0x00007ffccb253804 in process_bridge_200OK (msg=0x7ffcccdd2600, extra_headers=0xd5, body=<value optimized out>, tuple=0x7ffcc9518de0, entity=<value optimized out>) at logic.c:755
#4 0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value optimized out>, msg=0x7ffcccdd2600, key=<value optimized out>, body=0x7fffc2ce15d0, extra_headers=0x7fffc2ce15c0, b2bl_key=0x7fffc2ce23f0, hash_index=649, local_index=1)
  at logic.c:1133
#5 0x00007ffccb2565e1 in b2b_logic_notify (src=1, msg=0x7ffcccdd2600, key=0x7ffcc9525e40, type=1, param=0x7fffc2ce23f0) at logic.c:2040
#6 0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110, htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678
#7 0x00007ffccc744b71 in run_trans_callbacks (type=256, trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value optimized out>, code=<value optimized out>) at t_hooks.c:212
#8 0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110, p_msg=<value optimized out>, branch=<value optimized out>, msg_status=<value optimized out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391
#9 0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600) at t_reply.c:1540
#10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at forward.c:574
#11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>, len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207
#12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424
#13 0x000000000042d763 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:884
#14 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1557

#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1 0x0000003564c34085 in abort () from /lib64/libc.so.6
#2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value optimized out>, file=0x7ffccb262d75 "records.c", func=<value optimized out>, line=595) at mem/q_malloc.c:450
#3 0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0, hash_index=<value optimized out>, not_del_b2be=1) at records.c:595
#4 0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705
#5 0x000000000046dbf2 in destroy_modules () at sr_module.c:371
#6 0x00000000004298a1 in cleanup (show_status=1) at main.c:348
#7 0x000000000042a360 in handle_sigs () at main.c:549
#8 0x000000000042db66 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:1024
#9 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1557

Thanks,
Tolga

On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan <tolga at netbrains.com> wrote:
Thank you -- I've recompiled and enabled the memory debug. I have the log file from the whole experience available here:

http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log

(note - real phone numbers and domain names in the log have been replaced with placeholders)

The key item seems to be:

CRITICAL:core:qm_free: freeing already freed pointer, first free: logic.c: process_bridge_200OK(740) - aborting

Although this appears to be after we've already decided we're going to crash, as I see "INFO:core:cleanup: cleanup" and "NFO:core:handle_sigs: child process 24788 exited by a signal 6" above this part of the log.

Also worth noting is the existance of "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here" throughout the log.

Also, here's the stack trace at crash time. Note that there were two core files generated for the same crash, so this is the backtrace from each:

#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1 0x0000003564c34085 in abort () from /lib64/libc.so.6
#2 0x000000000049d370 in qm_free ()
#3 0x00007ffccb253804 in process_bridge_200OK () from /usr/lib64/opensips/modules/b2b_logic.so
#4 0x00007ffccb254ba2 in b2b_logic_notify_reply () from /usr/lib64/opensips/modules/b2b_logic.so
#5 0x00007ffccb2565e1 in b2b_logic_notify () from /usr/lib64/opensips/modules/b2b_logic.so
#6 0x00007ffccb47a7a7 in b2b_tm_cback () from /usr/lib64/opensips/modules/b2b_entities.so
#7 0x00007ffccc744b71 in run_trans_callbacks () from /usr/lib64/opensips/modules/tm.so
#8 0x00007ffccc74fa12 in local_reply () from /usr/lib64/opensips/modules/tm.so
#9 0x00007ffccc750cc5 in reply_received () from /usr/lib64/opensips/modules/tm.so
#10 0x00000000004266da in forward_reply ()
#11 0x0000000000452ad8 in receive_msg ()
#12 0x0000000000496c95 in udp_rcv_loop ()
#13 0x000000000042d763 in main ()

#0 0x0000003564c328a5 in raise () from /lib64/libc.so.6
#1 0x0000003564c34085 in abort () from /lib64/libc.so.6
#2 0x000000000049d370 in qm_free ()
#3 0x00007ffccb25a003 in b2bl_delete () from /usr/lib64/opensips/modules/b2b_logic.so
#4 0x00007ffccb25a3d5 in destroy_b2bl_htable () from /usr/lib64/opensips/modules/b2b_logic.so
#5 0x000000000046dbf2 in destroy_modules ()
#6 0x00000000004298a1 in cleanup ()
#7 0x000000000042a360 in handle_sigs ()
#8 0x000000000042db66 in main ()

I am unfamiliar with this codebase. Can anyone garner anything useful from the logs?

Thanks,
Tolga

On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu <bogdan at opensips.org> wrote:
Hello Tolga,

The crash seems to be in the memory manager, most probably because of memory corruption. To troubleshoot such issues you need to compile-in the memory debugger - see http://www.opensips.org/Documentation/TroubleShooting-OutOfMem .

Using memlog=1 + memdump=10 you should get a lot of logs related to mem ops, including a final report + abort() when the corruption is detected.

Regards,
 Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 05/22/2013 01:29 AM, Tolga Tarhan wrote:
Hello,

While using the B2BUA module in OpenSIPS 1.9.0, we've encountered a consistent segfault. We are using a refer scenario just like the one in the B2BUA sample docs, and after several REFERs for the same call (to different destinations), OpenSIPS crashes with a segfault. The core file indicates the following backtrace:

#0 0x000000000049a334 in fm_malloc ()
#1 0x00007fdaecd96230 in shm_malloc_unsafe (type=B2B_CLIENT, entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value optimized out>, msg=0x0) at ../../mem/shm_mem.h:248
#2 shm_malloc (type=B2B_CLIENT, entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value optimized out>, msg=0x0) at ../../mem/shm_mem.h:258
#3 b2bl_create_new_entity (type=B2B_CLIENT, entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value optimized out>, msg=0x0) at logic.c:293
#4 0x00007fdaecd96882 in b2bl_new_client (to_uri=<value optimized out>, from_uri=<value optimized out>, tuple=<value optimized out>, ssid=0x7fdaeb026c00, msg=<value optimized out>) at logic.c:607
#5 0x00007fdaecda3579 in process_bridge_200OK (msg=0x7fdaee8e8b30, extra_headers=0x7fdaeb03d578, body=<value optimized out>, tuple=0x7fdaeb01ada8, entity=<value optimized out>) at logic.c:816
#6 0x00007fdaecda46c2 in b2b_logic_notify_reply (src=<value optimized out>, msg=0x7fdaee8e8b30, key=<value optimized out>, body=0x7fff2d3468b0, extra_headers=0x7fff2d3468a0, b2bl_key=0x7fff2d3476d0, hash_index=649, local_index=0)
  at logic.c:1133
#7 0x00007fdaecda6081 in b2b_logic_notify (src=1, msg=0x7fdaee8e8b30, key=0x7fdaeb03d500, type=1, param=0x7fff2d3476d0) at logic.c:2040
#8 0x00007fdaecfca7ad in b2b_tm_cback (t=0x7fdaeb054118, htable=0x7fdaeb014630, ps=<value optimized out>) at dlg.c:2678
#9 0x00007fdaee291441 in run_trans_callbacks (type=256, trans=0x7fdaeb054118, req=<value optimized out>, rpl=<value optimized out>, code=<value optimized out>) at t_hooks.c:212
#10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118, p_msg=<value optimized out>, branch=<value optimized out>, msg_status=<value optimized out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391
#11 0x00007fdaee29d31d in reply_received (p_msg=0x7fdaee8e8b30) at t_reply.c:1540
#12 0x000000000042625a in forward_reply ()
#13 0x0000000000451c28 in receive_msg ()
#14 0x0000000000494e45 in udp_rcv_loop ()
#15 0x000000000042d1a3 in main ()

I'm not really sure how to diagnose this one. Any hints/fixes/suggestions would be very appreciated.

Thanks,
Tolga

_______________________________________________
Users mailing list
Users at lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20130528/70fc855f/attachment-0001.htm>