[OpenSIPS-Users] opensips crash when doing load test

Tue Oct 4 15:08:18 CEST 2016

Hi Bogdan,

I have printed the "t" variable value. It says "$1 = (struct cell *) 0x0".
Is there any link available to upload/share the core files? Because size of the core file 45KB and couldn't attach in e-mail.

(gdb) frame 0
#0  final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
383             if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
(gdb) list +
378             current_processing_ctx = my_ctx;
379             /* set the T context too */
380             set_t( t );
381
382             /* out-of-lock do the cancel I/O */
383             if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
384                     set_cancel_extra_hdrs( CANCEL_REASON_SIP_480, sizeof(CANCEL_REASON_SIP_480)-1);
385                     cancel_branch(t, r_buf->branch );
386                     set_cancel_extra_hdrs( NULL, 0);
387             }
(gdb) p t
$1 = (struct cell *) 0x0

Regards,
Agalya

From: Bogdan-Andrei Iancu [mailto:bogdan at opensips.org]
Sent: Tuesday, October 04, 2016 6:44 AM
To: Ramachandran, Agalya (Contractor) <Agalya_Ramachandran at comcast.com>; OpenSIPS users mailling list <users at lists.opensips.org>
Subject: Re: opensips crash when doing load test

Hello Agalya,

In frame 0, could you print the "t" variable ? BTW, is there any way to get access to the core file ?

Best regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

http://www.opensips-solutions.com
On 30.09.2016 16:48, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,

One more dump

#0  final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
383             if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64 libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 nspr-4.10.8-2.el7_1.x86_64 nss-3.19.1-19.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 nss-util-3.19.1-9.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64 openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0  final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
#1  timer_routine (ticks=<optimized out>, set=<optimized out>) at timer.c:1066
#2  0x000000000049d3cd in handle_timer_job () at timer.c:597
#3  0x000000000059223e in handle_io (fm=<optimized out>, fm=<optimized out>, fm=<optimized out>, idx=<optimized out>, event_type=2)
    at net/net_udp.c:265
#4  io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, repeat=<optimized out>) at net/../io_wait_loop.h:225
#5  udp_rcv_loop (si=si at entry=0x7fded841dcb0) at net/net_udp.c:308
#6  0x0000000000593e05 in udp_start_processes (chd_rank=chd_rank at entry=0x84c570 <chd_rank.10940>, startup_done=startup_done at entry=0x0)
    at net/net_udp.c:448
#7  0x0000000000419b73 in main_loop () at main.c:736
#8  main (argc=<optimized out>, argv=<optimized out>) at main.c:1282

From: Ramachandran, Agalya (Contractor)
Sent: Friday, September 30, 2016 9:44 AM
To: 'Bogdan-Andrei Iancu' <bogdan at opensips.org><mailto:bogdan at opensips.org>; OpenSIPS users mailling list <users at lists.opensips.org><mailto:users at lists.opensips.org>
Subject: opensips crash when doing load test

Hi Bogdan,

I have taken 2.1.4 code base, added the below fixes on top of it

1.       Fix potential starvation for the timer tasks. 66c337cc89a2b5c3e1bbd78c289371efcda21886

2.       rest_client: Improve polling logic during TCP connect - c40f2b057209b8078520abc95e5d1bf5e5a28db6

3.       rest_client: Add "connect_poll_interval" module parameter - f5abfb3dee649a14ae39ad74d7f04454f815748f

4.       rest_client: Fix bogus condition -3ba7be6692c6987faecb9c4c904035d5e7c0f916

When I try to perform load test on that code base, randomly am seeing OpenSIPS is crashing.
Here is the full trace of the core file. Let me know if you need any additional information or logs.
Am seeing crash is occurring at start_timer_extra_processes().

[root at poc-siplb-cmc-e-004 corefiles]# gdb /usr/local/sbin/opensips core.opensips.sig11.3977
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/sbin/opensips...done.
[New LWP 3977]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/opensips.pid'.
Program terminated with signal 11, Segmentation fault.
#0  final_response_handler (fr_tl=0x7f55fcd82948) at timer.c:383
383             if (is_invite(t) && should_cancel_branch(t, r_buf->branch) ) {
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64 libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 nspr-4.10.8-2.el7_1.x86_64 nss-3.19.1-19.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 nss-util-3.19.1-9.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64 openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0  final_response_handler (fr_tl=0x7f55fcd82948) at timer.c:383
#1  timer_routine (ticks=<optimized out>, set=<optimized out>) at timer.c:1066
#2  0x000000000049d03a in handle_timer_job () at timer.c:597
#3  start_timer_extra_processes (chd_rank=chd_rank at entry=0x84c570 <chd_rank.10940>) at timer.c:557
#4  0x0000000000419b9a in main_loop () at main.c:748
#5  main (argc=<optimized out>, argv=<optimized out>) at main.c:1282

Regards,
Agalya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20161004/d0a98e7a/attachment.htm>