[OpenSIPS-Users] opensips crash when doing load test

Tue Oct 4 12:44:15 CEST 2016

Hello Agalya,

In frame 0, could you print the "t" variable ? BTW, is there any way to 
get access to the core file ?

Best regards,

Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com

On 30.09.2016 16:48, Ramachandran, Agalya (Contractor) wrote:
>
> Hi Bogdan,
>
> One more dump
>
> #0 final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
>
> 383             if (is_invite(t) && should_cancel_branch(t, 
> r_buf->branch) ) {
>
> Missing separate debuginfos, use: debuginfo-install 
> cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64 
> keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64 
> libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64 
> libidn-1.28-4.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 
> libssh2-1.4.3-10.el7_2.1.x86_64 nspr-4.10.8-2.el7_1.x86_64 
> nss-3.19.1-19.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 
> nss-util-3.19.1-9.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64 
> openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64 
> xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
>
> (gdb) bt
>
> #0 final_response_handler (fr_tl=0x7fded2484110) at timer.c:383
>
> #1  timer_routine (ticks=<optimized out>, set=<optimized out>) at 
> timer.c:1066
>
> #2 0x000000000049d3cd in handle_timer_job () at timer.c:597
>
> #3 0x000000000059223e in handle_io (fm=<optimized out>, fm=<optimized 
> out>, fm=<optimized out>, idx=<optimized out>, event_type=2)
>
>     at net/net_udp.c:265
>
> #4 io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, 
> repeat=<optimized out>) at net/../io_wait_loop.h:225
>
> #5  udp_rcv_loop (si=si at entry=0x7fded841dcb0) at net/net_udp.c:308
>
> #6 0x0000000000593e05 in udp_start_processes 
> (chd_rank=chd_rank at entry=0x84c570 <chd_rank.10940>, 
> startup_done=startup_done at entry=0x0)
>
>     at net/net_udp.c:448
>
> #7 0x0000000000419b73 in main_loop () at main.c:736
>
> #8  main (argc=<optimized out>, argv=<optimized out>) at main.c:1282
>
> *From:* Ramachandran, Agalya (Contractor)
> *Sent:* Friday, September 30, 2016 9:44 AM
> *To:* 'Bogdan-Andrei Iancu' <bogdan at opensips.org>; OpenSIPS users 
> mailling list <users at lists.opensips.org>
> *Subject:* opensips crash when doing load test
>
> Hi Bogdan,
>
> I have taken 2.1.4 code base, added the below fixes on top of it
>
> 1.Fix potential starvation for the timer tasks. 
> 66c337cc89a2b5c3e1bbd78c289371efcda21886
>
> 2.rest_client: Improve polling logic during TCP connect - 
> c40f2b057209b8078520abc95e5d1bf5e5a28db6
>
> 3.rest_client: Add "connect_poll_interval" module parameter - 
> f5abfb3dee649a14ae39ad74d7f04454f815748f
>
> 4.rest_client: Fix bogus condition 
> -3ba7be6692c6987faecb9c4c904035d5e7c0f916
>
> When I try to perform load test on that code base, randomly am seeing 
> OpenSIPS is crashing.
>
> Here is the full trace of the core file. Let me know if you need any 
> additional information or logs.
>
> Am seeing crash is occurring at start_timer_extra_processes().
>
> [root at poc-siplb-cmc-e-004 corefiles]# gdb /usr/local/sbin/opensips 
> core.opensips.sig11.3977
>
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
>
> Copyright (C) 2013 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>
> and "show warranty" for details.
>
> This GDB was configured as "x86_64-redhat-linux-gnu".
>
> For bug reporting instructions, please see:
>
> <http://www.gnu.org/software/gdb/bugs/>...
>
> Reading symbols from /usr/local/sbin/opensips...done.
>
> [New LWP 3977]
>
> [Thread debugging using libthread_db enabled]
>
> Using host libthread_db library "/lib64/libthread_db.so.1".
>
> Core was generated by `/usr/local/sbin/opensips -P /var/run/opensips.pid'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0 final_response_handler (fr_tl=0x7f55fcd82948) at timer.c:383
>
> 383             if (is_invite(t) && should_cancel_branch(t, 
> r_buf->branch) ) {
>
> Missing separate debuginfos, use: debuginfo-install 
> cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64 
> keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-12.el7_2.x86_64 
> libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.centos.x86_64 
> libidn-1.28-4.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 
> libssh2-1.4.3-10.el7_2.1.x86_64 nspr-4.10.8-2.el7_1.x86_64 
> nss-3.19.1-19.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 
> nss-util-3.19.1-9.el7_2.x86_64 openldap-2.4.40-9.el7_2.x86_64 
> openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64 
> xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
>
> (gdb) bt
>
> #0 final_response_handler (fr_tl=0x7f55fcd82948) at timer.c:383
>
> #1  timer_routine (ticks=<optimized out>, set=<optimized out>) at 
> timer.c:1066
>
> #2 0x000000000049d03a in handle_timer_job () at timer.c:597
>
> #3 start_timer_extra_processes (chd_rank=chd_rank at entry=0x84c570 
> <chd_rank.10940>) at timer.c:557
>
> #4 0x0000000000419b9a in main_loop () at main.c:748
>
> #5  main (argc=<optimized out>, argv=<optimized out>) at main.c:1282
>
> Regards,
>
> Agalya
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20161004/1429ddb1/attachment.htm>