[OpenSIPS-Users] opensips crash when doing load test

Thu Oct 6 18:25:53 CEST 2016

Hi Bogdan,

When I try to apply patch, I see all the code are already available in 2.2. So I did not add any patch.
But did a code addition for 'REST_PUT' request in rest_client.c, rest_methods.c, rest_methohs.h
Tried to perform load test. Observed 2 issues.

1)      Tried to load 10,000 calls - But route[resume_http] is called only for 9985 calls.

Every time approximately 10-20 calls, route[resume_http]  is not called. But if I see the tcpdump, I am seeing 10,000 HTTP request and 10,000 HTTP 200 OK responses.

When printing the response in resume_http for every call-id, 10-20 calls response is not printed - which means resume is not called for these calls.

Am not filtering any response code.

route[resume_http] {

        xlog("L_INFO","Resp $rc in HTTP PUT!\n");

        xlog("L_INFO","route[relay] The content received from SM for $rm: [callId=$ci] : $var(body) in HTTP PUT\n");

        .....................

}
                For first few calls, $rc is 1(ASYNC_DONE), for others $rc is -5(ASYNC_SYNC)

2)      Tried to load more than 10,000 calls. When it reaches 25,000 calls after that some point opensips crashes.

I have two core dump generated as of now.
Coredump1:
--------------
(gdb) bt
#0  0x00007f8defacc840 in Curl_num_addresses () from /lib64/libcurl.so.4
#1  0x00007f8defaf09ee in Curl_connecthost () from /lib64/libcurl.so.4
#2  0x00007f8defae34ff in Curl_setup_conn () from /lib64/libcurl.so.4
#3  0x00007f8defae370c in Curl_connect () from /lib64/libcurl.so.4
#4  0x00007f8defaf32c0 in multi_runsingle () from /lib64/libcurl.so.4
#5  0x00007f8defaf4121 in curl_multi_perform () from /lib64/libcurl.so.4
#6  0x00007f8defd2b65b in start_async_http_req (msg=msg at entry=0x7f8df3740fc0, method=method at entry=REST_CLIENT_PUT,
    url=0x7f8df36ceb88 "http://test.comcast.net/RTCGSessionManager/rest/tel/session/createroom?", req_body=<optimized out>,
    req_ctype=<optimized out>, out_handle=out_handle at entry=0x7f8df3743300, body=body at entry=0x7f8df3743308, ctype=0x7f8df3743318)
    at rest_methods.c:265
#7  0x00007f8defd31b56 in w_async_rest_put (msg=0x7f8df3740fc0, resume_f=0x7ffd4afd8c90, resume_param=0x7ffd4afd8c88,
    gp_url=<optimized out>, gp_body=<optimized out>, gp_ctype=<optimized out>, body_pv=0x7f8df36f85a8 "N",
    ctype_pv=0x7f8df36f8700 "N", code_pv=0x7f8df36f88c0 "N") at rest_client.c:544
#8  0x00007f8df1031841 in t_handle_async (msg=0x7f8df3740fc0, a=0x7f8df36cf0c8, resume_route=2) at async.c:240
#9  0x0000000000424056 in do_action (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at action.c:1863
#10 0x000000000041c330 in run_action_list (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at action.c:172
#11 0x0000000000420637 in do_action (a=0x7f8df36cf3f8, msg=0x7f8df3740fc0) at action.c:1108
#12 0x000000000041c330 in run_action_list (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at action.c:172
#13 0x000000000041c1fd in run_actions (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at action.c:137
#14 0x000000000041ed55 in do_action (a=0x7f8df36ca080, msg=0x7f8df3740fc0) at action.c:745
#15 0x000000000041c330 in run_action_list (a=0x7f8df36c9e78, msg=0x7f8df3740fc0) at action.c:172
#16 0x0000000000420637 in do_action (a=0x7f8df36ca1a8, msg=0x7f8df3740fc0) at action.c:1108
#17 0x000000000041c330 in run_action_list (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:172
#18 0x000000000041c1fd in run_actions (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:137
#19 0x000000000041c3fd in run_top_route (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:204
#20 0x000000000042bb64 in receive_msg (
    buf=0x7eb340 <buf.8031> "INVITE sip:++12001000004 at 10.0.0.1:5060 SIP/2.0\r\nTo: sut <sip:+19086774567 at 10.0.0.0.1:5060;user=phone>\r\nFrom: sipp <sip:sipp at 10.0.0.2:5060;user=phone>;tag=26939SIPpTag0014404\r\nCall-ID: 14"..., len=837,
    rcv_info=0x7ffd4afdae00, existing_context=0x0) at receive.c:208
#21 0x0000000000521838 in udp_read_req (si=0x7f8df36bda50, bytes_read=0x7ffd4afdaec8) at net/proto_udp/proto_udp.c:192
#22 0x000000000050d05b in handle_io (fm=0x7f8df371e128, idx=0, event_type=1) at net/net_udp.c:259
#23 0x000000000050ba2b in io_wait_loop_epoll (h=0x80be40 <_worker_io>, t=1, repeat=0) at net/../io_wait_loop.h:221
#24 0x000000000050d3db in udp_rcv_loop (si=0x7f8df36bda50) at net/net_udp.c:311
#25 0x000000000050d973 in udp_start_processes (chd_rank=0x7d8128 <chd_rank.10725>, startup_done=0x0) at net/net_udp.c:375
#26 0x0000000000495241 in main_loop () at main.c:671
#27 0x0000000000497cec in main (argc=3, argv=0x7ffd4afdb1c8) at main.c:1271

Coredump2:
--------------
(gdb) bt
#0  0x00000000004affeb in qm_detach_free (qm=0x7f5aa3ff4010, frag=0x7f5aa40c50f0) at mem/q_malloc.c:281
#1  0x00000000004b1233 in qm_free (qm=0x7f5aa3ff4010, p=0x7f5aa40c50b0, file=0x7f5aa069fbf5 "rest_client.c",
    func=0x7f5aa06a01fd <__FUNCTION__.8581> "osips_free", line=205) at mem/q_malloc.c:494
#2  0x00007f5aa04774f6 in curl_thread_create_thunk () from /lib64/libcurl.so.4
#3  0x00007f5a9edb9dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f5aa42eb28d in clone () from /lib64/libc.so.6

Regards,
Agalya

From: Bogdan-Andrei Iancu [mailto:bogdan at opensips.org]
Sent: Wednesday, October 05, 2016 3:58 PM
To: Ramachandran, Agalya (Contractor) <Agalya_Ramachandran at comcast.com>; OpenSIPS users mailling list <users at lists.opensips.org>
Subject: Re: opensips crash when doing load test

That's a good news :)

Try to add one patch at a time (incrementally) and let's see when it starts to crash again.

Thanks and regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

http://www.opensips-solutions.com
On 05.10.2016 22:55, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,

With 2.2 version, am not seeing any crash. All looks good.

Regards,
Agalya

From: users-bounces at lists.opensips.org<mailto:users-bounces at lists.opensips.org> [mailto:users-bounces at lists.opensips.org] On Behalf Of Ramachandran, Agalya (Contractor)
Sent: Wednesday, October 05, 2016 10:50 AM
To: Bogdan-Andrei Iancu <bogdan at opensips.org><mailto:bogdan at opensips.org>; OpenSIPS users mailling list <users at lists.opensips.org><mailto:users at lists.opensips.org>
Subject: Re: [OpenSIPS-Users] opensips crash when doing load test

Sure. I will test and share the results by end of day or tomorrow.

Regards,
Agalya

From: Bogdan-Andrei Iancu [mailto:bogdan at opensips.org]
Sent: Wednesday, October 05, 2016 10:47 AM
To: Ramachandran, Agalya (Contractor) <Agalya_Ramachandran at comcast.com<mailto:Agalya_Ramachandran at comcast.com>>; OpenSIPS users mailling list <users at lists.opensips.org<mailto:users at lists.opensips.org>>
Subject: Re: opensips crash when doing load test

Yes, that is ok (without the REST) - just to be sure which patch introduces the bug.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

http://www.opensips-solutions.com
On 05.10.2016 17:45, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,

I have observed one more thing. In the same source code, when opensips acts as pure proxy no HTTP call is made opensips is not crashing.
I suspect the crashes may be due to the new async fix provided.

I can do load test in 2.2, but it  will be just pure-proxy load test and not with the REST queries made.
If that is OK, I can test and share the results.

Regards,
Agalya

From: Bogdan-Andrei Iancu [mailto:bogdan at opensips.org]
Sent: Wednesday, October 05, 2016 9:46 AM
To: Ramachandran, Agalya (Contractor) <Agalya_Ramachandran at comcast.com><mailto:Agalya_Ramachandran at comcast.com>; OpenSIPS users mailling list <users at lists.opensips.org><mailto:users at lists.opensips.org>
Subject: Re: opensips crash when doing load test

Hi Agalya,

That is really weird, as that field (my_T) is initialized (with the address of the transaction) when the transaction is created (for all the possible branches) and never reset or set again.

To avoid any regression due the patches you applied, could you please run your load test with the stock 2.2 code from git ? And see if the crash occurs again.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

http://www.opensips-solutions.com
On 04.10.2016 19:51, Ramachandran, Agalya (Contractor) wrote:
Hi Bogdan,

Please find the values. Any common folder you have access for the outside world to put files?
If yes, share me the details. I can upload the core file there.

(gdb) print r_buf
$1 = (struct retr_buf *) 0x7fded2484098
(gdb) print *r_buf
$2 = {activ_type = 0, buffer = {
    s = 0x7fded0ca79f0 "INVITE sip:++12001000004 at 10.0.0.1:5060<mailto:sip:++12001000004 at 10.0.0.1:5060> SIP/2.0\r\nRecord-Route: <sip:96.119.240.35;lr>\r\nTo: sut <sip:+12452123456 at 10.0.0.1:5060;user=phone><mailto:sip:+12452123456 at 10.0.0.1:5060;user=phone>\r\nFrom: sipp <sip:sipp at 10.0.0.2:5060;user=phone<mailto:sip:sipp at 10.0.0.2:5060;user=phone>"..., len = 968}, dst = {proto = 1,
    proto_reserved1 = 0, to = {s = {sa_family = 2, sa_data = "\023\304`w\347\375\000\000\000\000\000\000\000"}, sin = {
        sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 4259805024}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
        sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 4259805024, sin6_addr = {__in6_u = {
            __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
        sin6_scope_id = 0}}, send_sock = 0x7fded841dcb0}, retr_timer = {next_tl = 0x7fded1af3518, prev_tl = 0x7fded15c8ae0,
    ld_tl = 0x0, time_out = 169500000, timer_list = 0x7fded0acbc48, deleted = 1, set = 0}, fr_timer = {next_tl = 0x0, prev_tl = 0x0,
    ld_tl = 0x0, time_out = 165, timer_list = 0x7fded8005b00 <detached_timer>, deleted = 0, set = 0}, retr_list = RT_T2, my_T = 0x0,
  branch = 0}

Regards,
Agalya

From: Bogdan-Andrei Iancu [mailto:bogdan at opensips.org]
Sent: Tuesday, October 04, 2016 11:52 AM
To: Ramachandran, Agalya (Contractor) <Agalya_Ramachandran at comcast.com><mailto:Agalya_Ramachandran at comcast.com>; OpenSIPS users mailling list <users at lists.opensips.org><mailto:users at lists.opensips.org>
Subject: Re: opensips crash when doing load test

Could you also print r_buf and *r_buf  ?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20161006/6fbc1452/attachment-0001.htm>