[OpenSIPS-Users] Fwd: opensips is restarting after call is finished
Neven Grzancic
neven at softnet.si
Fri Dec 8 14:30:39 UTC 2023
Hi Bogdan,
good hit on the apport service - thanks!
here's the trace we get from one such crash:
[New LWP 1130]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()
(gdb) bt full
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x00007f334e06aad5 in acc_aaa_cdrs (dlg=<optimized out>, msg=<optimized out>, ctx=0x7f334fcc52c8)
at /build/opensips-3.4.2/modules/acc/acc.c:965
i = <optimized out>
j = <optimized out>
ret = 6
res = -1
nr_leg_vals = <optimized out>
start_time = {tv_sec = 1702044564, tv_usec = 417047}
core_s = {s = 0x7f33705e9878 "\006", len = 101}
leg_s = {s = 0x0, len = <optimized out>}
extra_s = {s = 0x0, len = <optimized out>}
send = 0x0
offset = <optimized out>
av_type = 32563
r_stat = <optimized out>
locked = 0
duration = 1338791072
ms_duration = 0
extra = <optimized out>
error = <optimized out>
__FUNCTION__ = "acc_aaa_cdrs"
#2 0x00007f334e078ade in acc_cdr_cb (t=0x0, type=5, ps=0x7ffdb12e09e0)
at /build/opensips-3.4.2/modules/acc/acc_logic.c:986
ctx = 0x7f334fcc52c8
dlg = 0x7f334fcc54e0
__FUNCTION__ = "acc_cdr_cb"
#3 0x00007f334f8142a0 in run_any_trans_callbacks (list=list at entry=0x7f334fcc6d58, type=type at entry=128,
trans=trans at entry=0x7f334fcc6ce8, req=0x7f334fcc8770, rpl=rpl at entry=0x7f33705e7498, code=code at entry=200)
at /build/opensips-3.4.2/modules/tm/t_hooks.c:214
params = {req = 0x7f334fcc8770, rpl = 0x7f33705e7498, code = 200, param = 0x7f334fcc5270,
extra1 = 0x7ffdb12e0b40, extra2 = 0x7f334fcc6e00}
cbp = 0x7f334fcc5260
backup = 0x55c8f7a02b00 <global_avps.lto_priv>
trans_backup = 0x7f334fcc6ce8
__FUNCTION__ = "run_any_trans_callbacks"
#4 0x00007f334f818443 in run_trans_callbacks (type=type at entry=128, trans=trans at entry=0x7f334fcc6ce8, req=<optimized out>, rpl=rpl at entry=0x7f33705e7498, code=code at entry=200) at /build/opensips-3.4.2/modules/tm/t_hooks.c:233
No locals.
#5 0x00007f334f818581 in run_trans_callbacks_locked (code=200, rpl=0x7f33705e7498, req=<optimized out>, trans=0x7f334fcc6ce8, type=128) at /build/opensips-3.4.2/modules/tm/t_hooks.c:255
No locals.
#6 run_trans_callbacks_locked (type=128, trans=0x7f334fcc6ce8, req=<optimized out>, rpl=0x7f33705e7498, code=200) at /build/opensips-3.4.2/modules/tm/t_hooks.c:246
No locals.
#7 0x00007f334f82ef22 in relay_reply (t=0x7f334fcc6ce8, p_msg=<optimized out>, branch=<optimized out>, msg_status=<optimized out>, cancel_bitmap=<optimized out>) at /build/opensips-3.4.2/modules/tm/t_reply.c:1408
relay = 0
save_clone = 0
buf = 0x7f33705e9620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP 212.13.249.94;received=212.13.249.94;rport=5060;branch=z9hG4bKUtNtK5Nj897rr\r\nFrom: <sip:386980731395"...
res_len = 536
relayed_code = 200
relayed_msg = 0x7f33705e7498
bm = {to_tag_val = {s = 0x7f336f891af8 "\240\350\240\367\310U", len = 0}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f334fcc6de8
cb_s = {s = 0x7f33705e9620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP 212.13.249.94;received=212.13.249.94;rport=5060;branch=z9hG4bKUtNtK5Nj897rr\r\nFrom: <sip:386980731395"..., len = 536}
text = {s = 0x0, len = -140509096}
__FUNCTION__ = "relay_reply"
#8 0x00007f334f830835 in reply_received (p_msg=0x7f33705e7498) at /build/opensips-3.4.2/modules/tm/t_reply.c:1681
msg_status = <optimized out>
last_uac_status = 0
branch = 0
reply_status = <optimized out>
timer = 0
cancel_bitmap = 0
uac = 0x7f334fcc6ec0
t = 0x7f334fcc6ce8
backup_list = <optimized out>
has_reply_route = <optimized out>
old_route_type = <optimized out>
__FUNCTION__ = "reply_received"
#9 0x000055c8f770e0e5 in forward_reply (msg=0x7f33705e7498) at /build/opensips-3.4.2/forward.c:499
new_buf = <optimized out>
to = <optimized out>
new_len = 1
mod = 0x7f336f891af8
proto = <optimized out>
id = <optimized out>
send_sock = <optimized out>
s = <optimized out>
len = <optimized out>
error = <optimized out>
skip = <optimized out>
error0 = <optimized out>
__FUNCTION__ = "forward_reply"
#10 0x000055c8f775542d in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>, existing_context=0x0, msg_flags=<optimized out>) at /build/opensips-3.4.2/receive.c:278
msg = 0x7f33705e7498
start = {tv_sec = 139858358410128, tv_usec = 139859026441837}
rc = 3
old_route_type = <optimized out>
tmp = <optimized out>
in_buff = {s = 0x55c8f7b24320 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 213.253.120.65:5060;branch=z9hG4bK4e83.b2fe4a51.0\r\nVia: SIP/2.0/UDP 212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP 212.13.249.94;received=212.13.24"..., len = 612}
ctx = 0x7f33705e8bc8
__FUNCTION__ = "receive_msg"
#11 0x000055c8f78addd7 in udp_read_req (si=<optimized out>, bytes_read=<optimized out>) at net/proto_udp/proto_udp.c:186
ri = {src_ip = {af = 2, len = 4, u = {addrl = {94317692413908, 94321635752912}, addr32 = {210593748, 21960, 4153932752, 21960}, addr16 = {26580, 3213, 21960, 0, 64464, 63383, 21960, 0}, addr = "\324g\215\f\310U\000\000\320\373\227\367\310U\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1098448341, 0}, addr32 = {1098448341, 0, 0, 0}, addr16 = {64981, 16760, 0, 0, 0, 0, 0, 0}, addr = "\325\375xA", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto = 1, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\324g\215\f\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 210593748}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 210593748, sin6_addr = {__in6_u = {__u6_addr8 = "\000\000\000\000\000\000\000\000 +\240\367\310U\000", __u6_addr16 = {0, 0, 0, 0, 11040, 63392, 21960, 0}, __u6_addr32 = {0, 0, 4154469152, 21960}}}, sin6_scope_id = 2973633109}}, bind_address = 0x7f336f8b52d8}
len = <optimized out>
tmp = <optimized out>
fromlen = 16
p = <optimized out>
msg = {s = 0x55c8f7b24320 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 213.253.120.65:5060;branch=z9hG4bK4e83.b2fe4a51.0\r\nVia: SIP/2.0/UDP 212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP 212.13.249.94;received=212.13.24"..., len = 612}
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 213.253.120.65:5060;branch=z9hG4bK4e83.b2fe4a51.0\r\nVia: SIP/2.0/UDP 212.13.249.92:5060;branch=z9hG4bK4e83.fd8885b6.0\r\nVia: SIP/2.0/UDP 212.13.249.94;received=212.13.24"...
__FUNCTION__ = "udp_read_req"
#12 0x000055c8f78c75da in handle_io.constprop.1 (fm=0x7f336f8e7788, event_type=event_type at entry=1, idx=<optimized out>) at net/net_udp.c:295
n = 0
read = 0
__FUNCTION__ = <optimized out>
#13 0x000055c8f78d4c40 in io_wait_loop_epoll.constprop.1.isra.0 (repeat=<optimized out>, t=<optimized out>, h=<optimized out>) at net/../io_wait_loop.h:305
ret = <optimized out>
n = <optimized out>
r = 0
i = <optimized out>
e = <optimized out>
ep_event = {events = 2, data = {ptr = 0xf7a0320000000000, fd = 0, u32 = 0, u64 = 17843316699223293952}}
fd = <optimized out>
curr_time = 78
error = <optimized out>
__FUNCTION__ = <optimized out>
#14 0x000055c8f78a42b5 in udp_start_processes (chd_rank=<optimized out>, startup_done=<optimized out>) at net/net_udp.c:528
si = <optimized out>
p_id = <optimized out>
i = <optimized out>
p = <optimized out>
ifp_udp_rcv = {proc_desc = 0x55c8f796c046 "UDP receiver", flags = 4, type = TYPE_UDP}
error = <optimized out>
__FUNCTION__ = "udp_start_processes"
#15 0x000055c8f76d575d in main_loop () at /build/opensips-3.4.2/main.c:239
startup_done = 0x0
last_check = 0
rc = <optimized out>
chd_rank = <optimized out>
startup_done = <optimized out>
last_check = <optimized out>
rc = <optimized out>
profiling_handler = <optimized out>
__FUNCTION__ = <optimized out>
error = <optimized out>
#16 main (argc=<optimized out>, argv=<optimized out>) at /build/opensips-3.4.2/main.c:968
c = <optimized out>
r = <optimized out>
tmp = 0x7ffdb12e2e53 ""
tmp_len = <optimized out>
port = <optimized out>
proto = 0
protos_no = <optimized out>
options = 0x55c8f78ea290 "f:cCm:M:b:l:n:N:rRvdDFEVhw:t:u:g:p:P:G:W:o:a:k:s:"
ret = <optimized out>
seed = 3254223836
rfd = <optimized out>
procs_no = <optimized out>
__FUNCTION__ = "main"
On 08.12.23 15:16, Bogdan-Andrei Iancu wrote:
> Simon, I do not need the corefile, usually it is huge - as per web instructions (see the prev link), extract the backtrace (using gdb) and share it with me.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
> https://www.opensips-solutions.com
> https://www.siphub.com
>
> On 08.12.2023 16:13, Simon Gajski wrote:
>> Hi Bogdan
>>
>> you were right
>> we have the core file now
>>
>> Please let me know where to send it (gzipped is approx 2MB)
>>
>>
>> BR
>> Simon
>>
>> Bogdan-Andrei Iancu je 08.12.2023 ob 14:17 napisal:
>>> Hi Simon,
>>>
>>> Maybe you have some apport service running on the machine, moving the core file.
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>>
>>> OpenSIPS Founder and Developer
>>> https://www.opensips-solutions.com
>>> https://www.siphub.com
>>>
>>> On 08.12.2023 13:35, Neven Grzancic wrote:
>>>> Dear Bogdan,
>>>>
>>>> we tried following https://opensips.org/Documentation/TroubleShooting-Crash and trying different things, and even though the syslog shows that the core file was dumped:
>>>>
>>>> Dec 8 12:25:13 sbc2 /usr/sbin/opensips[4363]: CRITICAL:core:sig_usr: segfault in process pid: 4363, id: 7
>>>> Dec 8 12:25:13 sbc2 kernel: [ 3394.269733] opensips[4363]: segfault at 0 ip 0000000000000000 sp 00007ffdc458f6a8 error 14 in opensips[55816fbd0000+1f000]
>>>> Dec 8 12:25:18 sbc2 mariadbd[922]: 2023-12-08 12:25:18 400 [Warning] Aborted connection 400 to db: 'opensips' user: 'opensips' host: 'localhost' (Got an error reading communication packets)
>>>> Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4356]: INFO:core:handle_sigs: child process 4363 exited by a signal 11
>>>> Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4356]: INFO:core:handle_sigs: core was generated
>>>> Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4356]: INFO:core:handle_sigs: terminating due to SIGCHLD
>>>> Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4359]: INFO:core:sig_usr: signal 15 received
>>>> Dec 8 12:25:18 sbc2 /usr/sbin/opensips[4360]: INFO:core:sig_usr: signal 15 received
>>>>
>>>> the core does not get generated, or at least not in the directory provided via options at runtime (-w /tmp). The issue is always repeatable (the daemon crashes after every finished call), if there are more active calls at the moment of the crash, whose calls remain active and proceed normally until termination (which again causes a crash). We also tried enabling debug mode, but in that case the process does not start successfully and runs into a timeout. In any case, any additional help or suggestion on how to successfully generate the core dump would be much appreciated, as we seem to have ran into a wall here...
>>>>
>>>> On 08.12.23 12:26, Simon Gajski wrote:
>>>>>
>>>>>
>>>>>
>>>>> -------- Posredovano sporočilo --------
>>>>> Zadeva: Re: [OpenSIPS-Users] opensips is restarting after call is finished
>>>>> Datum: Thu, 7 Dec 2023 17:52:37 +0200
>>>>> Od: Bogdan-Andrei Iancu <bogdan at opensips.org>
>>>>> Za: Simon Gajski <simon at softnet.si>, OpenSIPS users mailling list <users at lists.opensips.org>
>>>>>
>>>>>
>>>>>
>>>>> Hi Simon,
>>>>>
>>>>> Well, the log is very self explanatory:
>>>>> Dec 7 15:59:47 sbc2 /usr/sbin/opensips[28488]: INFO:core:handle_sigs: child process 28495 exited by a signal 11
>>>>> Dec 7 15:59:47 sbc2 /usr/sbin/opensips[28488]: INFO:core:handle_sigs: core was generated
>>>>> Dec 7 15:59:47 sbc2 /usr/sbin/opensips[28488]: INFO:core:handle_sigs: terminating due to SIGCHLD
>>>>>
>>>>> Or shortly, you opensips just crashed :). So see https://opensips.org/Documentation/TroubleShooting-Crash for how to report further.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Bogdan-Andrei Iancu
>>>>>
>>>>> OpenSIPS Founder and Developer
>>>>> https://www.opensips-solutions.com
>>>>> https://www.siphub.com
>>>>>
>>>>> On 07.12.2023 17:17, Simon Gajski via Users wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> we are running opensips 3.4.2
>>>>>> on Ubuntu 22.04.3 LTS
>>>>>> and also use RTPengine Version: 10.5.0.0+0~mr10.5.0.0 git-master-74075f63
>>>>>>
>>>>>> Opensips acts as SBC with RTP engine enabled.
>>>>>>
>>>>>> Calls are working fine, however after each call is finished, bellow action happens.
>>>>>>
>>>>>> We had same problem with opensips 3.2 and Ubuntu 20. So we did upgrade to latest stable release, and it is the same.
>>>>>>
>>>>>> Is this opensips script configuration issue or a bug? And how could I fix it?
>>>>>>
>>>>>> [....]
>>>>>>
>>>>>> Thanks for all tips.
>>>>>>
>>>>>> BR
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at lists.opensips.org
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>
>>>
>
--
Best regards / Lep pozdrav
Neven Grzancic, dipl.inž.el.(UN)
techsupport - systems administration & networking
-----------------------------------------------------------------------
Softnet d.o.o. tel: +386 1 8100 100 | the quieter you
Borovec 2 direct: +386 1 8100 107 | become, the more you
1236 Trzin fax: +386 1 8100 101 | are able to hear
-----------------------------------------------------------------------
More information about the Users
mailing list