[OpenSIPS-Users] sipcapture segfaults when capturing on raw interface in all 2.x versions of opensips
Michael Ulitskiy
mulitskiy at acedsl.com
Mon Jun 18 10:58:25 EDT 2018
Bogdan,
These are not hep packets. i'm using raw capture.
Cisco switch is configuted with RSPAN and delivers sip traffic to capture server on a particular vlan (vlan 990 in this case).
The config is simplest possible:
memlog=0
debug_mode=1
listen=udp:127.0.0.1:5060
listen=hep_udp:127.0.0.1:9063
children=5
dns=no
mpath="/usr/local/opensips/lib/opensips/modules" #path to modules
db_default_url="postgres://<user>:<password>@localhost:5432/sipcapture"
loadmodule "proto_hep.so"
loadmodule "proto_udp.so"
loadmodule "db_postgres.so"
modparam("db_postgres", "exec_query_threshold", 100000)
loadmodule "sipmsgops.so"
loadmodule "sipcapture.so"
modparam("sipcapture", "db_url", "postgres://<user>:<password>@localhost:5432/sipcapture")
#modparam("sipcapture", "table_name", "sip_capture")
modparam("sipcapture", "hep_capture_on", 1)
modparam("sipcapture", "raw_moni_capture_on", 1)
modparam("sipcapture", "raw_interface", "bond0.990")
modparam("sipcapture", "promiscious_on", 1)
modparam("sipcapture", "raw_socket_listen", "10.0.0.1:5060-5160")
modparam("sipcapture", "raw_moni_bpf_on", 1)
modparam("sipcapture", "capture_on", 1)
route {
if (!is_method("OPTIONS|REGISTER|NOTIFY|SUBSCRIBE")) {
sip_capture("sip_capture");
}
}
onreply_route {
if (!is_method("OPTIONS|REGISTER|NOTIFY|SUBSCRIBE")) {
sip_capture("sip_capture");
}
}
I'm not sure hot to isolate the particular packet that causes the crash due to high volume of sip traffic in RSPAN vlan.
If you want I could collect pcap over a few seconds when opensips crashes, but it'll definitely have multiple packets and I'm not sure how to isolate the one that caused the crash. Anyway it's easily reproducible and crashes almost instantaneously, so I guess if you start raw capture on a vlan interface with some sip traffic you'll see it right away.
Thanks,
Michael
On Thursday, June 14, 2018 12:34:02 PM you wrote:
> Hi Michael,
>
> Thanks for the information.
>
> The printing msg shows that the src and dst IPs are correct.
>
> Maybe a better approach will be to for me to try to reproduce this crash
> - is it possible for you to reduce to a cfg file and a HEP packet that
> produces the crash, so I can troubleshoot it ?
>
> Thanks and regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
> OpenSIPS Summit 2018
> http://www.opensips.org/events/Summit-2018Amsterdam
>
> On 06/05/2018 05:59 PM, Michael Ulitskiy wrote:
> > Hello,
> >
> > Here's msg->rcv:
> >
> > (gdb) frame 0
> > #0 0xb50bfb61 in w_sip_capture (msg=0xb738a810, table_name=<optimized out>, resume_f=0x0, resume_param=0x0)
> > at sipcapture.c:3433
> > 3433 sco.msg.s = h->u.hepv12.payload;
> > (gdb) p msg->rcv
> > $2 = {src_ip = {af = 2, len = 4, u = {addrl = {2389972783, 3040358736, 3037670688, 3037552928}, addr32 = {
> > 2389972783, 3040358736, 3037670688, 3037552928}, addr16 = {5935, 36468, 12624, 46392, 11552, 46351,
> > 24864, 46349}, addr = "/\027t\216P18µ -\017µ a\rµ"}}, dst_ip = {af = 2, len = 4, u = {addrl = {
> > 1246982722, 3037552928, 31, 3077604236}, addr32 = {1246982722, 3037552928, 31, 3077604236}, addr16 = {
> > 29250, 19027, 24864, 46349, 31, 0, 33676, 46960}, addr = "BrSJ a\rµ\037\000\000\000\214\203p·"}},
> > src_port = 4940, dst_port = 5060, proto = 1, proto_reserved1 = -1257415776, proto_reserved2 = -1257467391,
> > src_su = {s = {sa_family = 2, sa_data = "\023L/\027t\216\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
> > sin_port = 19475, sin_addr = {s_addr = 2389972783}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
> > sin6_family = 2, sin6_port = 19475, sin6_flowinfo = 2389972783, sin6_addr = {__in6_u = {
> > __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0,
> > 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x807b872 <set_proc_attrs+82>}
> >
> > If I set "hep_capture_on" set to 1 then it doesn't crash, but it doesn't store anything into db either.
> > Here's what it says:
> >
> > Jun 5 10:54:57 [3231] DBG:core:receive_msg: After parse_msg...
> > Jun 5 10:54:57 [3231] DBG:core:receive_msg: preparing to run routing scripts...
> > Jun 5 10:54:57 [3231] WARNING:sipcapture:w_sip_capture: not a hep message!
> > Jun 5 10:54:57 [3231] DBG:core:destroy_avp_list: destroying list (nil)
> > Jun 5 10:54:57 [3231] DBG:core:receive_msg: cleaning up
> >
> > Am I doing something wrong?
> > Thanks,
> >
> > Michael
> >
> > On Tuesday, June 05, 2018 01:39:03 PM Bogdan-Andrei Iancu wrote:
> >> Hi Michael,
> >>
> >> This crash seems to be triggered mainly by the "hep_capture_on" set to 0 .
> >>
> >> Just to check couple of things, using gdb, in frame 0, could you print
> >> msg->rcv ? it is interesting why those IPs are not valid (see the
> >> critical logs before the crash). Afterwards, the crash itself happened
> >> because of 'h' being null (hep_capture_on set to 0 ).
> >>
> >> Thanks,
> >>
> >> Bogdan-Andrei Iancu
> >>
> >> OpenSIPS Founder and Developer
> >> http://www.opensips-solutions.com
> >> OpenSIPS Summit 2018
> >> http://www.opensips.org/events/Summit-2018Amsterdam
> >>
> >> On 05/30/2018 10:55 PM, Michael Ulitskiy wrote:
> >>> Hello,
> >>>
> >>> I apologize for so delayed response.
> >>>
> >>> Here's opensips revision:
> >>>
> >>> root at clog1:/usr/local/opensips/etc/opensips# /usr/local/opensips/sbin/opensips -V
> >>> version: opensips 2.2.6 (i386/linux)
> >>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
> >>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
> >>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> >>> git revision: 24507b1ce
> >>> main.c compiled on 17:18:03 May 2 2018 with gcc 4.7.1
> >>>
> >>> debug log:
> >>>
> >>> May 30 15:29:28 [436] CRITICAL:sipcapture:ip_addr2a: unknown address family 841889588
> >>> May 30 15:29:28 [436] CRITICAL:sipcapture:ip_addr2a: unknown address family 825112118
> >>> May 30 15:29:28 [436] DBG:sipcapture:w_sip_capture: src_ip: [47.2]
> >>> May 30 15:29:28 [436] DBG:sipcapture:w_sip_capture: dst_ip: [66.1]
> >>> May 30 15:29:28 [436] DBG:sipcapture:w_sip_capture: dst_port: [5060]
> >>> May 30 15:29:28 [436] DBG:sipcapture:w_sip_capture: src_port: [4940]
> >>> May 30 15:29:28 [435] DBG:core:handle_sigs: status = 139
> >>> May 30 15:29:28 [435] INFO:core:handle_sigs: child process 436 exited by a signal 11
> >>> May 30 15:29:28 [435] INFO:core:handle_sigs: core was generated
> >>> May 30 15:29:28 [435] INFO:core:handle_sigs: terminating due to SIGCHLD
> >>> May 30 15:29:28 [437] INFO:core:sig_usr: signal 15 received
> >>>
> >>> backtrace:
> >>> (gdb) bt
> >>> #0 0xb50bfb61 in w_sip_capture (msg=0xb738a810, table_name=<optimized out>, resume_f=0x0, resume_param=0x0) at sipcapture.c:3433
> >>> #1 0x0806f227 in do_action (a=a at entry=0xb738968c, msg=msg at entry=0xb738a810) at action.c:1844
> >>> #2 0x08075c4f in run_action_list (a=0xb738968c, msg=0xb738a810) at action.c:172
> >>> #3 0x08073063 in do_action (a=a at entry=0xb7389700, msg=msg at entry=0xb738a810) at action.c:1108
> >>> #4 0x0807563d in run_action_list (msg=0xb738a810, a=<optimized out>) at action.c:172
> >>> #5 run_actions (a=a at entry=0xb7389700, msg=msg at entry=0xb738a810) at action.c:137
> >>> #6 0x0807622c in run_actions (msg=0xb738a810, a=0xb7389700) at action.c:130
> >>> #7 run_top_route (a=0xb7389700, msg=msg at entry=0xb738a810) at action.c:204
> >>> #8 0x0807e710 in receive_msg (
> >>> buf=buf at entry=0xb50de70a <buf.11361+42> "INVITE sip:229 at csp1.acepbx.com:5060;user=phone SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.22.172:5060;branch=z9hG4bKf34dc6c0CA495609\r\nFrom: \"Angel\" <sip:missgrp358901 at csp1.acepbx.com>;tag=FD8AA3F2-CD873993\r\nTo"..., len=<optimized out>, rcv_info=rcv_info at entry=0xbfc03f50, existing_context=existing_context at entry=0x0) at receive.c:208
> >>> #9 0xb50cf9c6 in raw_capture_rcv_loop (rsock=8, port1=5060, port2=port2 at entry=5160, ipip=0) at sipcapture.c:5247
> >>> #10 0xb50cfaf0 in raw_socket_process (rank=0) at sipcapture.c:2185
> >>> #11 0x080d0c59 in start_module_procs () at sr_module.c:763
> >>> #12 0x0805ee5c in main_loop () at main.c:654
> >>> #13 main (argc=3, argv=0xbfc04174) at main.c:1265
> >>>
> >>> full backtrace can be found here: https://www.aceinnovative.com/t/backtrace.txt
> >>>
> >>> Thank you,
> >>>
> >>> Michael
> >>>
> >>> On Thursday, May 10, 2018 08:24:51 PM Bogdan-Andrei Iancu wrote:
> >>>> Hi Michael,
> >>>>
> >>>> What is the exact OpenSIPs revision you have (do 'opensips -V') ?
> >>>>
> >>>> Also, could you extract the backtrace from the corefile ?
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Bogdan-Andrei Iancu
> >>>>
> >>>> OpenSIPS Founder and Developer
> >>>> http://www.opensips-solutions.com
> >>>> OpenSIPS Summit 2018
> >>>> http://www.opensips.org/events/Summit-2018Amsterdam
> >>>>
> >>>> On 05/08/2018 01:22 AM, Michael Ulitskiy wrote:
> >>>>> Hello,
> >>>>>
> >>>>> I'm trying to use opensips with sipcature module to collect sip messaging on a
> >>>>> mirror port and feed it to homer database. Unfortunately it segfaults. I've
> >>>>> tried to use all latest 2.x versions with the same result. 1.11.11 seems to
> >>>>> work fine, but is missing necessary features. Since homer provides sample
> >>>>> configs for opensips 2.2.x all below is for opensips-2.2.6.
> >>>>>
> >>>>> Here's excerpt from the log right before the crash:
> >>>>>
> >>>>> May 7 18:05:43 [10752] DBG:core:_parse_to: display={},
> >>>>> ruri={sip:19179620875 at 66.114.x.y}
> >>>>> May 7 18:05:43 [10752] CRITICAL:sipcapture:ip_addr2a: unknown address family
> >>>>> 841889590
> >>>>> May 7 18:05:43 [10752] CRITICAL:sipcapture:ip_addr2a: unknown address family
> >>>>> 825112118
> >>>>> May 7 18:05:43 [10752] DBG:sipcapture:w_sip_capture: src_ip: [67.2]
> >>>>> May 7 18:05:43 [10752] DBG:sipcapture:w_sip_capture: dst_ip: [66.1]
> >>>>> May 7 18:05:43 [10752] DBG:sipcapture:w_sip_capture: dst_port: [5060]
> >>>>> May 7 18:05:43 [10752] DBG:sipcapture:w_sip_capture: src_port: [5060]
> >>>>> May 7 18:05:43 [10750] DBG:core:handle_sigs: status = 139
> >>>>> May 7 18:05:43 [10750] INFO:core:handle_sigs: child process 10752 exited by a
> >>>>> signal 11
> >>>>> May 7 18:05:43 [10750] INFO:core:handle_sigs: core was generated
> >>>>> May 7 18:05:43 [10750] INFO:core:handle_sigs: terminating due to SIGCHLD
> >>>>>
> >>>>> As you can see opensips was unable to parse address family and source and
> >>>>> destination ip. The capture is done on vlan interface - bond0.9. Can it be the
> >>>>> reason?
> >>>>>
> >>>>> Here's is config:
> >>>>>
> >>>>> memlog=0
> >>>>> debug_mode=1
> >>>>> listen=udp:127.0.0.1:5060
> >>>>> children=5
> >>>>> dns=no
> >>>>> mpath="/usr/local/opensips/lib/opensips/modules" #path to modules
> >>>>> db_default_url="postgres://user:password@localhost:5432/sipcapture"
> >>>>>
> >>>>> loadmodule "db_postgres.so"
> >>>>> modparam("db_postgres", "exec_query_threshold", 100000)
> >>>>> loadmodule "sipmsgops.so"
> >>>>> loadmodule "sipcapture.so"
> >>>>> modparam("sipcapture", "db_url",
> >>>>> "postgres://user:password@localhost:5432/sipcapture")
> >>>>> modparam("sipcapture", "table_name", "sip_capture")
> >>>>> modparam("sipcapture", "hep_capture_on", 0)
> >>>>> modparam("sipcapture", "raw_moni_capture_on", 1)
> >>>>> modparam("sipcapture", "raw_interface", "bond0.990")
> >>>>> modparam("sipcapture", "promiscious_on", 1)
> >>>>> modparam("sipcapture", "raw_socket_listen", "10.0.0.1:5060-5160")
> >>>>> modparam("sipcapture", "raw_moni_bpf_on", 1)
> >>>>> modparam("sipcapture", "capture_on", 1)
> >>>>>
> >>>>> route {
> >>>>>
> >>>>> if (!is_method("OPTIONS|REGISTER|NOTIFY|SUBSCRIBE")) {
> >>>>> sip_capture();
> >>>>> }
> >>>>>
> >>>>> }
> >>>>>
> >>>>> onreply_route {
> >>>>> if (!is_method("OPTIONS|REGISTER|NOTIFY|SUBSCRIBE")) {
> >>>>> sip_capture();
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> I can provide core if somebody wants to look at it. Just not sure if this list
> >>>>> would allow such a large attachment.
> >>>>>
> >>>>> Please let me know if you have any idea how to fix it.
> >>>>> Thanks,
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Users mailing list
> >>>>> Users at lists.opensips.org
> >>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
More information about the Users
mailing list