[OpenSIPS-Devel] OpenSIPS Crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Thu Jun 27 06:40:41 EDT 2019
Hi Ben,
OK, thanks for the confirmation. Let me work out an additional fix for
this;)
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 06/26/2019 08:55 PM, Ben Newlin wrote:
>
> Bogdan,
>
> I was able to reproduce this crash on the newer 2.4 commit.
>
> Backtrace: https://pastebin.com/RaMXTZA4 <https://pastebin.com/RaMXTZA4>
>
> # opensips -V
>
> version: opensips 2.4.6 (x86_64/linux)
>
> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
> QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
>
> poll method support: poll, epoll, sigio_rt, select.
>
> git revision: 714afb97f
>
> main.c compiled on 17:22:40 Jun 26 2019 with gcc 7
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> *Date: *Wednesday, June 26, 2019 at 11:34 AM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>, OpenSIPS devel mailling
> list <devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> As per ticket, the issue you reported was already fixed, so going for
> the latest version in 2.4 branch should be fine with you.
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> https://www.opensips-solutions.com
> OpenSIPS Summit 2019
> https://www.opensips.org/events/Summit-2019Amsterdam/
>
> On 06/26/2019 06:21 PM, Ben Newlin wrote:
>
> Bogdan,
>
> I will test both with the latest and with that specific commit,
> but I cannot use the latest in production due to the issue I
> reported in 2.4.6. [1] Perhaps that specific commit is before
> whatever change is causing that issue.
>
> [1] https://github.com/OpenSIPS/opensips/issues/1736
> <https://github.com/OpenSIPS/opensips/issues/1736>
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Wednesday, June 26, 2019 at 10:56 AM
> *To: *OpenSIPS devel mailling list <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>, Ben Newlin
> <Ben.Newlin at genesys.com> <mailto:Ben.Newlin at genesys.com>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> While re-checking some of some data related to your report, I
> noticed that your opensips version (revision d025b4f61) is a bit
> old and it is missing some fixes that might be relevant for you
> (mainly 3a994ceeafc6830a1c3b3f2699a7dc1ea915c83a).
>
> Could you grab the latest 2.4 from git and give it a try please.
>
> Regards,
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> https://www.opensips-solutions.com
>
> OpenSIPS Summit 2019
>
> https://www.opensips.org/events/Summit-2019Amsterdam/
>
> On 06/26/2019 02:37 PM, Bogdan-Andrei Iancu wrote:
>
> Thank you Ben,
>
> This backtrace confirms the initial suspicion - I'm trying to
> find out a way to get it fixed in a more generic way; what you
> experience is just one face of the problem (with many faces
> :P) and there are other related reports.
>
> Regards,
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> https://www.opensips-solutions.com
>
> OpenSIPS Summit 2019
>
> https://www.opensips.org/events/Summit-2019Amsterdam/
>
> On 06/26/2019 02:34 AM, Ben Newlin wrote:
>
> Bogdan,
>
> I have now been able to reproduce this crash reliably, I
> believe. Please see the backtrace [1]. Is there some other
> information you would like me to collect when this occurs?
>
> [1] https://pastebin.com/n0Ph8XH5
> <https://pastebin.com/n0Ph8XH5>
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Friday, June 7, 2019 at 9:15 AM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>, OpenSIPS devel mailling
> list <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> How often/easy is to reproduce this crash (if possible) ?
> Brainstorming with Razvan, we suspect a race (on the msg
> save in shmem in transaction) between the process doing
> the cleanup after the async resume and the process running
> the failure route (due th 503).
>
> But this is just a supposition, eventually you can
> validate it or not by removing the async ??
>
> And on the double ACK - I'm not 100% it is a actually a
> double one, as the second has a smaller MF value (69,
> versus the 70 on the first ACK).
>
> Regards,
>
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> https://www.opensips-solutions.com
>
> OpenSIPS Summit 2019
>
> https://www.opensips.org/events/Summit-2019Amsterdam/
>
> On 06/07/2019 03:52 PM, Ben Newlin wrote:
>
> Bogdan,
>
> Sorry, I should have thought to actually look at the
> trace and examine this call.
>
> 1) Yes
>
> 2) The Called Party is 10.32.20.60, which is another
> OpenSIPS instance. The crashed instance received the
> "503 Service Unavailable" approximately 8-10 ms after
> sending the INVITE.
>
> There is a SIP trace of the exchange here:
> https://pastebin.com/6bttsSVD
> <https://pastebin.com/6bttsSVD>.
>
> One oddity I saw is that the crashed process appears
> to send (or at least siptrace) the ACK twice.
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Thursday, June 6, 2019 at 11:42 AM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>, Ben Newlin
> <Ben.Newlin at genesys.com> <mailto:Ben.Newlin at genesys.com>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> Thanks for "another" report :).
>
> Questions:
> 1) do you do any async for the INVITE in this crash ?
> 2) if it is an YES to (1), is the caller party
> generating the "503 Service Unavailable" (which
> triggers the crash) - 10.32.20.60 ?? - a really close
> (from net delay perspective) and fast to answer party ?
>
> Regards,
>
>
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> https://www.opensips-solutions.com
>
> OpenSIPS Summit 2019
>
> https://www.opensips.org/events/Summit-2019Amsterdam/
>
> On 06/05/2019 10:02 PM, Ben Newlin wrote:
>
> We have had another crash today.
>
> Backtrace is here: https://pastebin.com/q4RQC7kS
> <https://pastebin.com/q4RQC7kS>
>
> I found this in the log at the time of the crash:
>
> Jun 5 17:54:10 [4978] CRITICAL:core:sig_usr:
> segfault in process pid: 4978, id: 8
>
> Please let me know if any further information can
> be useful.
>
> Ben Newlin
>
> *From: *Devel <devel-bounces at lists.opensips.org>
> <mailto:devel-bounces at lists.opensips.org> on
> behalf of Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>
> *Reply-To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Date: *Friday, May 10, 2019 at 6:31 PM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> I found this in the log at the time of the crash:
>
> kernel: opensips[5003]: segfault at 30 ip
> 00007fbd4c8f59d0 sp 00007ffcaa850c80 error 6 in
> tm.so[7fbd4c887000+8e000]
>
> Ben Newlin
>
> *From: *Devel <devel-bounces at lists.opensips.org>
> <mailto:devel-bounces at lists.opensips.org> on
> behalf of Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>
> *Reply-To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Date: *Friday, May 10, 2019 at 5:44 PM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Subject: *[OpenSIPS-Devel] OpenSIPS Crash
>
> Hello,
>
> We had a crash today of our OpenSIPS instance.
>
> Backtrace is here: https://pastebin.com/QbRJimwx
>
> # opensips -V
>
> version: opensips 2.4.5 (x86_64/linux)
>
> flags: STATS: On, DISABLE_NAGLE, USE_MCAST,
> SHM_MMAP, PKG_MALLOC, F_MALLOC,
> FAST_LOCK-ADAPTIVE_WAIT
>
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE
> 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE
> 65535
>
> poll method support: poll, epoll, sigio_rt, select.
>
> git revision: d025b4f61
>
> main.c compiled on 20:58:31 May 9 2019 with gcc 7
>
> Ben Newlin
>
>
>
>
>
>
>
> _______________________________________________
>
> Devel mailing list
>
> Devel at lists.opensips.org
> <mailto:Devel at lists.opensips.org>
>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
>
> Devel mailing list
>
> Devel at lists.opensips.org <mailto:Devel at lists.opensips.org>
>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20190627/98e37ada/attachment-0001.html>
More information about the Devel
mailing list