[OpenSIPS-Devel] OpenSIPS Crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Mon Nov 19 09:03:30 EST 2018
Hi Ben,
With someadditional information that we got from a different feed, we
managed to reproduce the crashand Liviu is working on a fix as we speak.
I will keep you posted, but many thanks for your help in investigating
this issue.
Best regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 11/13/2018 07:41 PM, Ben Newlin wrote:
>
> Bogdan,
>
> Yes, we are setting acc_extra variables in our branch routes, which
> are sometimes (but not always) called from failure route. Are
> acc_extra variables not available for use in branch_routes?
>
> We don’t currently use drop_accounting anywhere in our script. If I
> call it before that branch_route then it will stop accounting for that
> call, right? We need to have accounting records for the call, so I’m
> not sure how that would resolve the issue?
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> *Date: *Tuesday, November 13, 2018 at 9:13 AM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>, OpenSIPS devel mailling
> list <devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> Thanks for the info. The crash happens when you try to set an acc
> extra variable in branch route (when a creating a new branch via
> failure route, on timeout).
>
> Now, do you use the drop accounting in your script ? and considering
> the above scenario, it is possible to have the drop acc before the
> branch route ?
>
> Regards,
>
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 11/12/2018 08:55 PM, Ben Newlin wrote:
>
> Bogdan,
>
> We upgraded to 2.4.3 and the crash reproduced today. Backtrace is
> available here: https://pastebin.com/CZxQnZdR
> <https://pastebin.com/CZxQnZdR>.
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Wednesday, November 7, 2018 at 6:18 AM
> *To: *OpenSIPS devel mailling list <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>, Ben Newlin
> <Ben.Newlin at genesys.com> <mailto:Ben.Newlin at genesys.com>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> The BT indicates a double free for the accounting context - and I
> noticed you use 2.4.1 version. And yes, there was an issue related
> to acc context, issue that was fixed starting 2.4.2. So, could you
> upgrade to the latest 2.4 and see if the crash still happens ? As
> I think the fix is already there.
>
> Regards,
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
> OpenSIPS Bootcamp 2018
>
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
>
> Jackpot - you get it right !! I will start digging into the
> trace, but please keep the corefile, I might need it later.
>
> Thanks and regards,
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
> OpenSIPS Bootcamp 2018
>
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 11/06/2018 10:24 PM, Ben Newlin wrote:
>
> Bogdan,
>
> I have reproduced this crash and verified this time that
> the flags were set.
>
> $ opensips -V
>
> version: opensips 2.4.1 (x86_64/linux)
>
> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
> PKG_MALLOC, QM_MALLOC, DBG_MALLOC,
> FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
>
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
> MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>
> poll method support: poll, epoll, sigio_rt, select.
>
> git revision: 5d042cffc
>
> main.c compiled on 23:38:55 Nov 5 2018 with gcc 7
>
> Backtrace is available here: https://pastebin.com/KTQjkCwq
> <https://pastebin.com/KTQjkCwq>
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Thursday, November 1, 2018 at 1:19 PM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>, OpenSIPS devel mailling
> list <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> According to the backtrace, the memory debugger was not
> activated. Do an "opensips -V" to check the resulting
> compile flags - do you see the DBG_MALLOC and QM_MALLOC ?
>
> Regards,
>
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
> OpenSIPS Bootcamp 2018
>
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/31/2018 05:04 PM, Ben Newlin wrote:
>
> Bogdan,
>
> I was able to compile with those options and the crash
> has occurred again. Backtrace is here:
> https://pastebin.com/dezi9xUU
> <https://pastebin.com/dezi9xUU>.
>
> Even though I had `memdump=1` set in my script, there
> was no extra memory debugging information in the logs
> prior to or at the time of the crash. I’m not sure if
> that is expected or not.
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Monday, October 29, 2018 at 8:11 AM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>, OpenSIPS devel
> mailling list <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> You can change the compile flags via the Makefile.conf
> file - the menuconfig is also updating that file. So
> during your build you can simply push a pre-modified
> Makefile.conf file with the options needed for memory
> debugging.
>
> Regards,
>
>
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
> OpenSIPS Bootcamp 2018
>
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/26/2018 05:14 PM, Ben Newlin wrote:
>
> Bogdan,
>
> Unfortunately, we have run into a similar issue
> before. Our build system is completely automated
> and there is no way to inject the `make
> menuconfig` interactive step into that process. If
> I were to be testing this locally I might be able
> to work something out, but I could never get such
> a build into our testing environment which is
> where the crashes are occurring.
>
> Do you have instructions for enabling memory
> debugging that do not require using the
> interactive TUI tool? What does the menuconfig
> program do when these options are selected? Are
> there some defines or other settings we can change
> ourselves and bypass menuconfig?
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Friday, October 26, 2018 at 4:59 AM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>, Ben Newlin
> <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> all the BT's points to crashes while doing memory
> ops. I suspect a memory corruption that randomly
> triggers crashes in different parts of the code.
>
> Could you try to re-compile with memory debugging
> support ? See
> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
> the "How to handle it" section.
>
> Regards,
>
>
>
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
> OpenSIPS Bootcamp 2018
>
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/24/2018 04:28 AM, Ben Newlin wrote:
>
> We have had 2 more crashes today.
>
> Crash 2: https://pastebin.com/rMruBQcZ
> <https://pastebin.com/rMruBQcZ>
>
> This crash appears to have occurred while
> processing an initial INVITE request. I could
> not see anything unusual about the request. I
> cannot tell if this crash is related to the
> others.
>
> Crash 3: https://pastebin.com/Gmk1m4NT
> <https://pastebin.com/Gmk1m4NT>
>
> This crash follows the pattern of the original
> crash I reported.
>
> Ben Newlin
>
> *From: *Devel
> <devel-bounces at lists.opensips.org>
> <mailto:devel-bounces at lists.opensips.org> on
> behalf of Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>
> *Reply-To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Date: *Monday, October 22, 2018 at 4:45 PM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Here is a better trace of the call:
> https://pastebin.com/gWpQR8E7
> <https://pastebin.com/gWpQR8E7>
>
> Ben Newlin
>
> *From: *Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>
> *Date: *Monday, October 22, 2018 at 4:34 PM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>
> *Subject: *OpenSIPS Crash
>
> Hello,
>
> We have been having sporadic crashes and I was
> recently able to recover a core dump for one.
> I have uploaded it here:
> https://pastebin.com/ABktcYcH
> <https://pastebin.com/ABktcYcH>.
>
> I picked out a Call-ID from the crash data and
> took a look in our tracing. I have uploaded it
> here: https://pastebin.com/ZEzUUKZ5
> <https://pastebin.com/ZEzUUKZ5>.
>
> It appears that a downstream server was
> extremely lagged and failed to respond to an
> INVITE. We sent the INVITE to another server
> and the call was connected, but then
> eventually the original server “caught up” and
> sent a burst of 200 OK responses. The crash
> seems to have occurred processing the ACK to
> one of these responses.
>
> Ben Newlin
>
>
>
>
>
>
>
>
> _______________________________________________
>
> Devel mailing list
>
> Devel at lists.opensips.org
> <mailto:Devel at lists.opensips.org>
>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
>
> Devel mailing list
>
> Devel at lists.opensips.org <mailto:Devel at lists.opensips.org>
>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20181119/9d6ecc7e/attachment-0001.html>
More information about the Devel
mailing list