[OpenSIPS-Devel] OpenSIPS Crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Wed Nov 7 06:18:43 EST 2018
Hi Ben,
The BT indicates a double free for the accounting context - and I
noticed you use 2.4.1 version. And yes, there was an issue related to
acc context, issue that was fixed starting 2.4.2. So, could you upgrade
to the latest 2.4 and see if the crash still happens ? As I think the
fix is already there.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 11/06/2018 11:13 PM, Bogdan-Andrei Iancu wrote:
> Jackpot - you get it right !! I will start digging into the trace, but
> please keep the corefile, I might need it later.
>
> Thanks and regards,
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
> On 11/06/2018 10:24 PM, Ben Newlin wrote:
>>
>> Bogdan,
>>
>> I have reproduced this crash and verified this time that the flags
>> were set.
>>
>> $ opensips -V
>>
>> version: opensips 2.4.1 (x86_64/linux)
>>
>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
>> QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
>>
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>
>> poll method support: poll, epoll, sigio_rt, select.
>>
>> git revision: 5d042cffc
>>
>> main.c compiled on 23:38:55 Nov 5 2018 with gcc 7
>>
>> Backtrace is available here: https://pastebin.com/KTQjkCwq
>> <https://pastebin.com/KTQjkCwq>
>>
>> Ben Newlin
>>
>> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
>> *Date: *Thursday, November 1, 2018 at 1:19 PM
>> *To: *Ben Newlin <Ben.Newlin at genesys.com>, OpenSIPS devel mailling
>> list <devel at lists.opensips.org>
>> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>>
>> Hi Ben,
>>
>> According to the backtrace, the memory debugger was not activated. Do
>> an "opensips -V" to check the resulting compile flags - do you see
>> the DBG_MALLOC and QM_MALLOC ?
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/31/2018 05:04 PM, Ben Newlin wrote:
>>
>> Bogdan,
>>
>> I was able to compile with those options and the crash has
>> occurred again. Backtrace is here: https://pastebin.com/dezi9xUU
>> <https://pastebin.com/dezi9xUU>.
>>
>> Even though I had `memdump=1` set in my script, there was no
>> extra memory debugging information in the logs prior to or at the
>> time of the crash. I’m not sure if that is expected or not.
>>
>> Ben Newlin
>>
>> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
>> <mailto:bogdan at opensips.org>
>> *Date: *Monday, October 29, 2018 at 8:11 AM
>> *To: *Ben Newlin <Ben.Newlin at genesys.com>
>> <mailto:Ben.Newlin at genesys.com>, OpenSIPS devel mailling list
>> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
>> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>>
>> Hi Ben,
>>
>> You can change the compile flags via the Makefile.conf file - the
>> menuconfig is also updating that file. So during your build you
>> can simply push a pre-modified Makefile.conf file with the
>> options needed for memory debugging.
>>
>> Regards,
>>
>>
>> Bogdan-Andrei Iancu
>>
>>
>>
>> OpenSIPS Founder and Developer
>>
>> http://www.opensips-solutions.com
>>
>> OpenSIPS Bootcamp 2018
>>
>> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/26/2018 05:14 PM, Ben Newlin wrote:
>>
>> Bogdan,
>>
>> Unfortunately, we have run into a similar issue before. Our
>> build system is completely automated and there is no way to
>> inject the `make menuconfig` interactive step into that
>> process. If I were to be testing this locally I might be able
>> to work something out, but I could never get such a build
>> into our testing environment which is where the crashes are
>> occurring.
>>
>> Do you have instructions for enabling memory debugging that
>> do not require using the interactive TUI tool? What does the
>> menuconfig program do when these options are selected? Are
>> there some defines or other settings we can change ourselves
>> and bypass menuconfig?
>>
>> Ben Newlin
>>
>> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
>> <mailto:bogdan at opensips.org>
>> *Date: *Friday, October 26, 2018 at 4:59 AM
>> *To: *OpenSIPS devel mailling list <devel at lists.opensips.org>
>> <mailto:devel at lists.opensips.org>, Ben Newlin
>> <Ben.Newlin at genesys.com> <mailto:Ben.Newlin at genesys.com>
>> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>>
>> Hi Ben,
>>
>> all the BT's points to crashes while doing memory ops. I
>> suspect a memory corruption that randomly triggers crashes in
>> different parts of the code.
>>
>> Could you try to re-compile with memory debugging support ?
>> See
>> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
>> the "How to handle it" section.
>>
>> Regards,
>>
>>
>>
>> Bogdan-Andrei Iancu
>>
>>
>>
>> OpenSIPS Founder and Developer
>>
>> http://www.opensips-solutions.com
>>
>> OpenSIPS Bootcamp 2018
>>
>> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/24/2018 04:28 AM, Ben Newlin wrote:
>>
>> We have had 2 more crashes today.
>>
>> Crash 2: https://pastebin.com/rMruBQcZ
>> <https://pastebin.com/rMruBQcZ>
>>
>> This crash appears to have occurred while processing an
>> initial INVITE request. I could not see anything unusual
>> about the request. I cannot tell if this crash is related
>> to the others.
>>
>> Crash 3: https://pastebin.com/Gmk1m4NT
>> <https://pastebin.com/Gmk1m4NT>
>>
>> This crash follows the pattern of the original crash I
>> reported.
>>
>> Ben Newlin
>>
>> *From: *Devel <devel-bounces at lists.opensips.org>
>> <mailto:devel-bounces at lists.opensips.org> on behalf of
>> Ben Newlin <Ben.Newlin at genesys.com>
>> <mailto:Ben.Newlin at genesys.com>
>> *Reply-To: *OpenSIPS devel mailling list
>> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
>> *Date: *Monday, October 22, 2018 at 4:45 PM
>> *To: *OpenSIPS devel mailling list
>> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
>> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>>
>> Here is a better trace of the call:
>> https://pastebin.com/gWpQR8E7 <https://pastebin.com/gWpQR8E7>
>>
>> Ben Newlin
>>
>> *From: *Ben Newlin <Ben.Newlin at genesys.com>
>> <mailto:Ben.Newlin at genesys.com>
>> *Date: *Monday, October 22, 2018 at 4:34 PM
>> *To: *OpenSIPS devel mailling list
>> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
>> *Subject: *OpenSIPS Crash
>>
>> Hello,
>>
>> We have been having sporadic crashes and I was recently
>> able to recover a core dump for one. I have uploaded it
>> here: https://pastebin.com/ABktcYcH
>> <https://pastebin.com/ABktcYcH>.
>>
>> I picked out a Call-ID from the crash data and took a
>> look in our tracing. I have uploaded it here:
>> https://pastebin.com/ZEzUUKZ5
>> <https://pastebin.com/ZEzUUKZ5>.
>>
>> It appears that a downstream server was extremely lagged
>> and failed to respond to an INVITE. We sent the INVITE to
>> another server and the call was connected, but then
>> eventually the original server “caught up” and sent a
>> burst of 200 OK responses. The crash seems to have
>> occurred processing the ACK to one of these responses.
>>
>> Ben Newlin
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> Devel mailing list
>>
>> Devel at lists.opensips.org <mailto:Devel at lists.opensips.org>
>>
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
> _______________________________________________
> Devel mailing list
> Devel at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20181107/d894eb81/attachment-0001.html>
More information about the Devel
mailing list