[OpenSIPS-Devel] OpenSIPS Crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Tue Nov 6 16:13:56 EST 2018
Jackpot - you get it right !! I will start digging into the trace, but
please keep the corefile, I might need it later.
Thanks and regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 11/06/2018 10:24 PM, Ben Newlin wrote:
>
> Bogdan,
>
> I have reproduced this crash and verified this time that the flags
> were set.
>
> $ opensips -V
>
> version: opensips 2.4.1 (x86_64/linux)
>
> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC,
> QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT, DBG_LOCK
>
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
>
> poll method support: poll, epoll, sigio_rt, select.
>
> git revision: 5d042cffc
>
> main.c compiled on 23:38:55 Nov 5 2018 with gcc 7
>
> Backtrace is available here: https://pastebin.com/KTQjkCwq
> <https://pastebin.com/KTQjkCwq>
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> *Date: *Thursday, November 1, 2018 at 1:19 PM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>, OpenSIPS devel mailling
> list <devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> According to the backtrace, the memory debugger was not activated. Do
> an "opensips -V" to check the resulting compile flags - do you see the
> DBG_MALLOC and QM_MALLOC ?
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/31/2018 05:04 PM, Ben Newlin wrote:
>
> Bogdan,
>
> I was able to compile with those options and the crash has
> occurred again. Backtrace is here: https://pastebin.com/dezi9xUU
> <https://pastebin.com/dezi9xUU>.
>
> Even though I had `memdump=1` set in my script, there was no extra
> memory debugging information in the logs prior to or at the time
> of the crash. I’m not sure if that is expected or not.
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Monday, October 29, 2018 at 8:11 AM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>, OpenSIPS devel mailling list
> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> You can change the compile flags via the Makefile.conf file - the
> menuconfig is also updating that file. So during your build you
> can simply push a pre-modified Makefile.conf file with the options
> needed for memory debugging.
>
> Regards,
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
> OpenSIPS Bootcamp 2018
>
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/26/2018 05:14 PM, Ben Newlin wrote:
>
> Bogdan,
>
> Unfortunately, we have run into a similar issue before. Our
> build system is completely automated and there is no way to
> inject the `make menuconfig` interactive step into that
> process. If I were to be testing this locally I might be able
> to work something out, but I could never get such a build into
> our testing environment which is where the crashes are occurring.
>
> Do you have instructions for enabling memory debugging that do
> not require using the interactive TUI tool? What does the
> menuconfig program do when these options are selected? Are
> there some defines or other settings we can change ourselves
> and bypass menuconfig?
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> <mailto:bogdan at opensips.org>
> *Date: *Friday, October 26, 2018 at 4:59 AM
> *To: *OpenSIPS devel mailling list <devel at lists.opensips.org>
> <mailto:devel at lists.opensips.org>, Ben Newlin
> <Ben.Newlin at genesys.com> <mailto:Ben.Newlin at genesys.com>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> all the BT's points to crashes while doing memory ops. I
> suspect a memory corruption that randomly triggers crashes in
> different parts of the code.
>
> Could you try to re-compile with memory debugging support ?
> See
> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
> the "How to handle it" section.
>
> Regards,
>
>
>
> Bogdan-Andrei Iancu
>
>
>
> OpenSIPS Founder and Developer
>
> http://www.opensips-solutions.com
>
> OpenSIPS Bootcamp 2018
>
> http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/24/2018 04:28 AM, Ben Newlin wrote:
>
> We have had 2 more crashes today.
>
> Crash 2: https://pastebin.com/rMruBQcZ
> <https://pastebin.com/rMruBQcZ>
>
> This crash appears to have occurred while processing an
> initial INVITE request. I could not see anything unusual
> about the request. I cannot tell if this crash is related
> to the others.
>
> Crash 3: https://pastebin.com/Gmk1m4NT
> <https://pastebin.com/Gmk1m4NT>
>
> This crash follows the pattern of the original crash I
> reported.
>
> Ben Newlin
>
> *From: *Devel <devel-bounces at lists.opensips.org>
> <mailto:devel-bounces at lists.opensips.org> on behalf of Ben
> Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>
> *Reply-To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
> *Date: *Monday, October 22, 2018 at 4:45 PM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Here is a better trace of the call:
> https://pastebin.com/gWpQR8E7 <https://pastebin.com/gWpQR8E7>
>
> Ben Newlin
>
> *From: *Ben Newlin <Ben.Newlin at genesys.com>
> <mailto:Ben.Newlin at genesys.com>
> *Date: *Monday, October 22, 2018 at 4:34 PM
> *To: *OpenSIPS devel mailling list
> <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
> *Subject: *OpenSIPS Crash
>
> Hello,
>
> We have been having sporadic crashes and I was recently
> able to recover a core dump for one. I have uploaded it
> here: https://pastebin.com/ABktcYcH
> <https://pastebin.com/ABktcYcH>.
>
> I picked out a Call-ID from the crash data and took a look
> in our tracing. I have uploaded it here:
> https://pastebin.com/ZEzUUKZ5 <https://pastebin.com/ZEzUUKZ5>.
>
> It appears that a downstream server was extremely lagged
> and failed to respond to an INVITE. We sent the INVITE to
> another server and the call was connected, but then
> eventually the original server “caught up” and sent a
> burst of 200 OK responses. The crash seems to have
> occurred processing the ACK to one of these responses.
>
> Ben Newlin
>
>
>
>
>
>
> _______________________________________________
>
> Devel mailing list
>
> Devel at lists.opensips.org <mailto:Devel at lists.opensips.org>
>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20181106/b6385ac8/attachment-0001.html>
More information about the Devel
mailing list