[OpenSIPS-Devel] OpenSIPS Crash

Bogdan-Andrei Iancu bogdan at opensips.org
Thu Nov 1 13:19:01 EDT 2018


Hi Ben,

According to the backtrace, the memory debugger was not activated. Do an 
"opensips -V" to check the resulting compile flags - do you see the 
DBG_MALLOC and QM_MALLOC ?

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/31/2018 05:04 PM, Ben Newlin wrote:
>
> Bogdan,
>
> I was able to compile with those options and the crash has occurred 
> again. Backtrace is here: https://pastebin.com/dezi9xUU.
>
> Even though I had `memdump=1` set in my script, there was no extra 
> memory debugging information in the logs prior to or at the time of 
> the crash. I’m not sure if that is expected or not.
>
> Ben Newlin
>
> *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
> *Date: *Monday, October 29, 2018 at 8:11 AM
> *To: *Ben Newlin <Ben.Newlin at genesys.com>, OpenSIPS devel mailling 
> list <devel at lists.opensips.org>
> *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
> Hi Ben,
>
> You can change the compile flags via the Makefile.conf file - the 
> menuconfig is also updating that file. So during your build you can 
> simply push a pre-modified Makefile.conf file with the options needed 
> for memory debugging.
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
>    http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>    http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/26/2018 05:14 PM, Ben Newlin wrote:
>
>     Bogdan,
>
>     Unfortunately, we have run into a similar issue before. Our build
>     system is completely automated and there is no way to inject the
>     `make menuconfig` interactive step into that process. If I were to
>     be testing this locally I might be able to work something out, but
>     I could never get such a build into our testing environment which
>     is where the crashes are occurring.
>
>     Do you have instructions for enabling memory debugging that do not
>     require using the interactive TUI tool? What does the menuconfig
>     program do when these options are selected? Are there some defines
>     or other settings we can change ourselves and bypass menuconfig?
>
>     Ben Newlin
>
>     *From: *Bogdan-Andrei Iancu <bogdan at opensips.org>
>     <mailto:bogdan at opensips.org>
>     *Date: *Friday, October 26, 2018 at 4:59 AM
>     *To: *OpenSIPS devel mailling list <devel at lists.opensips.org>
>     <mailto:devel at lists.opensips.org>, Ben Newlin
>     <Ben.Newlin at genesys.com> <mailto:Ben.Newlin at genesys.com>
>     *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
>     Hi Ben,
>
>     all the BT's points to crashes while doing memory ops. I suspect a
>     memory corruption that randomly triggers crashes in different
>     parts of the code.
>
>     Could you try to re-compile with memory debugging support ? See
>     http://www.opensips.org/Documentation/TroubleShooting-OutOfMem,
>     the "How to handle it" section.
>
>     Regards,
>
>
>     Bogdan-Andrei Iancu
>
>       
>
>     OpenSIPS Founder and Developer
>
>        http://www.opensips-solutions.com
>
>     OpenSIPS Bootcamp 2018
>
>        http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
>     On 10/24/2018 04:28 AM, Ben Newlin wrote:
>
>         We have had 2 more crashes today.
>
>         Crash 2: https://pastebin.com/rMruBQcZ
>         <https://pastebin.com/rMruBQcZ>
>
>         This crash appears to have occurred while processing an
>         initial INVITE request. I could not see anything unusual about
>         the request. I cannot tell if this crash is related to the others.
>
>         Crash 3: https://pastebin.com/Gmk1m4NT
>         <https://pastebin.com/Gmk1m4NT>
>
>         This crash follows the pattern of the original crash I reported.
>
>         Ben Newlin
>
>         *From: *Devel <devel-bounces at lists.opensips.org>
>         <mailto:devel-bounces at lists.opensips.org> on behalf of Ben
>         Newlin <Ben.Newlin at genesys.com> <mailto:Ben.Newlin at genesys.com>
>         *Reply-To: *OpenSIPS devel mailling list
>         <devel at lists.opensips.org> <mailto:devel at lists.opensips.org>
>         *Date: *Monday, October 22, 2018 at 4:45 PM
>         *To: *OpenSIPS devel mailling list <devel at lists.opensips.org>
>         <mailto:devel at lists.opensips.org>
>         *Subject: *Re: [OpenSIPS-Devel] OpenSIPS Crash
>
>         Here is a better trace of the call:
>         https://pastebin.com/gWpQR8E7 <https://pastebin.com/gWpQR8E7>
>
>         Ben Newlin
>
>         *From: *Ben Newlin <Ben.Newlin at genesys.com>
>         <mailto:Ben.Newlin at genesys.com>
>         *Date: *Monday, October 22, 2018 at 4:34 PM
>         *To: *OpenSIPS devel mailling list <devel at lists.opensips.org>
>         <mailto:devel at lists.opensips.org>
>         *Subject: *OpenSIPS Crash
>
>         Hello,
>
>         We have been having sporadic crashes and I was recently able
>         to recover a core dump for one. I have uploaded it here:
>         https://pastebin.com/ABktcYcH <https://pastebin.com/ABktcYcH>.
>
>         I picked out a Call-ID from the crash data and took a look in
>         our tracing. I have uploaded it here:
>         https://pastebin.com/ZEzUUKZ5 <https://pastebin.com/ZEzUUKZ5>.
>
>         It appears that a downstream server was extremely lagged and
>         failed to respond to an INVITE. We sent the INVITE to another
>         server and the call was connected, but then eventually the
>         original server “caught up” and sent a burst of 200 OK
>         responses. The crash seems to have occurred processing the ACK
>         to one of these responses.
>
>         Ben Newlin
>
>
>
>
>
>         _______________________________________________
>
>         Devel mailing list
>
>         Devel at lists.opensips.org <mailto:Devel at lists.opensips.org>
>
>         http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
>
>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20181101/e986582d/attachment-0001.html>


More information about the Devel mailing list