[OpenSIPS-Devel] OpenSIPS Crash

Wed Jun 26 09:00:51 EDT 2019

Bogdan,

Thanks for the response. I will get the info you need. When you say to print in frame 2, is there some command I must issue to put the current context into frame 2 (or 7 or 8) or do I just issue the command you provided? Apologies, I use GDB very seldomly.

Also, while we are using async support in this proxy, none of our current TLS traffic is using any async functions. Do you think other calls using async could be interfering with the TLS calls?

Ben Newlin

From: Bogdan-Andrei Iancu <bogdan at opensips.org>
Date: Wednesday, June 26, 2019 at 5:25 AM
To: Ben Newlin <Ben.Newlin at genesys.com>, OpenSIPS devel mailling list <devel at lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

In any of the corefile, in frame 2 could you print
    p *((struct qm_frag*)((char*)(p)-sizeof(struct qm_frag)))

Now, I see that the backtrace via libcurl & libcrypto may go via some error cleanup - see frame #7 and #8 - so , do you see any curl error prior to the crash (ideally from the same process) ?

If you can reproduce it easily, could you try without the async support, just as a PoC only.

Thanks,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/25/2019 10:04 PM, Ben Newlin wrote:
Yes, we are using tls_mgm and proto_tls. We have the following version of openssl-devel installed:

# yum info openssl-devel
Name        : openssl-devel
Arch        : x86_64
Epoch       : 1
Version     : 1.0.2k
Release     : 16.amzn2.1.1
Size        : 3.1 M
Repo        : installed
From repo   : amzn2-core
Summary     : Files for development of applications which will use OpenSSL
URL         : http://www.openssl.org/<http://www.openssl.org/>
License     : OpenSSL
Description : OpenSSL is a toolkit for supporting cryptography. The openssl-devel
            : package contains include files needed to develop applications which
            : support various cryptographic algorithms and protocols.

Ben Newlin

From: Bogdan-Andrei Iancu <bogdan at opensips.org><mailto:bogdan at opensips.org>
Date: Tuesday, June 25, 2019 at 2:54 PM
To: Ben Newlin <Ben.Newlin at genesys.com><mailto:Ben.Newlin at genesys.com>, OpenSIPS devel mailling list <devel at lists.opensips.org><mailto:devel at lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Thank you Ben :)

The backtraces report a double free, but the mem ops are performed via libcrypto ( opensips registers mem-specific functions to lib crypto).

Are you using in your opensips cfg any other module that might require ssl/crtypto, like tls_mgm, proto_wss ?

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/25/2019 06:49 PM, Ben Newlin wrote:
Bogdan,

You have good timing. :) I was just collecting the data from a crash this morning with the extra flags set.

# opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 22:27:12 Jun 24 2019 with gcc 7

[1] https://pastebin.com/aj1mvk0N<https://pastebin.com/aj1mvk0N>
[2] https://pastebin.com/kfSz9Ks0<https://pastebin.com/kfSz9Ks0>
[3] https://pastebin.com/hptH9yiA<https://pastebin.com/hptH9yiA>
[4] https://pastebin.com/GfYYZxzB<https://pastebin.com/GfYYZxzB>
[5] https://pastebin.com/CzYiNAch<https://pastebin.com/CzYiNAch>

Ben Newlin

From: Bogdan-Andrei Iancu <bogdan at opensips.org><mailto:bogdan at opensips.org>
Date: Tuesday, June 25, 2019 at 11:39 AM
To: Ben Newlin <Ben.Newlin at genesys.com><mailto:Ben.Newlin at genesys.com>, OpenSIPS devel mailling list <devel at lists.opensips.org><mailto:devel at lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

The backtrace + corefile should do it ;).

Best regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 07:19 PM, Ben Newlin wrote:
Bogdan,

Yes I will enable those. This seems to be reproducing fairly often so hopefully we will get some more info quickly. Is there anything else I need to collect at time of the crash with this enabled or is the core dump sufficient?

Ben Newlin

From: Bogdan-Andrei Iancu <bogdan at opensips.org><mailto:bogdan at opensips.org>
Date: Thursday, June 20, 2019 at 12:15 PM
To: Ben Newlin <Ben.Newlin at genesys.com><mailto:Ben.Newlin at genesys.com>, OpenSIPS devel mailling list <devel at lists.opensips.org><mailto:devel at lists.opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

Hi Ben,

This looks like very generic - a crash due a previous memory corruption - is it possible for you to run with QM_MALLOC + DBG_MALLOC ? otherwise it is rather impossible to identify the original source of corruption.

Thanks and regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer

  https://www.opensips-solutions.com<https://www.opensips-solutions.com>

OpenSIPS Summit 2019

  https://www.opensips.org/events/Summit-2019Amsterdam/<https://www.opensips.org/events/Summit-2019Amsterdam/>
On 06/20/2019 05:34 PM, Ben Newlin wrote:
We have had another crash this morning: https://pastebin.com/dmXYbhzC<https://pastebin.com/dmXYbhzC>

OpenSIPS version is the same as below.

Ben Newlin

From: Ben Newlin <Ben.Newlin at genesys.com><mailto:Ben.Newlin at genesys.com>
Date: Wednesday, June 19, 2019 at 2:38 PM
To: OpenSIPS devel mailling list <devel at lists.opensips.org><mailto:devel at lists.opensips.org>, Bogdan-Andrei Iancu <bogdan at opensips.org><mailto:bogdan at opensips.org>
Subject: Re: [OpenSIPS-Devel] OpenSIPS Crash

The Github issue has the version info for that, for all the other crashes the version is below.

$ opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 60097425d
main.c compiled on 18:06:35 Jun 13 2019 with gcc 7

Ben Newlin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/devel/attachments/20190626/a45b1ba1/attachment-0001.html>