[OpenSIPS-Users] OpenSIPs crashed

[Ahmed Munir]

Found coredump on one of the server, see some partial message below while
taking the back trace;



Core was generated by `/usr/sbin/opensips -P /var/run/opensips.pid -m 64 -u
opensips -g opensips'.

Program terminated with signal 11, Segmentation fault.

#0  0x00007f650687a069 in sip_msg_cloner () from
/usr/lib64/opensips/modules/tm.so

Missing separate debuginfos, use: debuginfo-install
opensips-1.6.3-notls.x86_64


Please advise what might be the reason causing opensips to crash.

On Thu, Jan 12, 2017 at 1:28 PM, Ahmed Munir <ahmedmunir007 at gmail.com>
wrote:

>
> The version currently running is 1.6.3.  Today again we got the opensips
> crashed issues i.e. 5 out of 8 were crashed due to below message (common on
> all 5);
>
> Jan 12 10:07:36 QORCLPSIPROXY02 kernel: opensips[2820]: segfault at 0 ip
> 00000000004a2936 sp 00007fff5cfaa430 error 6 in opensips[400000+13a000]
>
> Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Can't open 'core.2820':
> Permission denied
>
> Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Saved core dump of pid 2820
> (/usr/sbin/opensips) to /var/spool/abrt/ccpp-2017-01-12-10:07:36-2820
> (70246400 bytes)
>
> Jan 12 10:07:36 QORCLPSIPROXY02 abrtd: Directory 'ccpp-2017-01-12-10:07:36-2820'
> creation detected
>
> Jan 12 10:07:38 QORCLPSIPROXY02 kernel: Bridge firewalling registered
>
> Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Sending an email...
>
> Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: /usr/sbin/sendmail: No such file or
> directory
>
> Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: . . . message not sent.
>
> Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Error running '/bin/mailx'
>
> Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2017-01-12-10:07:36-2820' exited with 1
>
> Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2017-01-12-10:07:36-2820'
>
>
> We would like to know why we are getting this segmentation fault? Is there
> a way to backtrace the issue as don't have the core dump? Further added,
> did someone faced this similar issue(s) in past and got this fixed without
> upgrade?
>
> Note: we are running opensips for quite a while didn't faced this kind of
> issue and there is no changes made on opensips routing script.
>
>
>> From: Răzvan Crainea <razvan at opensips.org>
>>> To: users at lists.opensips.org
>>> Subject: Re: [OpenSIPS-Users] OpenSIPs crashed
>>> Message-ID: <2b35e2dc-6e11-1787-b87e-33bd29a328f3 at opensips.org>
>>> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>>>
>>> Hi, Ahmed!
>>>
>>> Make sure OpenSIPS is run as root and it is allowed to write in the
>>> /var/spool/abrt/ directory, otherwise it is unable to write the core
>>> dump, therefore we can't inspect it to say what is happening. If this
>>> does not work, make OpenSIPS write the core dump in a writeble directory
>>> by changing the /proc/sys/kernel/core_pattern settings.
>>> Also, please let us know the version of OpenSIPS you are running.
>>>
>>> Best regards,
>>>
>>> Răzvan Crainea
>>> OpenSIPS Solutions
>>> www.opensips-solutions.com
>>>
>>>
>>> On 01/11/2017 11:10 PM, Ahmed Munir wrote:
>>> > Hi,
>>> >
>>> > Our OpenSIPs service crashed with below error;
>>> >
>>> > Jan 11 12:16:19 QORBLPSIPROXY05 abrtd: Directory
>>> > 'ccpp-2017-01-11-12:16:19-2807' creation detected
>>> > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Saved core dump of pid
>>> > 2807 (/usr/sbin/opensips) to
>>> > /var/spool/abrt/ccpp-2017-01-11-12:16:19-2807 (70225920 bytes)
>>> > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Can't open 'core.2807':
>>> > Permission denied
>>> > Jan 11 12:16:19 QORBLPSIPROXY05 kernel: opensips[2807]: segfault at 29
>>> > ip 00000000004bae7a sp 00007fffdb7734d0 error 4 in
>>> opensips[400000+13a000]
>>> >
>>> >
>>> > We would like to know, what might be the reason for the crash.
>>> >
>>> > Further added, there is another server we are running OpenSIPs, the
>>> > opensips child processes utilizing 100% of CPU and the system load
>>> > average reach around 'load average: 20.01, 18.03, 24.00' as normally
>>> > it is below 1 (load  average).
>>> >
>>> > After looking into logs, unable to find the info what might causing
>>> > the CPU to spike.
>>> >
>>> > Please advise what useful steps to take for narrowing down this issue.
>>> >
>>> >
>>> > --
>>> > Regards,
>>> >
>>> > Ahmed Munir Chohan
>>> >
>>>
>>>
>>
>>
>> --
>> Regards,
>>
>> Ahmed Munir Chohan
>>
>>
>
>
> --
> Regards,
>
> Ahmed Munir Chohan
>
>


-- 
Regards,

Ahmed Munir Chohan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170112/13dfe7bb/attachment-0001.html>