<div dir="ltr"><div>Found coredump on one of the server, see some partial message below while taking the back trace;<br><br><br><br>
<div class="gmail-WordSection1">
<p class="MsoNormal" style="margin:2pt 0in;line-height:normal"><span style="font-size:9pt;font-family:"segoe ui",sans-serif;color:black">Core was generated by
`/usr/sbin/opensips -P /var/run/opensips.pid -m 64 -u opensips -g opensips'.</span></p>
<p class="MsoNormal" style="margin:2pt 0in;line-height:normal"><span style="font-size:9pt;font-family:"segoe ui",sans-serif;color:black">Program terminated with
signal 11, Segmentation fault.</span></p>
<p class="MsoNormal" style="margin:2pt 0in;line-height:normal"><span style="font-size:9pt;font-family:"segoe ui",sans-serif;color:black">#0 0x00007f650687a069 in
sip_msg_cloner () from /usr/lib64/opensips/modules/tm.so</span></p>
<p class="MsoNormal" style="margin:2pt 0in;line-height:normal"><span style="font-size:9pt;font-family:"segoe ui",sans-serif;color:black">Missing separate
debuginfos, use: debuginfo-install opensips-1.6.3-notls.x86_64</span></p>
</div>
<br><br></div>Please advise what might be the reason causing opensips to crash. <br><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 12, 2017 at 1:28 PM, Ahmed Munir <span dir="ltr"><<a href="mailto:ahmedmunir007@gmail.com" target="_blank">ahmedmunir007@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra">The version currently running is 1.6.3. Today again we got the opensips crashed issues i.e. 5 out of 8 were crashed due to below message (common on all 5);<br><br>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:07:36 QORCLPSIPROXY02 kernel: opensips[2820]: segfault at 0 ip
00000000004a2936 sp 00007fff5cfaa430 error 6 in opensips[400000+13a000]</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Can't open 'core.2820':
Permission denied</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Saved core dump of pid 2820
(/usr/sbin/opensips) to /var/spool/abrt/ccpp-2017-01-<wbr>12-10:07:36-2820 (70246400
bytes)</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:07:36 QORCLPSIPROXY02 abrtd: Directory
'ccpp-2017-01-12-10:07:36-<wbr>2820' creation detected</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:07:38 QORCLPSIPROXY02 kernel: Bridge firewalling registered</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Sending an email...</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: /usr/sbin/sendmail: No such file
or directory</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: . . . message not sent.</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Error running '/bin/mailx'</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: 'post-create' on
'/var/spool/abrt/ccpp-2017-01-<wbr>12-10:07:36-2820' exited with 1</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)" lang="EN-IN">Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Deleting problem directory
'/var/spool/abrt/ccpp-2017-01-<wbr>12-10:07:36-2820'</span></p>
<br><br></div><div class="gmail_extra">We would like to know why we are getting this segmentation fault? Is there a way to backtrace the issue as don't have the core dump? Further added, did someone faced this similar issue(s) in past and got this fixed without upgrade?<br><br></div><div class="gmail_extra">Note: we are running opensips for quite a while didn't faced this kind of issue and there is no changes made on opensips routing script.<br></div><div><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div class="gmail_extra"><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
From: Răzvan Crainea <<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>><br>
To: <a href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a><br>
Subject: Re: [OpenSIPS-Users] OpenSIPs crashed<br>
Message-ID: <<a href="mailto:2b35e2dc-6e11-1787-b87e-33bd29a328f3@opensips.org" target="_blank">2b35e2dc-6e11-1787-b87e-33bd2<wbr>9a328f3@opensips.org</a>><br>
Content-Type: text/plain; charset="utf-8"; Format="flowed"<br>
<br>
Hi, Ahmed!<br>
<br>
Make sure OpenSIPS is run as root and it is allowed to write in the<br>
/var/spool/abrt/ directory, otherwise it is unable to write the core<br>
dump, therefore we can't inspect it to say what is happening. If this<br>
does not work, make OpenSIPS write the core dump in a writeble directory<br>
by changing the /proc/sys/kernel/core_pattern settings.<br>
Also, please let us know the version of OpenSIPS you are running.<br>
<br>
Best regards,<br>
<br>
Răzvan Crainea<br>
OpenSIPS Solutions<br>
<a href="http://www.opensips-solutions.com" rel="noreferrer" target="_blank">www.opensips-solutions.com</a><div><div class="gmail-m_-5509751887500369gmail-h5"><br>
<br>
On 01/11/2017 11:10 PM, Ahmed Munir wrote:<br>
> Hi,<br>
><br>
> Our OpenSIPs service crashed with below error;<br>
><br>
> Jan 11 12:16:19 QORBLPSIPROXY05 abrtd: Directory<br>
> 'ccpp-<a href="tel:2017-01-11-12" value="+12017011112" target="_blank">2017-01-11-12</a>:16:19-2807<wbr>' creation detected<br>
> Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Saved core dump of pid<br>
> 2807 (/usr/sbin/opensips) to<br>
> /var/spool/abrt/ccpp-2017-01-1<wbr>1-12:16:19-2807 (70225920 bytes)<br>
> Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Can't open 'core.2807':<br>
> Permission denied<br>
> Jan 11 12:16:19 QORBLPSIPROXY05 kernel: opensips[2807]: segfault at 29<br>
> ip 00000000004bae7a sp 00007fffdb7734d0 error 4 in opensips[400000+13a000]<br>
><br>
><br>
> We would like to know, what might be the reason for the crash.<br>
><br>
> Further added, there is another server we are running OpenSIPs, the<br>
> opensips child processes utilizing 100% of CPU and the system load<br>
> average reach around 'load average: 20.01, 18.03, 24.00' as normally<br>
> it is below 1 (load average).<br>
><br>
> After looking into logs, unable to find the info what might causing<br>
> the CPU to spike.<br>
><br>
> Please advise what useful steps to take for narrowing down this issue.<br>
><br>
><br>
> --<br>
> Regards,<br>
><br>
> Ahmed Munir Chohan<br>
><br><br>
</div></div></blockquote></div><div><div class="gmail-m_-5509751887500369gmail-h5"><br><br clear="all"><br>-- <br><div class="gmail-m_-5509751887500369gmail-m_-8093862986160828663gmail_signature">Regards,<br><br>Ahmed Munir Chohan<br><br></div>
</div></div></div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail-m_-5509751887500369gmail_signature">Regards,<br><br>Ahmed Munir Chohan<br><br></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Regards,<br><br>Ahmed Munir Chohan<br><br></div>
</div></div></div></div></div>