<div dir="ltr">Hi Bogdan, <div><br></div><div>So, I've faced the problem once again, and followed the "Determining the cause" part of the link you gave me. </div><div>I've waited more than 20 minutes (near an hour), and re-sent some voice trafic to the problematic opensips server : memory errors just came after that. </div>
<div><br></div><div>So, I think it's a memory leak, no ? Now, what can I do to solve this ? </div><div><br></div><div>Thanks for your help, </div><div><br></div><div>Kevin</div><div><br></div><div><br></div></div><div class="gmail_extra">
<br clear="all"><div><b><div><span style="font-weight:normal">Bien cordialement, </span></div><div><span style="font-weight:normal">Best Regards, </span></div><div><span style="font-weight:normal"><br></span></div></b><b>Kevin MATHY</b> |<b> </b>Ingénieur VoIP<br>
<div><div><b><br></b></div></div></div>
<br><br><div class="gmail_quote">2014-08-20 17:22 GMT+02:00 Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><tt>Hi Kevin,<br>
<br>
So it is more a mem leak than a crash. Any chances to get a
memdump after memory is exhausted ? See:<br>
<a href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem" target="_blank">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a><br>
<br>
Regards,<br>
</tt><div class="">
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre></div><div><div class="h5">
On 20.08.2014 15:35, Kevin Mathy wrote:<br>
</div></div></div>
<blockquote type="cite"><div><div class="h5">
<div dir="ltr">Hi Bogdan,
<div><br>
</div>
<div>In fact, opensips didn't crashed : we got the previous
error messages in the logs, but then we switched our voice
traffic to another server, and opensips on the 1st one never
crashed... We restarted the service in the following night, to
be sure to make it rework well.</div>
<div><br>
</div>
<div>So, is there a way in this case to get some backtrace, even
if it didn't crashed ? Maybe something generated at startup ?</div>
<div><br>
</div>
<div>Thanks for your help, </div>
<div><br>
</div>
<div>Kevin</div>
</div>
<div class="gmail_extra"><br clear="all">
<div><b>
<div><span style="font-weight:normal">Bien cordialement, </span></div>
<div><span style="font-weight:normal">Best Regards, </span></div>
<div><span style="font-weight:normal"><br>
</span></div>
</b><b>Kevin MATHY</b> |<b> </b>Ingénieur VoIP<br>
<div>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2014-08-19 20:34 GMT+02:00
Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><tt>Hi Kevin,<br>
<br>
Try to extract at least a backtrace, to see what kind
of problem we are facing. Then we can decide if a mem
dump is required or not.<br>
<br>
Thanks,<br>
</tt>
<div>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
</div>
<div>
<div> On 19.08.2014 12:03, Kevin Mathy
wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi
Bogdan,</span>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Coming
back to you after some holidays, and
unfortunately, new Opensips crashes... </div>
<div style="font-family:arial,sans-serif;font-size:13px">Some
weeks ago, we upgraded our servers to OpenSIPS
1.11, but we faced a new crash, always with the
"out of pkg memory" error, last week.</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Here
is the exact version :</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">root@asbc2:/home/kemathy#
opensips -V<br>
version: opensips 1.11.2-notls (x86_64/linux)
<div><br>
flags: STATS: On, USE_IPV6, USE_TCP,
DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP,
PKG_MALLOC, DBG_QM_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT<br>
ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535<br>
poll method support: poll, epoll_lt,
epoll_et, sigio_rt, select.<br>
</div>
git revision: d41eb41<br>
main.c compiled on 19:23:35 Jul 23 2014 with
gcc 4.7</blockquote>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Here
are an extract of logs we get at the crash-time
(I masked phone numbers and IPs for security
purposes) :</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Aug
12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:parse_contacts: failed to parse
contact parameters<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:contact_parser: failed to parse
contacts<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:parse_contact: failed to parse
contact<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:pv_get_contact_body: failed to
parse contact hdr<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]: 90tzz16g@pbx
-- New REGISTER received from
188.94.XXX.XXX with Contact : -- IP
188.94.XXX.XXX -- PORT 5060<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]: 90tzz16g@pbx
-- REGISTER Failed because of : No Credentials
-- From 188.94.XXX.XXX<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:build_res_buf_from_sip_req: out of
pkg memory; needs 526<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:sl:sl_send_reply_helper: response
building failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:signaling:sig_send_reply_mod: failed to
send reply with sl module<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:auth:challenge: failed to send the
response<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25606]: iy4l7y1r@pbx
-- New REGISTER received from
188.94.XXX.XXX with Contact : <a href="mailto:sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0" target="_blank"><sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0></a>
-- IP 188.94.XXX.XXX -- PORT 5060<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25606]: iy4l7y1r@pbx
-- REGISTER Failed because of : No Credentials
-- From 188.94.XXX.XXX<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]: iy4l7y1r@pbx
-- New REGISTER received from
188.94.XXX.XXX with Contact : <a href="mailto:sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0" target="_blank"><sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0></a>
-- IP 188.94.XXX.XXX -- PORT 5060<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:core:db_allocate_rows: no memory left<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:db_mysql:db_mysql_convert_rows: no
private memory left<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:db_mysql:db_mysql_convert_result: error
while converting rows<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:db_mysql:db_mysql_store_result: error
while converting result<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:core:db_do_query: error while storing
result for query [select
contact,expires,q,callid,cseq,flags,cflags,user_agent,received,path,socket,methods,last_modified,sip_instance,attr
from location where username='0326xxxxxx'
order by q]<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:usrloc:db_load_urecord: db_query failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute:
driver error (1062): Duplicate entry '<a href="mailto:0326xxxxxx--sip:0326xxxxxx@188.94.XX.XXX:5060;transport=udp;line" target="_blank">0326xxxxxx--sip:0326xxxxxx@188.94.XX.XXX:5060;transport=udp;line</a>'
for key 'account_contact_idx'<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:usrloc:db_insert_ucontact: inserting
contact in db failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:usrloc:insert_ucontact: failed to insert
in database<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:core:build_res_buf_from_sip_req: out of
pkg memory; needs 373<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:sl:sl_send_reply_helper: response
building failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:signaling:sig_send_reply_mod: failed to
send reply with sl module<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:options:opt_reply: failed to send 200
via send_reply</blockquote>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">Unfortunately,
we don't have the memory dump :-s ... </div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">So,
I don't know if there's something we can try to
avoid those errors to happen again; maybe we
made something wrong in our configuration file
? </div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">We
could provide you anything you want to help us
find where's the problem.</div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">Thanks
a lot for your help, </div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div class="gmail_extra" style="font-family:arial,sans-serif;font-size:13px">
Kevin</div>
<div class="gmail_extra"><br clear="all">
<div><b>
<div><span style="font-weight:normal">Bien
cordialement, </span></div>
<div><span style="font-weight:normal">Best
Regards, </span></div>
<div><span style="font-weight:normal"><br>
</span></div>
</b><b>Kevin MATHY</b> |<b> </b>Ingénieur VoIP<br>
<div>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2014-07-08 8:43
GMT+02:00 Kevin Mathy <span dir="ltr"><<a href="mailto:k.mathy@hexanet.fr" target="_blank">k.mathy@hexanet.fr</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Bogdan,
<div><br>
</div>
<div>Sounds really good :-) ! I'm going to
check if there's no configuration change
between 1.9's and 1.11's modules
version, and then will upgrade our
version ! </div>
<div><br>
</div>
<div>I hope this will correct our problem
;-)</div>
<div>
<div><br>
</div>
<div>Thanks for your help, </div>
<div><br>
</div>
<div>Kevin</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra">
<div><br clear="all">
<div> <b>
<div><span style="font-weight:normal">Bien
cordialement, </span></div>
<div><span style="font-weight:normal">Best
Regards, </span></div>
<div><span style="font-weight:normal"><br>
</span></div>
</b><b>Kevin MATHY</b> |<b> </b>Ingénieur
VoIP<br>
<div>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
</div>
<div class="gmail_quote">2014-07-07
18:40 GMT+02:00 Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>:
<div>
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div><tt>Hi Kevin,</tt><tt><br>
</tt><tt><br>
</tt><tt>OK, found the
problem. As expected, this
was fixed in 1.11, but not
ported to 1.9 as this
version is not maintained
any </tt><tt>more. Here
is the link to the fix for
your crash: <br>
</tt><tt><br>
</tt><tt><a href="https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75" target="_blank">https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75</a></tt>
<div> <tt><br>
<br>
Regards,<br>
</tt>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
</div>
</div>
<img></blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
<br>
</div></div><img></blockquote>
<br>
</div>
</blockquote></div><br></div>
<br>
<img src="http://www.hexanet.fr/sites/files/hexanet/files/utilities/signature_hexanet.gif">