<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix"><tt>Hi Kevin,<br>
<br>
Any chance to get the mem dump after "waiting" ?<br>
<br>
Thanks & Regards,<br>
</tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a></pre>
On 04.09.2014 13:09, Kevin Mathy wrote:<br>
</div>
<blockquote
cite="mid:CAOmzhXkMkKFxgGR2Ayy_Fsc9ER1g575Zn-idcKoz-Gbipcf5sQ@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Bogdan,
<div><br>
</div>
<div>So, I've faced the problem once again, and followed the
"Determining the cause" part of the link you gave me. </div>
<div>I've waited more than 20 minutes (near an hour), and
re-sent some voice trafic to the problematic opensips server :
memory errors just came after that. </div>
<div><br>
</div>
<div>So, I think it's a memory leak, no ? Now, what can I do to
solve this ? </div>
<div><br>
</div>
<div>Thanks for your help, </div>
<div><br>
</div>
<div>Kevin</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra">
<br clear="all">
<div><b>
<div><span style="font-weight:normal">Bien cordialement, </span></div>
<div><span style="font-weight:normal">Best Regards, </span></div>
<div><span style="font-weight:normal"><br>
</span></div>
</b><b>Kevin MATHY</b> |<b> </b>Ingénieur VoIP<br>
<div>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2014-08-20 17:22 GMT+02:00
Bogdan-Andrei Iancu <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:bogdan@opensips.org"
target="_blank">bogdan@opensips.org</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><tt>Hi Kevin,<br>
<br>
So it is more a mem leak than a crash. Any chances to
get a memdump after memory is exhausted ? See:<br>
<a moz-do-not-send="true"
href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem"
target="_blank">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a><br>
<br>
Regards,<br>
</tt>
<div class="">
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
</div>
<div>
<div class="h5"> On 20.08.2014 15:35, Kevin Mathy
wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="h5">
<div dir="ltr">Hi Bogdan,
<div><br>
</div>
<div>In fact, opensips didn't crashed : we got the
previous error messages in the logs, but then we
switched our voice traffic to another server,
and opensips on the 1st one never crashed... We
restarted the service in the following night, to
be sure to make it rework well.</div>
<div><br>
</div>
<div>So, is there a way in this case to get some
backtrace, even if it didn't crashed ? Maybe
something generated at startup ?</div>
<div><br>
</div>
<div>Thanks for your help, </div>
<div><br>
</div>
<div>Kevin</div>
</div>
<div class="gmail_extra"><br clear="all">
<div><b>
<div><span style="font-weight:normal">Bien
cordialement, </span></div>
<div><span style="font-weight:normal">Best
Regards, </span></div>
<div><span style="font-weight:normal"><br>
</span></div>
</b><b>Kevin MATHY</b> |<b> </b>Ingénieur VoIP<br>
<div>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2014-08-19 20:34
GMT+02:00 Bogdan-Andrei Iancu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"
target="_blank">bogdan@opensips.org</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><tt>Hi Kevin,<br>
<br>
Try to extract at least a backtrace, to
see what kind of problem we are facing.
Then we can decide if a mem dump is
required or not.<br>
<br>
Thanks,<br>
</tt>
<div>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
</div>
<div>
<div> On 19.08.2014 12:03, Kevin Mathy
wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr"><span
style="font-family:arial,sans-serif;font-size:13px">Hi
Bogdan,</span>
<div
style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div
style="font-family:arial,sans-serif;font-size:13px">Coming
back to you after some holidays,
and unfortunately, new Opensips
crashes... </div>
<div
style="font-family:arial,sans-serif;font-size:13px">Some
weeks ago, we upgraded our servers
to OpenSIPS 1.11, but we faced a
new crash, always with the "out of
pkg memory" error, last week.</div>
<div
style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div
style="font-family:arial,sans-serif;font-size:13px">Here
is the exact version :</div>
<div
style="font-family:arial,sans-serif;font-size:13px">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">root@asbc2:/home/kemathy#
opensips -V<br>
version: opensips 1.11.2-notls
(x86_64/linux)
<div><br>
flags: STATS: On, USE_IPV6,
USE_TCP, DISABLE_NAGLE,
USE_MCAST, SHM_MEM, SHM_MMAP,
PKG_MALLOC, DBG_QM_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT<br>
ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE
1024, BUF_SIZE 65535<br>
poll method support: poll,
epoll_lt, epoll_et, sigio_rt,
select.<br>
</div>
git revision: d41eb41<br>
main.c compiled on 19:23:35 Jul
23 2014 with gcc 4.7</blockquote>
</div>
<div
style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div
style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div
style="font-family:arial,sans-serif;font-size:13px">Here
are an extract of logs we get at
the crash-time (I masked phone
numbers and IPs for security
purposes) :</div>
<div
style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div
style="font-family:arial,sans-serif;font-size:13px">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Aug
12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:parse_contacts:
failed to parse contact
parameters<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:contact_parser:
failed to parse contacts<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:parse_contact: failed
to parse contact<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:pv_get_contact_body:
failed to parse contact hdr<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
90tzz16g@pbx -- New REGISTER
received from
188.94.XXX.XXX with Contact :
-- IP 188.94.XXX.XXX -- PORT
5060<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
90tzz16g@pbx -- REGISTER Failed
because of : No Credentials --
From 188.94.XXX.XXX<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:core:build_res_buf_from_sip_req:
out of pkg memory; needs 526<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:sl:sl_send_reply_helper:
response building failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:signaling:sig_send_reply_mod:
failed to send reply with sl
module<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25622]:
ERROR:auth:challenge: failed to
send the response<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25606]:
iy4l7y1r@pbx -- New REGISTER
received from
188.94.XXX.XXX with Contact : <a
moz-do-not-send="true"
href="mailto:sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0"
target="_blank"><sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0></a>
-- IP 188.94.XXX.XXX -- PORT
5060<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25606]:
iy4l7y1r@pbx -- REGISTER Failed
because of : No Credentials --
From 188.94.XXX.XXX<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
iy4l7y1r@pbx -- New REGISTER
received from
188.94.XXX.XXX with Contact : <a
moz-do-not-send="true"
href="mailto:sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0"
target="_blank"><sip:0326xxxxxx@188.94.XXX.XXX:5060;transport=udp;line=6364d3f0></a>
-- IP 188.94.XXX.XXX -- PORT
5060<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:core:db_allocate_rows: no
memory left<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:db_mysql:db_mysql_convert_rows:
no private memory left<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:db_mysql:db_mysql_convert_result:
error while converting rows<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:db_mysql:db_mysql_store_result:
error while converting result<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:core:db_do_query: error
while storing result for query
[select
contact,expires,q,callid,cseq,flags,cflags,user_agent,received,path,socket,methods,last_modified,sip_instance,attr
from location where
username='0326xxxxxx' order by
q]<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:usrloc:db_load_urecord:
db_query failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute:
driver error (1062): Duplicate
entry '<a moz-do-not-send="true"
href="mailto:0326xxxxxx--sip:0326xxxxxx@188.94.XX.XXX:5060;transport=udp;line"
target="_blank">0326xxxxxx--sip:0326xxxxxx@188.94.XX.XXX:5060;transport=udp;line</a>'
for key 'account_contact_idx'<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:usrloc:db_insert_ucontact:
inserting contact in db failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25614]:
ERROR:usrloc:insert_ucontact:
failed to insert in database<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:core:build_res_buf_from_sip_req:
out of pkg memory; needs 373<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:sl:sl_send_reply_helper:
response building failed<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:signaling:sig_send_reply_mod:
failed to send reply with sl
module<br>
Aug 12 11:48:34 asbc2
/usr/local/sbin/opensips[25620]:
ERROR:options:opt_reply: failed
to send 200 via send_reply</blockquote>
</div>
<div
style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px">Unfortunately,
we don't have the memory dump :-s
... </div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px">So,
I don't know if there's something
we can try to avoid those errors
to happen again; maybe we made
something wrong in our
configuration file ? </div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px">We
could provide you anything you
want to help us find where's the
problem.</div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px">Thanks
a lot for your help, </div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div class="gmail_extra"
style="font-family:arial,sans-serif;font-size:13px">
Kevin</div>
<div class="gmail_extra"><br
clear="all">
<div><b>
<div><span
style="font-weight:normal">Bien
cordialement, </span></div>
<div><span
style="font-weight:normal">Best
Regards, </span></div>
<div><span
style="font-weight:normal"><br>
</span></div>
</b><b>Kevin MATHY</b> |<b> </b>Ingénieur
VoIP<br>
<div>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">2014-07-08
8:43 GMT+02:00 Kevin Mathy <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:k.mathy@hexanet.fr"
target="_blank">k.mathy@hexanet.fr</a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Hi Bogdan,
<div><br>
</div>
<div>Sounds really good :-)
! I'm going to check if
there's no configuration
change between 1.9's and
1.11's modules version,
and then will upgrade our
version ! </div>
<div><br>
</div>
<div>I hope this will
correct our problem ;-)</div>
<div>
<div><br>
</div>
<div>Thanks for your
help, </div>
<div><br>
</div>
<div>Kevin</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra">
<div><br clear="all">
<div> <b>
<div><span
style="font-weight:normal">Bien
cordialement, </span></div>
<div><span
style="font-weight:normal">Best
Regards, </span></div>
<div><span
style="font-weight:normal"><br>
</span></div>
</b><b>Kevin MATHY</b>
|<b> </b>Ingénieur
VoIP<br>
<div>
<div><b><br>
</b></div>
</div>
</div>
<br>
<br>
</div>
<div class="gmail_quote">2014-07-07
18:40 GMT+02:00
Bogdan-Andrei Iancu <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>:
<div>
<div><br>
<blockquote
class="gmail_quote"
style="margin:0 0
0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<div><tt>Hi
Kevin,</tt><tt><br>
</tt><tt><br>
</tt><tt>OK,
found the
problem. As
expected, this
was fixed in
1.11, but not
ported to 1.9
as this
version is not
maintained any
</tt><tt>more.
Here is the
link to the
fix for your
crash: <br>
</tt><tt><br>
</tt><tt><a
moz-do-not-send="true"
href="https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75"
target="_blank">https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75</a></tt>
<div> <tt><br>
<br>
Regards,<br>
</tt>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
</div>
</div>
<img moz-do-not-send="true"></blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
<br>
</div>
</div>
<img moz-do-not-send="true"></blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<img moz-do-not-send="true"
src="http://www.hexanet.fr/sites/files/hexanet/files/utilities/signature_hexanet.gif"></blockquote>
<br>
</body>
</html>