<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Denis,<br>
<br>
Could you put in an archive the core file and corresponding
opensips binary and modules - and have them available for download
somewhere ? and I try to look into on my side.<br>
<br>
Regards,<br>
</tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a class="moz-txt-link-freetext" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div class="moz-cite-prefix">On 04/12/2017 05:11 PM, Denis wrote:<br>
</div>
<blockquote cite="mid:2741321492006278@web30o.yandex.ru" type="cite">
<div>Hello, Bogdan!</div>
<div> </div>
<div>Is there any additional information that i should gather
about problem?</div>
<div>Or everything is clear?</div>
<div> </div>
<div>Thank you.</div>
<div> </div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div>06.04.2017, 15:18, "Denis via Users"
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org"><users@lists.opensips.org></a>:</div>
<blockquote type="cite">
<div>Bogdan, I copied tar to the backup server and restore
opensips from tar there (on the same path as in main was).</div>
<div>Now, when entering to the gdb on backup, it does not show
any warning messages, but the result is the same</div>
<div> </div>
<div>
<div>f 10</div>
<div>#0 0x<span>0000000000000000</span> in ?? ()</div>
<div>(gdb) p p_cell->method.s</div>
<div>No symbol "p_cell" in current context.</div>
<div>(gdb) p p_cell->uas.request</div>
<div>No symbol "p_cell" in current context.</div>
<div>(gdb) p p_cell->uas.status</div>
<div>No symbol "p_cell" in current context.</div>
<div> </div>
</div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>06.04.2017, 14:38, "Bogdan-Andrei Iancu" <<a
moz-do-not-send="true" href="mailto:bogdan@opensips.org"><a class="moz-txt-link-abbreviated" href="mailto:bogdan@opensips.org">bogdan@opensips.org</a></a>>:</div>
<blockquote type="cite">
<div bgcolor="#FFFFFF">Oh, that is good !<br>
<br>
When starting gdb, point to the old opensips binary. To
solve the problem with the modules (which are libraries),
see:<br>
<a moz-do-not-send="true"
href="http://stackoverflow.com/questions/84341/how-do-i-prepend-a-directory-the-library-path-when-loading-a-core-file-in-gdb-on">http://stackoverflow.com/questions/84341/how-do-i-prepend-a-directory-the-library-path-when-loading-a-core-file-in-gdb-on</a><br>
(the first answer)<br>
<br>
Regards,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/06/2017 12:31 PM, Denis wrote:</div>
<blockquote cite="mid:3045121491471102@web26o.yandex.ru"
type="cite">
<div>Before making upgrade i made tar archive of the
directory where opensips has been installed. Can this
help us?</div>
<div>It was crash on time only. I do not think, that o can
reproduce it.</div>
<div> </div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>06.04.2017, 12:27, "Bogdan-Andrei Iancu" <a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a></a>:</div>
<blockquote type="cite">
<div bgcolor="#FFFFFF">yeah, the opensips binary and
module libs do not match the corefile anymore
:(.....Do you have the old build ? Also, was this
crash on time only ? or can you reproduce it ?<br>
<br>
Regards,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/06/2017 12:17 PM, Denis wrote:</div>
<blockquote
cite="mid:2941591491470223@web26o.yandex.ru"
type="cite">
<div>Hello, Bogdan.</div>
<div> </div>
<div>Probably i made a big mistake.</div>
<div>I have already updated Opensips from latest git
(to solve another problem)</div>
<div> </div>
<div>So, when i enter in gdb i get such warning
message</div>
<div>
<div>
<div> </div>
<div>[New LWP 14909]</div>
<div>warning: .dynamic section for
"/usr/local/opensips2.2/lib64/opensips/modules/dialog.so"
is not at the expected address (wrong library
or version mismatch?)</div>
</div>
</div>
<div> </div>
<div>Anyway,</div>
<div> </div>
<div>
<div>f 10</div>
<div>#10 0x00007fdf6c4ac158 in ?? ()</div>
<div>
<div> </div>
<div>p p_cell->method.s</div>
<div>No symbol "p_cell" in current context.</div>
<div>
<div> </div>
<div>p p_cell->uas.request</div>
<div>No symbol "p_cell" in current context.</div>
<div> </div>
<div>
<div>p p_cell->uas.status</div>
<div>No symbol "p_cell" in current context.</div>
</div>
</div>
</div>
</div>
<div> </div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>06.04.2017, 11:08, "Bogdan-Andrei Iancu" <a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a></a>:</div>
<blockquote type="cite">
<div bgcolor="#FFFFFF">Hi Denis,<br>
<br>
Please get some more info from the corefile :<br>
<br>
f 10<br>
p p_cell->method.s<br>
p p_cell->uas.request<br>
p p_cell->uas.status<br>
<br>
Thanks,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/05/2017 11:56 AM, Denis wrote:</div>
<blockquote
cite="mid:5931331491382613@web27o.yandex.ru"
type="cite">
<div>Ok, Bogdan, I'll be waiting any result.</div>
<div> </div>
<div>Thank you.</div>
<div> </div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>05.04.2017, 11:53, "Bogdan-Andrei Iancu"
<a moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div>
<blockquote type="cite">
<div bgcolor="#FFFFFF">Hi Denis,<br>
<br>
Many thanks for the output - this confirm
a suspicious of mine in regards to the
cause of the crash - let me try to see if
we can reproduce it on our side or if we
need more info from your side.<br>
Please do not delete the corefile for the
moment :)<br>
<br>
Best regards,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/05/2017 11:03 AM, Denis wrote:</div>
<blockquote
cite="mid:5350841491379420@web18o.yandex.ru"
type="cite">
<div>Hello, Bogdan!</div>
<div> </div>
<div>
<div>f 5</div>
<div>#5 0x00007fdf6844c4f2 in
free_dlg_dlg (dlg=0x7fdf6c4ac158) at
dlg_hash.c:183</div>
<div>183
destroy_dlg_callbacks_list(dlg->cbs.first);</div>
<div> </div>
<div>
<div>p *dlg->cbs.first</div>
<div>$1 = {types = 96, callback =
0x7fdf<span><span><span><span><span>67531300</span></span></span></span></span>
<acc_dlg_callback>, param =
0x0, callback_param_free =
0x7fdf675318fe
<dlg_free_acc_mask>, next =
0x7fdf<span><span><span><span><span>84191080</span></span></span></span></span>}</div>
<div> </div>
<div>
<div>p *dlg->cbs.first->next</div>
<div>$2 = {types = 96, callback =
0x7fdf<span><span><span><span><span>67531300</span></span></span></span></span>
<acc_dlg_callback>, param
= 0x7fdf83c9f340,
callback_param_free =
0x7fdf675318fe
<dlg_free_acc_mask>, next
= 0x0}</div>
<div> </div>
<div>
<div>p
*dlg->cbs.first->next->next</div>
<div>Cannot access memory at
address 0x0</div>
</div>
</div>
</div>
</div>
<div> </div>
<div>
<div>p
*dlg->cbs.first->next->next->next</div>
<div>Cannot access memory at address
0x20</div>
<div> </div>
<div>and so on with ....at address
0x20</div>
<div> </div>
</div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>04.04.2017, 18:43, "Bogdan-Andrei
Iancu" <a moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div>
<blockquote type="cite">
<div bgcolor="#FFFFFF">Super :)....we
got to something useful.<br>
<br>
Now, again in GDB please do:<br>
<br>
f 5<br>
p *dlg->cbs.first<br>
p *dlg->cbs.first->next<br>
p
*dlg->cbs.first->next->next<br>
<br>
and keep adding "->next" until
you get error from gdb.<br>
<br>
Thanks,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/04/2017 06:07 PM, Денис
Путято wrote:</div>
<blockquote
cite="mid:3037001491318426@web47g.yandex.ru"
type="cite">
<div>Sorry, Bogdan, i missed that</div>
<div> </div>
<div>I found such CRITICAL message</div>
<div>"CRITICAL:core:qm_free:
freeing already freed pointer,
first free: acc_logic.c:
dlg_free_acc_mask(211) -
aborting"</div>
<div> </div>
<div>And after that and till crash
only
"WARNING:core:utimer_ticker:...."
reports in syslog.</div>
<div> </div>
<div>
<div>(gdb)</div>
<div>f 2</div>
<div>#2 0x<span><span><span><span><span><span>0000000000515592</span></span></span></span></span></span>
in qm_free (qm=<optimized
out>,
p=p@entry=0x7fdf83c9f340,
file=file@entry=0x7fdf6753ad62
"acc_logic.c",</div>
<div>
func=func@entry=0x7fdf6753af90
<__FUNCTION__.12529>
"dlg_free_acc_mask",
line=line@entry=211) at
mem/q_malloc.c:483</div>
<div>483
abort();</div>
<div> </div>
<div> </div>
<div>
<div>p *((struct qm_frag*)
((char*)p-sizeof(struct
qm_frag)))</div>
<div>$1 = {size = 8, u =
{nxt_free = 0x7fdf71a8c6b0,
is_free = <span><span><span><span><span><span>140597661320880</span></span></span></span></span></span>},
file = 0x7fdf6753ad62
"acc_logic.c", func =
0x7fdf6753af90
<__FUNCTION__.12529>
"dlg_free_acc_mask", line =
211,</div>
<div> check = <span><span><span><span><span><span>17361641481138401520</span></span></span></span></span></span>}</div>
</div>
</div>
<div> </div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>04.04.2017, 17:58,
"Bogdan-Andrei Iancu" <a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a></a>:</div>
<blockquote type="cite">
<div bgcolor="#FFFFFF">The
RADIUS part is not relevant.<br>
<br>
Still you didn;t answer to my
question on the CRITICAL error
- according to the backtrace,
you should have a CRITICAL
message in the opensips log,
just before the crash.<br>
<br>
In GDB, do:<br>
f 2<br>
p *((struct qm_frag*)
((char*)p-sizeof(struct
qm_frag))) <br>
<br>
Regards,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/04/2017 05:55 PM,
Denis wrote:</div>
<blockquote
cite="mid:2993391491317739@web47g.yandex.ru"
type="cite">
<div>During crash there was
another problem with LAN
network.</div>
<div>Because of this problem
Opensips could not connect
to radius server.</div>
<div>
<div>rc_ip_hostname:
couldn't look up host by
addr: xxxxxx</div>
<div>rc_send_server: no
reply from RADIUS server
unknown:1812</div>
</div>
<div> </div>
<div>No more critical
messages.</div>
<div> </div>
<div>Bogdan, i am sorry, but</div>
<div>"in frame 2". What does
it mean?</div>
<div> </div>
<div> </div>
<div>-- </div>
<div>С уважением, Денис.</div>
<div>Best regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>04.04.2017, 16:35,
"Bogdan-Andrei Iancu" <a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a></a>:</div>
<blockquote type="cite">
<div bgcolor="#FFFFFF">Any
CRITICAL message into
the log before the crash
?<br>
<br>
also, in gdb in frame 2,
please print:<br>
p *((struct
qm_frag*)
((char*)p-sizeof(struct
qm_frag)))<br>
<br>
<br>
Thanks,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/04/2017 03:19
PM, Denis wrote:</div>
<blockquote
cite="mid:2219671491308376@web4m.yandex.ru"
type="cite">
<div>
<div>p
shm_block->first_frag</div>
<div>$1 = (struct
qm_frag *)
0x7fdf6b9d44f8</div>
<div>
<div> </div>
<div>p
shm_block->last_frag_end</div>
<div>$2 = (struct
qm_frag_end *)
0x7fe02899efd0</div>
<div> </div>
<div>
<div> p
mem_block->first_frag</div>
<div>$3 =
(struct
qm_frag *)
0x7fe<span><span><span><span><span><span><span><span>029214508</span></span></span></span></span></span></span></span></div>
<div> </div>
<div>
<div>p
mem_block->last_frag_end</div>
<div>$4 =
(struct
qm_frag_end *)
0x7fe0691defe0</div>
</div>
</div>
</div>
<div> </div>
</div>
<div> </div>
<div>-- </div>
<div>С уважением,
Денис.</div>
<div>Best regards,
Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>04.04.2017,
14:51,
"Bogdan-Andrei
Iancu" <a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a></a>:</div>
<blockquote
type="cite">
<div
bgcolor="#FFFFFF">OK,please
enter gdb and
print:<br>
p
shm_block->first_frag<br>
p
shm_block->last_frag_end<br>
<br>
p
mem_block->first_frag<br>
p
mem_block->last_frag_end<br>
<br>
Thanks,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On 04/04/2017
01:42 PM, Denis
wrote:</div>
<blockquote
cite="mid:1579141491302539@web17o.yandex.ru"
type="cite">
<div>Hello,
Bogdan!</div>
<div> </div>
<div>No, i
cannot see in
the log any
messages you
mentioned it
your letter.</div>
<div>The
situation
dealing with
crash had such
stage.</div>
<div> </div>
<div>1. I
prepared to
migrate to
2.2.3 from
2.2.2. But do
not have a
time to
restart
Opensips</div>
<div>2. As a
result "i
waited" when
Opensips have
crashed.
During crash
of 2.2.2 no
core file has
been
generated, but
Opensips has
been restarted
using cron.</div>
<div>3. After 2)
Opensips
became 2.2.3.
It was working
about 1 minute
and crashed
again, but in
that time,
core file has
been generated
and i attached
information
from it.</div>
<div>So, between
2) and 3) , as
i mentioned
above, there
were no
messages about
memory, but i
see many
messages like</div>
<div>'WARNING:core:utimer_ticker:
utimer task
<tm-utimer>
already
scheduled for
71500 ms</div>
<div>WARNING:core:timer_ticker:
timer task
<pike-clean>
already
scheduled for
72400 ms</div>
<div>WARNING:core:timer_ticker:
timer task
<nh-timer>
already
scheduled for
72400 ms</div>
<div>WARNING:core:timer_ticker:
timer task
<dlg-timer>
already
scheduled for
72400 ms</div>
<div>WARNING:core:timer_ticker:
timer task
<blcore-expire>
already
scheduled for
72400 ms'</div>
<div>and so on</div>
<div> </div>
<div>As for
about version</div>
<div> </div>
<div>
<div>version:
opensips 2.2.3
(x86_64/linux)</div>
<div>flags:
STATS: On,
DISABLE_NAGLE,
USE_MCAST,
SHM_MMAP,
PKG_MALLOC,
QM_MALLOC,
DBG_MALLOC,
FAST_LOCK-ADAPTIVE_WAIT</div>
<div>ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE
65535</div>
<div>poll
method
support: poll,
epoll_lt,
epoll_et,
sigio_rt,
select.</div>
<div>git
revision: <span><span><span><span><span><span><span><span><span>7922417</span></span></span></span></span></span></span></span></span></div>
<div>main.c
compiled on
06:29:28 Mar
23 2017 with
gcc 5.4.0</div>
<div> </div>
</div>
<div>-- </div>
<div>С
уважением,
Денис.</div>
<div>Best
regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<div>04.04.2017,
12:43,
"Bogdan-Andrei
Iancu" <a
moz-do-not-send="true"
href="mailto:bogdan@opensips.org"><a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a></a>:</div>
<blockquote
type="cite">
<div
bgcolor="#FFFFFF">Hello
Denis,<br>
<br>
Thank you for
the report. Do
you see any
CRITICAL log
just before
the crash,
something
like:<br>
"bad
pointer
0xxxxxxxx (out
of memory
block!) -
aborting"<br>
?<br>
If yes, please
post here the
exact line.<br>
<br>
Also, do you
use the latest
OpenSIPS 2.2
from GIT repo
? (run
"opensips -v")<br>
<br>
Regards,
<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a moz-do-not-send="true" href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a moz-do-not-send="true" href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre>
<div>On
04/04/2017
10:58 AM,
Denis via
Users wrote:</div>
<blockquote
cite="mid:325131491292730@web16g.yandex.ru"
type="cite">
<div>Hello!</div>
<div> </div>
<div>Server::
OpenSIPS
(2.2.3
(x86_64/linux))</div>
<div> </div>
<div>Thank you
for any help.</div>
<div> </div>
<div>P.S.
Opensips has
been compiled
with QM_MALLOC
and DBG_MALLOC
flags</div>
<div> </div>
<div>-- </div>
<div>С
уважением,
Денис.</div>
<div>Best
regards, Denis</div>
<div> </div>
<div> </div>
<div> </div>
<pre>_______________________________________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a moz-do-not-send="true" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
,
<p>_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a moz-do-not-send="true"
href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a></p>
</blockquote>
</blockquote>
<br>
</body>
</html>