<div>Hello, Bogdan!</div><div> </div><div>Is there any additional information that i should gather about problem?</div><div>Or everything is clear?</div><div> </div><div>Thank you.</div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div> </div><div> </div><div>06.04.2017, 15:18, "Denis via Users" <users@lists.opensips.org>:</div><blockquote type="cite"><div>Bogdan, I copied tar to the backup server and restore opensips from tar there (on the same path as in main was).</div><div>Now, when entering to the gdb on backup, it does not show any warning messages, but the result is the same</div><div> </div><div><div>f 10</div><div>#0 0x<span>0000000000000000</span> in ?? ()</div><div>(gdb) p p_cell->method.s</div><div>No symbol "p_cell" in current context.</div><div>(gdb) p p_cell->uas.request</div><div>No symbol "p_cell" in current context.</div><div>(gdb) p p_cell->uas.status</div><div>No symbol "p_cell" in current context.</div><div> </div></div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>06.04.2017, 14:38, "Bogdan-Andrei Iancu" <<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">Oh, that is good !<br /><br />When starting gdb, point to the old opensips binary. To solve the problem with the modules (which are libraries), see:<br /> <a href="http://stackoverflow.com/questions/84341/how-do-i-prepend-a-directory-the-library-path-when-loading-a-core-file-in-gdb-on">http://stackoverflow.com/questions/84341/how-do-i-prepend-a-directory-the-library-path-when-loading-a-core-file-in-gdb-on</a><br />(the first answer)<br /><br />Regards,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/06/2017 12:31 PM, Denis wrote:</div><blockquote cite="mid:3045121491471102@web26o.yandex.ru" type="cite"><div>Before making upgrade i made tar archive of the directory where opensips has been installed. Can this help us?</div><div>It was crash on time only. I do not think, that o can reproduce it.</div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>06.04.2017, 12:27, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">yeah, the opensips binary and module libs do not match the corefile anymore :(.....Do you have the old build ? Also, was this crash on time only ? or can you reproduce it ?<br /><br />Regards,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/06/2017 12:17 PM, Denis wrote:</div><blockquote cite="mid:2941591491470223@web26o.yandex.ru" type="cite"><div>Hello, Bogdan.</div><div> </div><div>Probably i made a big mistake.</div><div>I have already updated Opensips from latest git (to solve another problem)</div><div> </div><div>So, when i enter in gdb i get such warning message</div><div><div><div> </div><div>[New LWP 14909]</div><div>warning: .dynamic section for "/usr/local/opensips2.2/lib64/opensips/modules/dialog.so" is not at the expected address (wrong library or version mismatch?)</div></div></div><div> </div><div>Anyway,</div><div> </div><div><div>f 10</div><div>#10 0x00007fdf6c4ac158 in ?? ()</div><div><div> </div><div>p p_cell->method.s</div><div>No symbol "p_cell" in current context.</div><div><div> </div><div>p p_cell->uas.request</div><div>No symbol "p_cell" in current context.</div><div> </div><div><div>p p_cell->uas.status</div><div>No symbol "p_cell" in current context.</div></div></div></div></div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>06.04.2017, 11:08, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">Hi Denis,<br /><br />Please get some more info from the corefile :<br /><br /> f 10<br /> p p_cell->method.s<br /> p p_cell->uas.request<br /> p p_cell->uas.status<br /><br />Thanks,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/05/2017 11:56 AM, Denis wrote:</div><blockquote cite="mid:5931331491382613@web27o.yandex.ru" type="cite"><div>Ok, Bogdan, I'll be waiting any result.</div><div> </div><div>Thank you.</div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>05.04.2017, 11:53, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">Hi Denis,<br /><br />Many thanks for the output - this confirm a suspicious of mine in regards to the cause of the crash - let me try to see if we can reproduce it on our side or if we need more info from your side.<br />Please do not delete the corefile for the moment :)<br /><br />Best regards,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/05/2017 11:03 AM, Denis wrote:</div><blockquote cite="mid:5350841491379420@web18o.yandex.ru" type="cite"><div>Hello, Bogdan!</div><div> </div><div><div>f 5</div><div>#5 0x00007fdf6844c4f2 in free_dlg_dlg (dlg=0x7fdf6c4ac158) at dlg_hash.c:183</div><div>183 destroy_dlg_callbacks_list(dlg->cbs.first);</div><div> </div><div><div>p *dlg->cbs.first</div><div>$1 = {types = 96, callback = 0x7fdf<span><span><span><span><span>67531300</span></span></span></span></span> <acc_dlg_callback>, param = 0x0, callback_param_free = 0x7fdf675318fe <dlg_free_acc_mask>, next = 0x7fdf<span><span><span><span><span>84191080</span></span></span></span></span>}</div><div> </div><div><div>p *dlg->cbs.first->next</div><div>$2 = {types = 96, callback = 0x7fdf<span><span><span><span><span>67531300</span></span></span></span></span> <acc_dlg_callback>, param = 0x7fdf83c9f340, callback_param_free = 0x7fdf675318fe <dlg_free_acc_mask>, next = 0x0}</div><div> </div><div><div>p *dlg->cbs.first->next->next</div><div>Cannot access memory at address 0x0</div></div></div></div></div><div> </div><div><div>p *dlg->cbs.first->next->next->next</div><div>Cannot access memory at address 0x20</div><div> </div><div>and so on with ....at address 0x20</div><div> </div></div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>04.04.2017, 18:43, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">Super :)....we got to something useful.<br /><br />Now, again in GDB please do:<br /><br />f 5<br />p *dlg->cbs.first<br />p *dlg->cbs.first->next<br />p *dlg->cbs.first->next->next<br /><br />and keep adding "->next" until you get error from gdb.<br /><br />Thanks,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/04/2017 06:07 PM, Денис Путято wrote:</div><blockquote cite="mid:3037001491318426@web47g.yandex.ru" type="cite"><div>Sorry, Bogdan, i missed that</div><div> </div><div>I found such CRITICAL message</div><div>"CRITICAL:core:qm_free: freeing already freed pointer, first free: acc_logic.c: dlg_free_acc_mask(211) - aborting"</div><div> </div><div>And after that and till crash only "WARNING:core:utimer_ticker:...." reports in syslog.</div><div> </div><div><div>(gdb)</div><div>f 2</div><div>#2 0x<span><span><span><span><span><span>0000000000515592</span></span></span></span></span></span> in qm_free (qm=<optimized out>, p=p@entry=0x7fdf83c9f340, file=file@entry=0x7fdf6753ad62 "acc_logic.c",</div><div> func=func@entry=0x7fdf6753af90 <__FUNCTION__.12529> "dlg_free_acc_mask", line=line@entry=211) at mem/q_malloc.c:483</div><div>483 abort();</div><div> </div><div> </div><div><div>p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))</div><div>$1 = {size = 8, u = {nxt_free = 0x7fdf71a8c6b0, is_free = <span><span><span><span><span><span>140597661320880</span></span></span></span></span></span>}, file = 0x7fdf6753ad62 "acc_logic.c", func = 0x7fdf6753af90 <__FUNCTION__.12529> "dlg_free_acc_mask", line = 211,</div><div> check = <span><span><span><span><span><span>17361641481138401520</span></span></span></span></span></span>}</div></div></div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>04.04.2017, 17:58, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">The RADIUS part is not relevant.<br /><br />Still you didn;t answer to my question on the CRITICAL error - according to the backtrace, you should have a CRITICAL message in the opensips log, just before the crash.<br /><br />In GDB, do:<br /> f 2<br /> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag))) <br /><br />Regards,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/04/2017 05:55 PM, Denis wrote:</div><blockquote cite="mid:2993391491317739@web47g.yandex.ru" type="cite"><div>During crash there was another problem with LAN network.</div><div>Because of this problem Opensips could not connect to radius server.</div><div><div>rc_ip_hostname: couldn't look up host by addr: xxxxxx</div><div>rc_send_server: no reply from RADIUS server unknown:1812</div></div><div> </div><div>No more critical messages.</div><div> </div><div>Bogdan, i am sorry, but</div><div>"in frame 2". What does it mean?</div><div> </div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>04.04.2017, 16:35, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">Any CRITICAL message into the log before the crash ?<br /><br />also, in gdb in frame 2, please print:<br /> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))<br /> <br /><br />Thanks,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/04/2017 03:19 PM, Denis wrote:</div><blockquote cite="mid:2219671491308376@web4m.yandex.ru" type="cite"><div><div>p shm_block->first_frag</div><div>$1 = (struct qm_frag *) 0x7fdf6b9d44f8</div><div><div> </div><div>p shm_block->last_frag_end</div><div>$2 = (struct qm_frag_end *) 0x7fe02899efd0</div><div> </div><div><div> p mem_block->first_frag</div><div>$3 = (struct qm_frag *) 0x7fe<span><span><span><span><span><span><span><span>029214508</span></span></span></span></span></span></span></span></div><div> </div><div><div>p mem_block->last_frag_end</div><div>$4 = (struct qm_frag_end *) 0x7fe0691defe0</div></div></div></div><div> </div></div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>04.04.2017, 14:51, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">OK,please enter gdb and print:<br /> p shm_block->first_frag<br /> p shm_block->last_frag_end<br /><br /> p mem_block->first_frag<br /> p mem_block->last_frag_end<br /><br />Thanks,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/04/2017 01:42 PM, Denis wrote:</div><blockquote cite="mid:1579141491302539@web17o.yandex.ru" type="cite"><div>Hello, Bogdan!</div><div> </div><div>No, i cannot see in the log any messages you mentioned it your letter.</div><div>The situation dealing with crash had such stage.</div><div> </div><div>1. I prepared to migrate to 2.2.3 from 2.2.2. But do not have a time to restart Opensips</div><div>2. As a result "i waited" when Opensips have crashed. During crash of 2.2.2 no core file has been generated, but Opensips has been restarted using cron.</div><div>3. After 2) Opensips became 2.2.3. It was working about 1 minute and crashed again, but in that time, core file has been generated and i attached information from it.</div><div>So, between 2) and 3) , as i mentioned above, there were no messages about memory, but i see many messages like</div><div>'WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 71500 ms</div><div>WARNING:core:timer_ticker: timer task <pike-clean> already scheduled for 72400 ms</div><div>WARNING:core:timer_ticker: timer task <nh-timer> already scheduled for 72400 ms</div><div>WARNING:core:timer_ticker: timer task <dlg-timer> already scheduled for 72400 ms</div><div>WARNING:core:timer_ticker: timer task <blcore-expire> already scheduled for 72400 ms'</div><div>and so on</div><div> </div><div>As for about version</div><div> </div><div><div>version: opensips 2.2.3 (x86_64/linux)</div><div>flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</div><div>ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535</div><div>poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.</div><div>git revision: <span><span><span><span><span><span><span><span><span>7922417</span></span></span></span></span></span></span></span></span></div><div>main.c compiled on 06:29:28 Mar 23 2017 with gcc 5.4.0</div><div> </div></div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div><div>04.04.2017, 12:43, "Bogdan-Andrei Iancu" <a href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a>:</div><blockquote type="cite"><div bgcolor="#FFFFFF">Hello Denis,<br /><br />Thank you for the report. Do you see any CRITICAL log just before the crash, something like:<br /> "bad pointer 0xxxxxxxx (out of memory block!) - aborting"<br />?<br />If yes, please post here the exact line.<br /><br />Also, do you use the latest OpenSIPS 2.2 from GIT repo ? (run "opensips -v")<br /><br />Regards,<pre>Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com/">http://www.opensips-solutions.com</a>
OpenSIPS Summit May 2017 Amsterdam
<a href="http://www.opensips.org/events/Summit-2017Amsterdam.html">http://www.opensips.org/events/Summit-2017Amsterdam.html</a>
</pre><div>On 04/04/2017 10:58 AM, Denis via Users wrote:</div><blockquote cite="mid:325131491292730@web16g.yandex.ru" type="cite"><div>Hello!</div><div> </div><div>Server:: OpenSIPS (2.2.3 (x86_64/linux))</div><div> </div><div>Thank you for any help.</div><div> </div><div>P.S. Opensips has been compiled with QM_MALLOC and DBG_MALLOC flags</div><div> </div><div>-- </div><div>С уважением, Денис.</div><div>Best regards, Denis</div><div> </div><div> </div><div> </div> <pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre></blockquote></div></blockquote></blockquote></div></blockquote></blockquote></div></blockquote></blockquote></div></blockquote></blockquote></div></blockquote></blockquote></div></blockquote></blockquote></div></blockquote></blockquote></div></blockquote></blockquote></div></blockquote>,<p>_______________________________________________<br />Users mailing list<br /><a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br /><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a></p></blockquote>