[OpenSIPS-Users] Opensips crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Thu Apr 6 05:26:55 EDT 2017
yeah, the opensips binary and module libs do not match the corefile
anymore :(.....Do you have the old build ? Also, was this crash on time
only ? or can you reproduce it ?
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Summit May 2017 Amsterdam
http://www.opensips.org/events/Summit-2017Amsterdam.html
On 04/06/2017 12:17 PM, Denis wrote:
> Hello, Bogdan.
> Probably i made a big mistake.
> I have already updated Opensips from latest git (to solve another problem)
> So, when i enter in gdb i get such warning message
> [New LWP 14909]
> warning: .dynamic section for
> "/usr/local/opensips2.2/lib64/opensips/modules/dialog.so" is not at
> the expected address (wrong library or version mismatch?)
> Anyway,
> f 10
> #10 0x00007fdf6c4ac158 in ?? ()
> p p_cell->method.s
> No symbol "p_cell" in current context.
> p p_cell->uas.request
> No symbol "p_cell" in current context.
> p p_cell->uas.status
> No symbol "p_cell" in current context.
> --
> С уважением, Денис.
> Best regards, Denis
> 06.04.2017, 11:08, "Bogdan-Andrei Iancu" <bogdan at opensips.org>:
>> Hi Denis,
>>
>> Please get some more info from the corefile :
>>
>> f 10
>> p p_cell->method.s
>> p p_cell->uas.request
>> p p_cell->uas.status
>>
>> Thanks,
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>
>> OpenSIPS Summit May 2017 Amsterdam
>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>> On 04/05/2017 11:56 AM, Denis wrote:
>>> Ok, Bogdan, I'll be waiting any result.
>>> Thank you.
>>> --
>>> С уважением, Денис.
>>> Best regards, Denis
>>> 05.04.2017, 11:53, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>> <mailto:bogdan at opensips.org>:
>>>> Hi Denis,
>>>>
>>>> Many thanks for the output - this confirm a suspicious of mine in
>>>> regards to the cause of the crash - let me try to see if we can
>>>> reproduce it on our side or if we need more info from your side.
>>>> Please do not delete the corefile for the moment :)
>>>>
>>>> Best regards,
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developer
>>>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>>>
>>>> OpenSIPS Summit May 2017 Amsterdam
>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>> On 04/05/2017 11:03 AM, Denis wrote:
>>>>> Hello, Bogdan!
>>>>> f 5
>>>>> #5 0x00007fdf6844c4f2 in free_dlg_dlg (dlg=0x7fdf6c4ac158) at
>>>>> dlg_hash.c:183
>>>>> 183 destroy_dlg_callbacks_list(dlg->cbs.first);
>>>>> p *dlg->cbs.first
>>>>> $1 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>,
>>>>> param = 0x0, callback_param_free = 0x7fdf675318fe
>>>>> <dlg_free_acc_mask>, next = 0x7fdf84191080}
>>>>> p *dlg->cbs.first->next
>>>>> $2 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>,
>>>>> param = 0x7fdf83c9f340, callback_param_free = 0x7fdf675318fe
>>>>> <dlg_free_acc_mask>, next = 0x0}
>>>>> p *dlg->cbs.first->next->next
>>>>> Cannot access memory at address 0x0
>>>>> p *dlg->cbs.first->next->next->next
>>>>> Cannot access memory at address 0x20
>>>>> and so on with ....at address 0x20
>>>>> --
>>>>> С уважением, Денис.
>>>>> Best regards, Denis
>>>>> 04.04.2017, 18:43, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>> <mailto:bogdan at opensips.org>:
>>>>>> Super :)....we got to something useful.
>>>>>>
>>>>>> Now, again in GDB please do:
>>>>>>
>>>>>> f 5
>>>>>> p *dlg->cbs.first
>>>>>> p *dlg->cbs.first->next
>>>>>> p *dlg->cbs.first->next->next
>>>>>>
>>>>>> and keep adding "->next" until you get error from gdb.
>>>>>>
>>>>>> Thanks,
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developer
>>>>>> http://www.opensips-solutions.com
>>>>>> <http://www.opensips-solutions.com/>
>>>>>>
>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>> On 04/04/2017 06:07 PM, Денис Путято wrote:
>>>>>>> Sorry, Bogdan, i missed that
>>>>>>> I found such CRITICAL message
>>>>>>> "CRITICAL:core:qm_free: freeing already freed pointer, first
>>>>>>> free: acc_logic.c: dlg_free_acc_mask(211) - aborting"
>>>>>>> And after that and till crash only
>>>>>>> "WARNING:core:utimer_ticker:...." reports in syslog.
>>>>>>> (gdb)
>>>>>>> f 2
>>>>>>> #2 0x0000000000515592 in qm_free (qm=<optimized out>,
>>>>>>> p=p at entry=0x7fdf83c9f340, file=file at entry=0x7fdf6753ad62
>>>>>>> "acc_logic.c",
>>>>>>> func=func at entry=0x7fdf6753af90 <__FUNCTION__.12529>
>>>>>>> "dlg_free_acc_mask", line=line at entry=211) at mem/q_malloc.c:483
>>>>>>> 483 abort();
>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>> $1 = {size = 8, u = {nxt_free = 0x7fdf71a8c6b0, is_free =
>>>>>>> 140597661320880}, file = 0x7fdf6753ad62 "acc_logic.c", func =
>>>>>>> 0x7fdf6753af90 <__FUNCTION__.12529> "dlg_free_acc_mask", line = 211,
>>>>>>> check = 17361641481138401520}
>>>>>>> --
>>>>>>> С уважением, Денис.
>>>>>>> Best regards, Denis
>>>>>>> 04.04.2017, 17:58, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>> The RADIUS part is not relevant.
>>>>>>>>
>>>>>>>> Still you didn;t answer to my question on the CRITICAL error -
>>>>>>>> according to the backtrace, you should have a CRITICAL message
>>>>>>>> in the opensips log, just before the crash.
>>>>>>>>
>>>>>>>> In GDB, do:
>>>>>>>> f 2
>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>> http://www.opensips-solutions.com
>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>
>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>> On 04/04/2017 05:55 PM, Denis wrote:
>>>>>>>>> During crash there was another problem with LAN network.
>>>>>>>>> Because of this problem Opensips could not connect to radius
>>>>>>>>> server.
>>>>>>>>> rc_ip_hostname: couldn't look up host by addr: xxxxxx
>>>>>>>>> rc_send_server: no reply from RADIUS server unknown:1812
>>>>>>>>> No more critical messages.
>>>>>>>>> Bogdan, i am sorry, but
>>>>>>>>> "in frame 2". What does it mean?
>>>>>>>>> --
>>>>>>>>> С уважением, Денис.
>>>>>>>>> Best regards, Denis
>>>>>>>>> 04.04.2017, 16:35, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>>>> Any CRITICAL message into the log before the crash ?
>>>>>>>>>>
>>>>>>>>>> also, in gdb in frame 2, please print:
>>>>>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>
>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>> On 04/04/2017 03:19 PM, Denis wrote:
>>>>>>>>>>> p shm_block->first_frag
>>>>>>>>>>> $1 = (struct qm_frag *) 0x7fdf6b9d44f8
>>>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>>>> $2 = (struct qm_frag_end *) 0x7fe02899efd0
>>>>>>>>>>> p mem_block->first_frag
>>>>>>>>>>> $3 = (struct qm_frag *) 0x7fe029214508
>>>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>>>> $4 = (struct qm_frag_end *) 0x7fe0691defe0
>>>>>>>>>>> --
>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>> 04.04.2017, 14:51, "Bogdan-Andrei Iancu"
>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>> OK,please enter gdb and print:
>>>>>>>>>>>> p shm_block->first_frag
>>>>>>>>>>>> p shm_block->last_frag_end
>>>>>>>>>>>>
>>>>>>>>>>>> p mem_block->first_frag
>>>>>>>>>>>> p mem_block->last_frag_end
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>
>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>> On 04/04/2017 01:42 PM, Denis wrote:
>>>>>>>>>>>>> Hello, Bogdan!
>>>>>>>>>>>>> No, i cannot see in the log any messages you mentioned it
>>>>>>>>>>>>> your letter.
>>>>>>>>>>>>> The situation dealing with crash had such stage.
>>>>>>>>>>>>> 1. I prepared to migrate to 2.2.3 from 2.2.2. But do not
>>>>>>>>>>>>> have a time to restart Opensips
>>>>>>>>>>>>> 2. As a result "i waited" when Opensips have crashed.
>>>>>>>>>>>>> During crash of 2.2.2 no core file has been generated, but
>>>>>>>>>>>>> Opensips has been restarted using cron.
>>>>>>>>>>>>> 3. After 2) Opensips became 2.2.3. It was working about 1
>>>>>>>>>>>>> minute and crashed again, but in that time, core file has
>>>>>>>>>>>>> been generated and i attached information from it.
>>>>>>>>>>>>> So, between 2) and 3) , as i mentioned above, there were
>>>>>>>>>>>>> no messages about memory, but i see many messages like
>>>>>>>>>>>>> 'WARNING:core:utimer_ticker: utimer task <tm-utimer>
>>>>>>>>>>>>> already scheduled for 71500 ms
>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <pike-clean> already
>>>>>>>>>>>>> scheduled for 72400 ms
>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <nh-timer> already
>>>>>>>>>>>>> scheduled for 72400 ms
>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <dlg-timer> already
>>>>>>>>>>>>> scheduled for 72400 ms
>>>>>>>>>>>>> WARNING:core:timer_ticker: timer task <blcore-expire>
>>>>>>>>>>>>> already scheduled for 72400 ms'
>>>>>>>>>>>>> and so on
>>>>>>>>>>>>> As for about version
>>>>>>>>>>>>> version: opensips 2.2.3 (x86_64/linux)
>>>>>>>>>>>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
>>>>>>>>>>>>> PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>>>>>>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>>>>>>>>>>>>> MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>>>>>>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt,
>>>>>>>>>>>>> select.
>>>>>>>>>>>>> git revision: 7922417
>>>>>>>>>>>>> main.c compiled on 06:29:28 Mar 23 2017 with gcc 5.4.0
>>>>>>>>>>>>> --
>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>> 04.04.2017, 12:43, "Bogdan-Andrei Iancu"
>>>>>>>>>>>>> <bogdan at opensips.org> <mailto:bogdan at opensips.org>:
>>>>>>>>>>>>>> Hello Denis,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you for the report. Do you see any CRITICAL log
>>>>>>>>>>>>>> just before the crash, something like:
>>>>>>>>>>>>>> "bad pointer 0xxxxxxxx (out of memory block!) - aborting"
>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>> If yes, please post here the exact line.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, do you use the latest OpenSIPS 2.2 from GIT repo ?
>>>>>>>>>>>>>> (run "opensips -v")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>>>>>> On 04/04/2017 10:58 AM, Denis via Users wrote:
>>>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>>>> Server:: OpenSIPS (2.2.3 (x86_64/linux))
>>>>>>>>>>>>>>> Thank you for any help.
>>>>>>>>>>>>>>> P.S. Opensips has been compiled with QM_MALLOC and
>>>>>>>>>>>>>>> DBG_MALLOC flags
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>>>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170406/047bc428/attachment-0001.html>
More information about the Users
mailing list