[OpenSIPS-Users] Opensips crash
Bogdan-Andrei Iancu
bogdan at opensips.org
Wed Apr 5 04:53:17 EDT 2017
Hi Denis,
Many thanks for the output - this confirm a suspicious of mine in
regards to the cause of the crash - let me try to see if we can
reproduce it on our side or if we need more info from your side.
Please do not delete the corefile for the moment :)
Best regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
OpenSIPS Summit May 2017 Amsterdam
http://www.opensips.org/events/Summit-2017Amsterdam.html
On 04/05/2017 11:03 AM, Denis wrote:
> Hello, Bogdan!
> f 5
> #5 0x00007fdf6844c4f2 in free_dlg_dlg (dlg=0x7fdf6c4ac158) at
> dlg_hash.c:183
> 183 destroy_dlg_callbacks_list(dlg->cbs.first);
> p *dlg->cbs.first
> $1 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>, param
> = 0x0, callback_param_free = 0x7fdf675318fe <dlg_free_acc_mask>, next
> = 0x7fdf84191080}
> p *dlg->cbs.first->next
> $2 = {types = 96, callback = 0x7fdf67531300 <acc_dlg_callback>, param
> = 0x7fdf83c9f340, callback_param_free = 0x7fdf675318fe
> <dlg_free_acc_mask>, next = 0x0}
> p *dlg->cbs.first->next->next
> Cannot access memory at address 0x0
> p *dlg->cbs.first->next->next->next
> Cannot access memory at address 0x20
> and so on with ....at address 0x20
> --
> С уважением, Денис.
> Best regards, Denis
> 04.04.2017, 18:43, "Bogdan-Andrei Iancu" <bogdan at opensips.org>:
>> Super :)....we got to something useful.
>>
>> Now, again in GDB please do:
>>
>> f 5
>> p *dlg->cbs.first
>> p *dlg->cbs.first->next
>> p *dlg->cbs.first->next->next
>>
>> and keep adding "->next" until you get error from gdb.
>>
>> Thanks,
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>
>> OpenSIPS Summit May 2017 Amsterdam
>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>> On 04/04/2017 06:07 PM, Денис Путято wrote:
>>> Sorry, Bogdan, i missed that
>>> I found such CRITICAL message
>>> "CRITICAL:core:qm_free: freeing already freed pointer, first free:
>>> acc_logic.c: dlg_free_acc_mask(211) - aborting"
>>> And after that and till crash only "WARNING:core:utimer_ticker:...."
>>> reports in syslog.
>>> (gdb)
>>> f 2
>>> #2 0x0000000000515592 in qm_free (qm=<optimized out>,
>>> p=p at entry=0x7fdf83c9f340, file=file at entry=0x7fdf6753ad62 "acc_logic.c",
>>> func=func at entry=0x7fdf6753af90 <__FUNCTION__.12529>
>>> "dlg_free_acc_mask", line=line at entry=211) at mem/q_malloc.c:483
>>> 483 abort();
>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>> $1 = {size = 8, u = {nxt_free = 0x7fdf71a8c6b0, is_free =
>>> 140597661320880}, file = 0x7fdf6753ad62 "acc_logic.c", func =
>>> 0x7fdf6753af90 <__FUNCTION__.12529> "dlg_free_acc_mask", line = 211,
>>> check = 17361641481138401520}
>>> --
>>> С уважением, Денис.
>>> Best regards, Denis
>>> 04.04.2017, 17:58, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>> <mailto:bogdan at opensips.org>:
>>>> The RADIUS part is not relevant.
>>>>
>>>> Still you didn;t answer to my question on the CRITICAL error -
>>>> according to the backtrace, you should have a CRITICAL message in
>>>> the opensips log, just before the crash.
>>>>
>>>> In GDB, do:
>>>> f 2
>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>
>>>> Regards,
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developer
>>>> http://www.opensips-solutions.com <http://www.opensips-solutions.com/>
>>>>
>>>> OpenSIPS Summit May 2017 Amsterdam
>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>> On 04/04/2017 05:55 PM, Denis wrote:
>>>>> During crash there was another problem with LAN network.
>>>>> Because of this problem Opensips could not connect to radius server.
>>>>> rc_ip_hostname: couldn't look up host by addr: xxxxxx
>>>>> rc_send_server: no reply from RADIUS server unknown:1812
>>>>> No more critical messages.
>>>>> Bogdan, i am sorry, but
>>>>> "in frame 2". What does it mean?
>>>>> --
>>>>> С уважением, Денис.
>>>>> Best regards, Denis
>>>>> 04.04.2017, 16:35, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>> <mailto:bogdan at opensips.org>:
>>>>>> Any CRITICAL message into the log before the crash ?
>>>>>>
>>>>>> also, in gdb in frame 2, please print:
>>>>>> p *((struct qm_frag*) ((char*)p-sizeof(struct qm_frag)))
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developer
>>>>>> http://www.opensips-solutions.com
>>>>>> <http://www.opensips-solutions.com/>
>>>>>>
>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>> On 04/04/2017 03:19 PM, Denis wrote:
>>>>>>> p shm_block->first_frag
>>>>>>> $1 = (struct qm_frag *) 0x7fdf6b9d44f8
>>>>>>> p shm_block->last_frag_end
>>>>>>> $2 = (struct qm_frag_end *) 0x7fe02899efd0
>>>>>>> p mem_block->first_frag
>>>>>>> $3 = (struct qm_frag *) 0x7fe029214508
>>>>>>> p mem_block->last_frag_end
>>>>>>> $4 = (struct qm_frag_end *) 0x7fe0691defe0
>>>>>>> --
>>>>>>> С уважением, Денис.
>>>>>>> Best regards, Denis
>>>>>>> 04.04.2017, 14:51, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>> OK,please enter gdb and print:
>>>>>>>> p shm_block->first_frag
>>>>>>>> p shm_block->last_frag_end
>>>>>>>>
>>>>>>>> p mem_block->first_frag
>>>>>>>> p mem_block->last_frag_end
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>> http://www.opensips-solutions.com
>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>
>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>> On 04/04/2017 01:42 PM, Denis wrote:
>>>>>>>>> Hello, Bogdan!
>>>>>>>>> No, i cannot see in the log any messages you mentioned it your
>>>>>>>>> letter.
>>>>>>>>> The situation dealing with crash had such stage.
>>>>>>>>> 1. I prepared to migrate to 2.2.3 from 2.2.2. But do not have
>>>>>>>>> a time to restart Opensips
>>>>>>>>> 2. As a result "i waited" when Opensips have crashed. During
>>>>>>>>> crash of 2.2.2 no core file has been generated, but Opensips
>>>>>>>>> has been restarted using cron.
>>>>>>>>> 3. After 2) Opensips became 2.2.3. It was working about 1
>>>>>>>>> minute and crashed again, but in that time, core file has been
>>>>>>>>> generated and i attached information from it.
>>>>>>>>> So, between 2) and 3) , as i mentioned above, there were no
>>>>>>>>> messages about memory, but i see many messages like
>>>>>>>>> 'WARNING:core:utimer_ticker: utimer task <tm-utimer> already
>>>>>>>>> scheduled for 71500 ms
>>>>>>>>> WARNING:core:timer_ticker: timer task <pike-clean> already
>>>>>>>>> scheduled for 72400 ms
>>>>>>>>> WARNING:core:timer_ticker: timer task <nh-timer> already
>>>>>>>>> scheduled for 72400 ms
>>>>>>>>> WARNING:core:timer_ticker: timer task <dlg-timer> already
>>>>>>>>> scheduled for 72400 ms
>>>>>>>>> WARNING:core:timer_ticker: timer task <blcore-expire> already
>>>>>>>>> scheduled for 72400 ms'
>>>>>>>>> and so on
>>>>>>>>> As for about version
>>>>>>>>> version: opensips 2.2.3 (x86_64/linux)
>>>>>>>>> flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
>>>>>>>>> PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>>>>>>>>> MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>>>>>>> git revision: 7922417
>>>>>>>>> main.c compiled on 06:29:28 Mar 23 2017 with gcc 5.4.0
>>>>>>>>> --
>>>>>>>>> С уважением, Денис.
>>>>>>>>> Best regards, Denis
>>>>>>>>> 04.04.2017, 12:43, "Bogdan-Andrei Iancu" <bogdan at opensips.org>
>>>>>>>>> <mailto:bogdan at opensips.org>:
>>>>>>>>>> Hello Denis,
>>>>>>>>>>
>>>>>>>>>> Thank you for the report. Do you see any CRITICAL log just
>>>>>>>>>> before the crash, something like:
>>>>>>>>>> "bad pointer 0xxxxxxxx (out of memory block!) - aborting"
>>>>>>>>>> ?
>>>>>>>>>> If yes, please post here the exact line.
>>>>>>>>>>
>>>>>>>>>> Also, do you use the latest OpenSIPS 2.2 from GIT repo ? (run
>>>>>>>>>> "opensips -v")
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>>> OpenSIPS Founder and Developer
>>>>>>>>>> http://www.opensips-solutions.com
>>>>>>>>>> <http://www.opensips-solutions.com/>
>>>>>>>>>>
>>>>>>>>>> OpenSIPS Summit May 2017 Amsterdam
>>>>>>>>>> http://www.opensips.org/events/Summit-2017Amsterdam.html
>>>>>>>>>> On 04/04/2017 10:58 AM, Denis via Users wrote:
>>>>>>>>>>> Hello!
>>>>>>>>>>> Server:: OpenSIPS (2.2.3 (x86_64/linux))
>>>>>>>>>>> Thank you for any help.
>>>>>>>>>>> P.S. Opensips has been compiled with QM_MALLOC and
>>>>>>>>>>> DBG_MALLOC flags
>>>>>>>>>>> --
>>>>>>>>>>> С уважением, Денис.
>>>>>>>>>>> Best regards, Denis
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170405/830f5702/attachment-0001.html>
More information about the Users
mailing list