[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"
Kevin Mathy
k.mathy at hexanet.fr
Tue Jul 8 08:43:31 CEST 2014
Hi Bogdan,
Sounds really good :-) ! I'm going to check if there's no configuration
change between 1.9's and 1.11's modules version, and then will upgrade our
version !
I hope this will correct our problem ;-)
Thanks for your help,
Kevin
*Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
2014-07-07 18:40 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
> Hi Kevin,
>
> OK, found the problem. As expected, this was fixed in 1.11, but not ported
> to 1.9 as this version is not maintained any more. Here is the link to
> the fix for your crash:
>
>
> https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75
>
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>
> On 07.07.2014 18:15, Kevin Mathy wrote:
>
> Hi Bogdan,
>
> Opensips has crashed once again; I hope this time the memlog will be
> right to be analysed !
> Here is the link to download the file : ..................
>
> If you need further informations, feel free to ask me !
>
> Thanks for your help,
>
> Kevin
>
>
>
>
> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>
>
>
> 2014-07-04 11:54 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>
>> Hi Kevin,
>>
>> Indeed, it looks compiled ok now (with the memory debugger). Keep in mind
>> to set (in this order):
>> mem_log=4
>> mem_dump=1
>> to avoid runtime logs, but to get the mem dumps.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>
>> On 04.07.2014 12:41, Kevin Mathy wrote:
>>
>> Hi Bogdan,
>>
>> We've recompiled OpenSIPS with requested modules (I think) :
>>
>>> root at asbc2:/home/kemathy# opensips -V
>>> version: opensips 1.9.2-notls (x86_64/linux)
>>> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
>>> SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*, FAST_LOCK-ADAPTIVE_WAIT
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>> svnrevision: unknown
>>> @(#) $Id$
>>> main.c compiled on 22:12:10 Jul 3 2014 with gcc 4.7
>>
>>
>> Now we'll keep an eye on our server to check if everything is OK, and
>> if the memory error still occur; as we upgraded from 1.9.1 to 1.9.2...
>>
>> I'll get back to you with some logs if needed ;-)
>>
>> Kevin
>>
>>
>>
>>
>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>>
>>
>>
>> 2014-07-02 9:37 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:
>>
>>> Hi Bogdan,
>>>
>>> Hummm, right, opensips doesn't seem to have been compiled with the
>>> requested modules for memory debugging...
>>>
>>> root at asbc2:/home/kemathy# opensips -V
>>>> version: opensips 1.9.2-notls (x86_64/linux)
>>>> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
>>>> SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>> svnrevision: unknown
>>>> @(#) $Id$
>>>> main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7
>>>
>>>
>>> So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and
>>> try again to export the memdump log...
>>>
>>> I'll get back to you when done.
>>>
>>> Thanks a lot for your help !
>>>
>>> Kevin
>>>
>>>
>>>
>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>>>
>>>
>>>
>>> 2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>
>>> Hi Kevin,
>>>>
>>>> Unfortunately the logs are not correct - are you sure you properly
>>>> compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the
>>>> FM_MALLOC flags ? As the logs show the standard memory manager (without
>>>> debugging).
>>>> Check it with "opensips -V" to see the list of compiled flags.
>>>>
>>>> I tried to get some ideas by only looking at the available memory and
>>>> how many fragments were allocated in each process - indeed, there are some
>>>> processes using maybe like 2 or 3 times more PKG, but not sure if a leak.
>>>> Getting the proper logs (which will be huge) will tell us more.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>
>>>> On 01.07.2014 18:11, Kevin Mathy wrote:
>>>>
>>>> Hi Bogdan,
>>>>
>>>> I have now a memdump log, as we restarted opensips this afternoon for
>>>> a configuration maintenance... But the file is too big, even if I try to
>>>> put it to pastebin.com ... So, here is the file; I don't want to give
>>>> the link on the mailing-list :-)
>>>>
>>>>
>>>> [removed]
>>>>
>>>> I hope this will help understanding our problem's cause :-) ...
>>>>
>>>> Thanks for your help,
>>>>
>>>> Kevin
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>>>>
>>>>
>>>>
>>>> 2014-06-30 16:30 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:
>>>>
>>>>> Hi Bogdan,
>>>>>
>>>>> Ooops, I thought my two first mails have been cancelled :-)
>>>>> I prefer waiting till there's no traffic, so I'll send a SIGUSR1 comme
>>>>> this evening, and reply to this topic with the log.
>>>>>
>>>>> I'll try working with MI statistics to make some memory usage graphs
>>>>> better than with Cacti...
>>>>>
>>>>> I'll come back to you with logs; thanks for all !
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>>>>>
>>>>>
>>>>>
>>>>> 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>>>
>>>>>
>>>>> Hi Kevin,
>>>>>>
>>>>>> There is no need to send your email three times ;). One time is
>>>>>> enough.
>>>>>>
>>>>>> Waiting and taking the dump when there is not traffic is good (but
>>>>>> not a must) - the idea is to be sure that all temporary memory (used for
>>>>>> processing the current traffic) was freed - so what you still have in
>>>>>> memory is configuration data or possible leaks.
>>>>>> If you do not have the luxury of waiting, you can do it whenever you
>>>>>> can.
>>>>>>
>>>>>> Once again, do not look at the memory usage reported by OS - it is
>>>>>> irrelevant as OpenSIPS is doing its own internal memory management.
>>>>>>
>>>>>> Check the memory usage via MI, see the mem related statistics:
>>>>>>
>>>>>> http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Bogdan-Andrei Iancu
>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>
>>>>>> On 30.06.2014 12:01, Kevin Mathy wrote:
>>>>>>
>>>>>> Hi Bogdan,
>>>>>>
>>>>>> If I want to send a SIGUSR1, may I have to wait 20 minutes after
>>>>>> the last call ? 20 minutes without any call ? I don't understand well this
>>>>>> sentence :
>>>>>>
>>>>>>> It is highly recommended to do this after waiting about 20 minutes
>>>>>>> to be sure that as much as possile memory is freed - all temporary memory
>>>>>>> used during processing is freed by lack of load on the proxy
>>>>>>
>>>>>>
>>>>>> Also, last week-end, the traffic reduced a lot, and between last
>>>>>> friday, when the free system's memory was around 170M, and this morning,
>>>>>> the free memory seems to have increased : this morning, it was around 700M,
>>>>>> before the traffic comes back.
>>>>>>
>>>>>> So, opensips seems to well free the memory, isn't it ?
>>>>>>
>>>>>> Thanks for your help,
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>>
>>>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>>>>
>>>>>>> Hi Kevin,
>>>>>>>
>>>>>>> There is no need to wait for a crash. From time to time, you can
>>>>>>> send a SIGUSR1 to a worker process (or a process you suspect as running out
>>>>>>> of pkg mem) -> the process will do a pkg dump to the log.
>>>>>>>
>>>>>>> Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9
>>>>>>> is no longer maintained and there were some fixes in the memory manager
>>>>>>> since then.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Bogdan-Andrei Iancu
>>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>>
>>>>>>> On 27.06.2014 10 <27.06.2014%2010>:36, Kevin Mathy wrote:
>>>>>>>
>>>>>>> Hi Bogdan,
>>>>>>>
>>>>>>> I've set given options, and now I'm waiting for a new crash of the
>>>>>>> service... Where the memdump will be located ? In another logfile than
>>>>>>> opensips.log, or in the same ?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>> * Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>
>>>>>>> :
>>>>>>>
>>>>>>>> Kevin,
>>>>>>>>
>>>>>>>> Restarting should not make you loose ongoing calls (even if you use
>>>>>>>> the dialog module), do do not worry on that.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
>
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140708/466fd695/attachment-0001.htm>
More information about the Users
mailing list