[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"
Bogdan-Andrei Iancu
bogdan at opensips.org
Fri Jul 4 11:54:26 CEST 2014
Hi Kevin,
Indeed, it looks compiled ok now (with the memory debugger). Keep in
mind to set (in this order):
mem_log=4
mem_dump=1
to avoid runtime logs, but to get the mem dumps.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
http://www.opensips-solutions.com
On 04.07.2014 12:41, Kevin Mathy wrote:
> Hi Bogdan,
>
> We've recompiled OpenSIPS with requested modules (I think) :
>
> root at asbc2:/home/kemathy# opensips -V
> version: opensips 1.9.2-notls (x86_64/linux)
> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
> SHM_MEM, SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*,
> FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
> 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> svnrevision: unknown
> @(#) $Id$
> main.c compiled on 22:12:10 Jul 3 2014 with gcc 4.7
>
>
> Now we'll keep an eye on our server to check if everything is OK, and
> if the memory error still occur; as we upgraded from 1.9.1 to 1.9.2...
>
> I'll get back to you with some logs if needed ;-)
>
> Kevin
>
>
>
> *
> Bien cordialement,
> Best Regards,
>
> **Kevin MATHY* |**Ingénieur VoIP
> *
> *
>
>
> 2014-07-02 9:37 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr
> <mailto:k.mathy at hexanet.fr>>:
>
> Hi Bogdan,
>
> Hummm, right, opensips doesn't seem to have been compiled with the
> requested modules for memory debugging...
>
> root at asbc2:/home/kemathy# opensips -V
> version: opensips 1.9.2-notls (x86_64/linux)
> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
> SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC,
> FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
> MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> svnrevision: unknown
> @(#) $Id$
> main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7
>
>
> So I think I'll have to re-compile opensips with QM_DBG_MALLOC,
> and try again to export the memdump log...
>
> I'll get back to you when done.
>
> Thanks a lot for your help !
>
> Kevin
>
>
> *
> Bien cordialement,
> Best Regards,
>
> **Kevin MATHY* |**Ingénieur VoIP
> *
> *
>
>
> 2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu
> <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>
> Hi Kevin,
>
> Unfortunately the logs are not correct - are you sure you
> properly compiled the mem debug ? like adding the
> QM_DBG_MALLOC and removing the FM_MALLOC flags ? As the logs
> show the standard memory manager (without debugging).
> Check it with "opensips -V" to see the list of compiled flags.
>
> I tried to get some ideas by only looking at the available
> memory and how many fragments were allocated in each process -
> indeed, there are some processes using maybe like 2 or 3 times
> more PKG, but not sure if a leak.
> Getting the proper logs (which will be huge) will tell us more.
>
>
> Regards,
>
> Bogdan-Andrei Iancu
> OpenSIPS Founder and Developer
> http://www.opensips-solutions.com
>
> On 01.07.2014 18:11, Kevin Mathy wrote:
>> Hi Bogdan,
>>
>> I have now a memdump log, as we restarted opensips this
>> afternoon for a configuration maintenance... But the file is
>> too big, even if I try to put it to pastebin.com
>> <http://pastebin.com> ... So, here is the file; I don't want
>> to give the link on the mailing-list :-)
>
> [removed]
>
>> I hope this will help understanding our problem's cause :-) ...
>>
>> Thanks for your help,
>>
>> Kevin
>>
>>
>>
>>
>>
>>
>> *
>> Bien cordialement,
>> Best Regards,
>>
>> **Kevin MATHY* |**Ingénieur VoIP
>> *
>> *
>>
>>
>> 2014-06-30 16:30 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr
>> <mailto:k.mathy at hexanet.fr>>:
>>
>> Hi Bogdan,
>>
>> Ooops, I thought my two first mails have been cancelled :-)
>> I prefer waiting till there's no traffic, so I'll send a
>> SIGUSR1 comme this evening, and reply to this topic with
>> the log.
>>
>> I'll try working with MI statistics to make some memory
>> usage graphs better than with Cacti...
>>
>> I'll come back to you with logs; thanks for all !
>>
>> Kevin
>>
>>
>>
>> *
>> Bien cordialement,
>> Best Regards,
>>
>> **Kevin MATHY* |**Ingénieur VoIP
>> *
>> *
>>
>>
>> 2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu
>> <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>
>> Hi Kevin,
>>
>> There is no need to send your email three times ;).
>> One time is enough.
>>
>> Waiting and taking the dump when there is not traffic
>> is good (but not a must) - the idea is to be sure
>> that all temporary memory (used for processing the
>> current traffic) was freed - so what you still have
>> in memory is configuration data or possible leaks.
>> If you do not have the luxury of waiting, you can do
>> it whenever you can.
>>
>> Once again, do not look at the memory usage reported
>> by OS - it is irrelevant as OpenSIPS is doing its own
>> internal memory management.
>>
>> Check the memory usage via MI, see the mem related
>> statistics:
>> http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developer
>> http://www.opensips-solutions.com
>>
>> On 30.06.2014 12:01, Kevin Mathy wrote:
>>> Hi Bogdan,
>>>
>>> If I want to send a SIGUSR1, may I have to wait 20
>>> minutes after the last call ? 20 minutes without any
>>> call ? I don't understand well this sentence :
>>>
>>> It is highly recommended to do this after
>>> waiting about 20 minutes to be sure that as much
>>> as possile memory is freed - all temporary
>>> memory used during processing is freed by lack
>>> of load on the proxy
>>>
>>>
>>> Also, last week-end, the traffic reduced a lot, and
>>> between last friday, when the free system's memory
>>> was around 170M, and this morning, the free memory
>>> seems to have increased : this morning, it was
>>> around 700M, before the traffic comes back.
>>>
>>> So, opensips seems to well free the memory, isn't it ?
>>>
>>> Thanks for your help,
>>>
>>> Kevin
>>>
>>>
>>> *
>>> Bien cordialement,
>>> Best Regards,
>>>
>>> **Kevin MATHY* |**Ingénieur VoIP
>>> *
>>> *
>>>
>>>
>>> 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu
>>> <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>>
>>> Hi Kevin,
>>>
>>> There is no need to wait for a crash. From time
>>> to time, you can send a SIGUSR1 to a worker
>>> process (or a process you suspect as running out
>>> of pkg mem) -> the process will do a pkg dump to
>>> the log.
>>>
>>> Also, I would strongly advice upgrading to 1.11
>>> (latest LTS) - 1.9 is no longer maintained and
>>> there were some fixes in the memory manager
>>> since then.
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Founder and Developer
>>> http://www.opensips-solutions.com
>>>
>>> On 27.06.2014 10 <tel:27.06.2014%2010>:36, Kevin
>>> Mathy wrote:
>>>> Hi Bogdan,
>>>>
>>>> I've set given options, and now I'm waiting for
>>>> a new crash of the service... Where the memdump
>>>> will be located ? In another logfile than
>>>> opensips.log, or in the same ?
>>>>
>>>> Thanks
>>>>
>>>> *
>>>> Bien cordialement,
>>>> Best Regards,
>>>>
>>>> **Kevin MATHY* |**Ingénieur VoIP
>>>> *
>>>> *
>>>>
>>>>
>>>> 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu
>>>> <bogdan at opensips.org <mailto:bogdan at opensips.org>>:
>>>>
>>>> Kevin,
>>>>
>>>> Restarting should not make you loose
>>>> ongoing calls (even if you use the dialog
>>>> module), do do not worry on that.
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>> OpenSIPS Founder and Developer
>>>> http://www.opensips-solutions.com
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140704/600976dc/attachment-0001.htm>
More information about the Users
mailing list