[OpenSIPS-Users] OpenSIPS crashed after "out of pkg memory"

Kevin Mathy k.mathy at hexanet.fr
Mon Aug 18 14:25:49 CEST 2014 

Hi Bogdan,

Coming back to you after some holidays, and unfortunately, new Opensips
crashes...
Some weeks ago, we upgraded our servers to OpenSIPS 1.11, but we faced a
new crash, always with the "out of pkg memory" error, last week.

Here is the exact version :

> root at asbc2:/home/kemathy# opensips -V
> version: opensips 1.11.2-notls (x86_64/linux)
> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
> SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> git revision: d41eb41
> main.c compiled on 19:23:35 Jul 23 2014 with gcc 4.7



Here are an extract of logs we get at the crash-time (I masked phone
numbers and IPs for security purposes) :

Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:core:parse_contacts: failed to parse contact parameters
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:core:contact_parser: failed to parse contacts
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:core:parse_contact: failed to parse contact
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:core:pv_get_contact_body: failed to parse contact hdr
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g at pbx --
> New REGISTER received from 188.94.XXX.XXX with Contact :  -- IP
> 188.94.XXX.XXX -- PORT 5060
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]: 90tzz16g at pbx --
> REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 526
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:sl:sl_send_reply_helper: response building failed
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25622]:
> ERROR:auth:challenge: failed to send the response
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r at pbx --
> New REGISTER received from 188.94.XXX.XXX with Contact :
> <sip:0326xxxxxx at 188.94.XXX.XXX:5060;transport=udp;line=6364d3f0> -- IP
> 188.94.XXX.XXX -- PORT 5060
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25606]: iy4l7y1r at pbx --
> REGISTER Failed because of : No Credentials -- From 188.94.XXX.XXX
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]: iy4l7y1r at pbx --
> New REGISTER received from 188.94.XXX.XXX with Contact :
> <sip:0326xxxxxx at 188.94.XXX.XXX:5060;transport=udp;line=6364d3f0> -- IP
> 188.94.XXX.XXX -- PORT 5060
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:core:db_allocate_rows: no memory left
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:db_mysql:db_mysql_convert_rows: no private memory left
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:db_mysql:db_mysql_convert_result: error while converting rows
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:db_mysql:db_mysql_store_result: error while converting result
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:core:db_do_query: error while storing result for query [select
> contact,expires,q,callid,cseq,flags,cflags,user_agent,received,path,socket,methods,last_modified,sip_instance,attr
> from location where username='0326xxxxxx' order by q]
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:usrloc:db_load_urecord: db_query failed
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> CRITICAL:db_mysql:wrapper_single_mysql_stmt_execute: driver error (1062):
> Duplicate entry '0326xxxxxx--sip:0326xxxxxx at 188.94.XX.XXX:5060;transport=udp;line'
> for key 'account_contact_idx'
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:usrloc:db_insert_ucontact: inserting contact in db failed
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25614]:
> ERROR:usrloc:insert_ucontact: failed to insert in database
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]:
> ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 373
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]:
> ERROR:sl:sl_send_reply_helper: response building failed
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]:
> ERROR:signaling:sig_send_reply_mod: failed to send reply with sl module
> Aug 12 11:48:34 asbc2 /usr/local/sbin/opensips[25620]:
> ERROR:options:opt_reply: failed to send 200 via send_reply


Unfortunately, we don't have the memory dump :-s ...

So, I don't know if there's something we can try to avoid those errors to
happen again; maybe we made something wrong in our configuration file ?
We could provide you anything you want to help us find where's the problem.

Thanks a lot for your help,

Kevin



*Bien cordialement, Best Regards,  **Kevin MATHY* | Ingénieur VoIP



2014-07-08 8:43 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:

> Hi Bogdan,
>
> Sounds really good :-) ! I'm going to check if there's no configuration
> change between 1.9's and 1.11's modules version, and then will upgrade our
> version !
>
> I hope this will correct our problem ;-)
>
> Thanks for your help,
>
> Kevin
>
>
>
>
> *Bien cordialement, Best Regards, **Kevin MATHY* | Ingénieur VoIP
>
>
>
> 2014-07-07 18:40 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>
>  Hi Kevin,
>>
>> OK, found the problem. As expected, this was fixed in 1.11, but not
>> ported to 1.9 as this version is not maintained any more. Here is the
>> link to the fix for your crash:
>>
>>
>> https://github.com/OpenSIPS/opensips/commit/eef59c1d7b47ecc35438c01e7dcbdfeea903115a#diff-30710a8823b167e0aae5a14bb4878b75
>>
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>
>> On 07.07.2014 18:15, Kevin Mathy wrote:
>>
>> Hi Bogdan,
>>
>>  Opensips has crashed once again; I hope this time the memlog will be
>> right to be analysed !
>> Here is the link to download the file : ..................
>>
>>  If you need further informations, feel free to ask me !
>>
>>  Thanks for your help,
>>
>>  Kevin
>>
>>
>>
>>
>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>
>>
>>
>> 2014-07-04 11:54 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>
>>>  Hi Kevin,
>>>
>>> Indeed, it looks compiled ok now (with the memory debugger). Keep in
>>> mind to set (in this order):
>>>     mem_log=4
>>>     mem_dump=1
>>> to avoid runtime logs, but to get the mem dumps.
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>
>>>   On 04.07.2014 12:41, Kevin Mathy wrote:
>>>
>>>  Hi Bogdan,
>>>
>>>  We've recompiled OpenSIPS with requested modules (I think) :
>>>
>>>> root at asbc2:/home/kemathy# opensips -V
>>>> version: opensips 1.9.2-notls (x86_64/linux)
>>>> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
>>>> SHM_MMAP, PKG_MALLOC, *DBG_QM_MALLOC*, FAST_LOCK-ADAPTIVE_WAIT
>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>> svnrevision: unknown
>>>> @(#) $Id$
>>>> main.c compiled on 22:12:10 Jul  3 2014 with gcc 4.7
>>>
>>>
>>>  Now we'll keep an eye on our server to check if everything is OK, and
>>> if the memory error still occur; as we upgraded from 1.9.1 to 1.9.2...
>>>
>>>  I'll get back to you with some logs if needed ;-)
>>>
>>>  Kevin
>>>
>>>
>>>
>>>
>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>
>>>
>>>
>>> 2014-07-02 9:37 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:
>>>
>>>> Hi Bogdan,
>>>>
>>>>  Hummm, right, opensips doesn't seem to have been compiled with the
>>>> requested modules for memory debugging...
>>>>
>>>>   root at asbc2:/home/kemathy# opensips -V
>>>>> version: opensips 1.9.2-notls (x86_64/linux)
>>>>> flags: STATS: On, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST,
>>>>> SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535
>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>>> svnrevision: unknown
>>>>> @(#) $Id$
>>>>> main.c compiled on 11:15:37 Jun 20 2014 with gcc 4.7
>>>>
>>>>
>>>>   So I think I'll have to re-compile opensips with QM_DBG_MALLOC, and
>>>> try again to export the memdump log...
>>>>
>>>>  I'll get back to you when done.
>>>>
>>>>  Thanks a lot for your help !
>>>>
>>>>  Kevin
>>>>
>>>>
>>>>
>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>>
>>>>
>>>>
>>>>  2014-07-01 17:31 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>>
>>>>  Hi Kevin,
>>>>>
>>>>> Unfortunately the logs are not correct - are you sure you properly
>>>>> compiled the mem debug ? like adding the QM_DBG_MALLOC and removing the
>>>>> FM_MALLOC flags ? As the logs show the standard memory manager (without
>>>>> debugging).
>>>>> Check it with "opensips -V" to see the list of compiled flags.
>>>>>
>>>>> I tried to get some ideas by only looking at the available memory and
>>>>> how many fragments were allocated in each process - indeed, there are some
>>>>> processes using maybe like 2 or 3 times more PKG, but not sure if a leak.
>>>>> Getting the proper logs (which will be huge) will tell us more.
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Bogdan-Andrei Iancu
>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>
>>>>>  On 01.07.2014 18:11, Kevin Mathy wrote:
>>>>>
>>>>> Hi Bogdan,
>>>>>
>>>>>  I have now a memdump log, as we restarted opensips this afternoon
>>>>> for a configuration maintenance... But the file is too big, even if I try
>>>>> to put it to pastebin.com ... So, here is the file; I don't want to
>>>>> give the link on the mailing-list :-)
>>>>>
>>>>>
>>>>> [removed]
>>>>>
>>>>>   I hope this will help understanding our problem's cause :-) ...
>>>>>
>>>>>  Thanks for your help,
>>>>>
>>>>>  Kevin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>>>
>>>>>
>>>>>
>>>>> 2014-06-30 16:30 GMT+02:00 Kevin Mathy <k.mathy at hexanet.fr>:
>>>>>
>>>>>> Hi Bogdan,
>>>>>>
>>>>>>  Ooops, I thought my two first mails have been cancelled :-)
>>>>>> I prefer waiting till there's no traffic, so I'll send a SIGUSR1
>>>>>> comme this evening, and reply to this topic with the log.
>>>>>>
>>>>>>  I'll try working with MI statistics to make some memory usage
>>>>>> graphs better than with Cacti...
>>>>>>
>>>>>>  I'll come back to you with logs; thanks for all !
>>>>>>
>>>>>>  Kevin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>>>>
>>>>>>
>>>>>>
>>>>>>  2014-06-30 11:54 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>:
>>>>>>
>>>>>>
>>>>>>  Hi Kevin,
>>>>>>>
>>>>>>> There is no need to send your email three times ;). One time is
>>>>>>> enough.
>>>>>>>
>>>>>>> Waiting and taking the dump when there is not traffic is good (but
>>>>>>> not a must) - the idea is to be sure that all temporary memory (used for
>>>>>>> processing the current traffic) was freed - so what you still have in
>>>>>>> memory is configuration data or possible leaks.
>>>>>>> If you do not have the luxury of waiting, you can do it whenever you
>>>>>>> can.
>>>>>>>
>>>>>>> Once again, do not look at the memory usage reported by OS - it is
>>>>>>> irrelevant as OpenSIPS is doing its own internal memory management.
>>>>>>>
>>>>>>> Check the memory usage via MI, see the mem related statistics:
>>>>>>>
>>>>>>> http://www.opensips.org/Documentation/Interface-CoreStatistics-1-11
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Bogdan-Andrei Iancu
>>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>>
>>>>>>>   On 30.06.2014 12:01, Kevin Mathy wrote:
>>>>>>>
>>>>>>>   Hi Bogdan,
>>>>>>>
>>>>>>>  If I want to send a SIGUSR1, may I have to wait 20 minutes after
>>>>>>> the last call ? 20 minutes without any call ? I don't understand well this
>>>>>>> sentence :
>>>>>>>
>>>>>>>>  It is highly recommended to do this after waiting about 20
>>>>>>>> minutes to be sure that as much as possile memory is freed - all temporary
>>>>>>>> memory used during processing is freed by lack of load on the proxy
>>>>>>>
>>>>>>>
>>>>>>>  Also, last week-end, the traffic reduced a lot, and between last
>>>>>>> friday, when the free system's memory was around 170M, and this morning,
>>>>>>> the free memory seems to have increased : this morning, it was around 700M,
>>>>>>> before the traffic comes back.
>>>>>>>
>>>>>>>  So, opensips seems to well free the memory, isn't it ?
>>>>>>>
>>>>>>>  Thanks for your help,
>>>>>>>
>>>>>>>  Kevin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur VoIP
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2014-06-27 10:38 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org>
>>>>>>> :
>>>>>>>
>>>>>>>>  Hi Kevin,
>>>>>>>>
>>>>>>>> There is no need to wait for a crash. From time to time, you can
>>>>>>>> send a SIGUSR1 to a worker process (or a process you suspect as running out
>>>>>>>> of pkg mem) -> the process will do a pkg dump to the log.
>>>>>>>>
>>>>>>>> Also, I would strongly advice upgrading to 1.11 (latest LTS) - 1.9
>>>>>>>> is no longer maintained and there were some fixes in the memory manager
>>>>>>>> since then.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>>>
>>>>>>>>   On 27.06.2014 10 <27.06.2014%2010>:36, Kevin Mathy wrote:
>>>>>>>>
>>>>>>>>  Hi Bogdan,
>>>>>>>>
>>>>>>>>  I've set given options, and now I'm waiting for a new crash of
>>>>>>>> the service... Where the memdump will be located ? In another logfile than
>>>>>>>> opensips.log, or in the same ?
>>>>>>>>
>>>>>>>>  Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> * Bien cordialement,  Best Regards,  **Kevin MATHY* | Ingénieur
>>>>>>>> VoIP
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-06-26 18:32 GMT+02:00 Bogdan-Andrei Iancu <bogdan at opensips.org
>>>>>>>> >:
>>>>>>>>
>>>>>>>>>  Kevin,
>>>>>>>>>
>>>>>>>>> Restarting should not make you loose ongoing calls (even if you
>>>>>>>>> use the dialog module), do do not worry on that.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Bogdan-Andrei Iancu
>>>>>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>

-- 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20140818/a32238c4/attachment-0001.htm>

More information about the Users mailing list