[OpenSIPS-Users] Opensips crashing due to out of memory error
Anca Vamanu
anca at opensips.org
Thu Apr 21 15:16:19 CEST 2011
Hi John,
yes.. it is something in libxml library ( this is what presence uses). I
will try to find it.
Thanks,
--
Anca Vamanu
OpenSIPS Developer
On 04/20/2011 06:51 PM, John Khvatov wrote:
> Hi Anca,
>
> On 20.04.2011, at 13:24, Anca Vamanu wrote:
>
>> Hi John,
>>
>> Thanks for the logs.
>> Do you get the memory statistics from opensips statistics or from top?
>> In the logs I don't see a shared memory leak..
> Hm... Maybe the memory leak in the system libraries?
>
> Memory usage is calculated as follows:
>
> # ps aux | head -n 1
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> # ps aux | grep -E "/usr/sbin/opensips -[P]"
> opensips 8389 0.0 0.0 451444 9880 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8395 0.2 0.0 451448 5216 ? S Apr19 2:06 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8396 0.0 0.0 451448 5184 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8398 0.0 0.0 451452 5160 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8399 0.0 0.1 451684 12732 ? S Apr19 0:10 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8400 0.0 0.1 451704 12708 ? S Apr19 0:09 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8402 0.0 0.1 451708 12644 ? S Apr19 0:09 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8403 0.0 0.1 451708 12640 ? S Apr19 0:10 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8405 0.0 0.0 451448 5088 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8406 0.0 0.0 451448 5088 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8409 0.0 0.0 451448 5088 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8410 0.0 0.0 451448 5088 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8411 0.0 0.0 450644 3148 ? S Apr19 0:07 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8412 0.0 0.1 451448 11572 ? S Apr19 0:11 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8413 0.0 0.0 451444 5304 ? S Apr19 0:05 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8414 0.0 0.0 451452 9660 ? S Apr19 0:01 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8418 0.0 0.0 451452 9336 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8426 0.0 0.0 451452 9684 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8428 0.0 0.0 451452 5920 ? S Apr19 0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> opensips 8429 0.0 0.0 451416 6544 ? S Apr19 0:01 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
> # ps aux | grep -E "/usr/sbin/opensips -[P]" | awk '{a += $6} END {print a}'
> 157684
>
> i.e. sum of RSS value, where RSS (man ps)
> RSS - resident set size, the non-swapped physical memory that a task has used (in kiloBytes). (alias rssize, rsz).
>
> How can I help?
>
>
>> On 04/20/2011 10:18 AM, John Khvatov wrote:
>>> Hello Anca,
>>>
>>> On 19.04.2011, at 13:21, Anca Vamanu wrote:
>>>
>>>> Hi John,
>>>>
>>>> That looks very bad...
>>>> Please help me investigate this.
>>>>
>>>> What you can do is to compile with memory debugging support - in Makefile.defs search -DDBG_QM_MALLOC line, uncomment it, move it above -DF_MALLOC line and comment this line with -DF_MALLOC. Then recompile and reinstall all. Also set mem_dump=1 in your config.
>>> I've built OpenSIPS from latest SVN head (branch 1.6) as you said.
>>>
>>>> After a couple of hours, when you see that the memory is high - you can either shutdown opensips or ask it to dump the memory by calling 'kill -SIGUSR1 lowest_pid' , where lowest_pid is the smallest pid when doing 'ps aux | grep opensips'. It will dump lots of info in the log file(this why is better to do this quite soon - maybe after 1-2 hours in the rhythm the memory increases for you). In the logs you will see all the chunks of memory and where they were allocated. Look from the end if there is a function repeated many times. You can also send the part of the log with the dump to me ( not on the list, because it will be quite large).
>>> Done.
>>>
>>> SIGUSR1:
>>> http://dev.sgu.ru/pub/memdump.log.gz
>>>
>>> SIGTERM:
>>> http://dev.sgu.ru/pub/memdump.log.shutdown.gz
>>>
>>>
>>>> On 04/19/2011 10:30 AM, John Khvatov wrote:
>>>>> Hello Anca,
>>>>>
>>>>> Sorry, the memory leak is still here:
>>>>>
>>>>> http://dev.sgu.ru/pub/opensips_memory_usage.2.png
>>>>>
>>>>> BTW, dialog module was not used in my script. I've removed loadmodule before update.
>>>>>
>>>>>
>>>>> On 12.04.2011, at 0:00, John Khvatov wrote:
>>>>>
>>>>>> Hello Anca,
>>>>>>
>>>>>> Thank you for the advice. I built OpenSIPS from rev7863. BTW, rev7824 was used in production.
>>>>>>
>>>>>> I'll let you know the result within a few days.
>>>>>>
>>>>>>
>>>>>> On 11.04.2011, at 17:48, Anca Vamanu wrote:
>>>>>>
>>>>>>> Hi John,
>>>>>>>
>>>>>>> Can you please take the pua and pua_usrloc and dialog modules from svn? There were some problems discovered in there and fixed.
>>>>>>> The svn version of these modules ( 1.6 branch) is stable, we are using it in production.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Anca Vamanu
>>>>>>> OpenSIPS Developer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 04/11/2011 12:45 AM, John Khvatov wrote:
>>>>>>>> On 10.04.2011, at 8:35, Ronald Cepres wrote:
>>>>>>>>
>>>>>>>>> Have you tried increasing the amount of allocated memory for OpenSIPS?
>>>>>>>> I have same problem with OpenSIPS since 1.6.0 release. OpenSIPS has huge memory leak somewhere. Increasing the amount of allocated memory helps only delay crash.
>>>>>>>>
>>>>>>>> Memory usage graph (1.6.4, KkB is MB):
>>>>>>>> http://dev.sgu.ru/pub/opensips_memory_usage.png
>>>>>>>>
>>>>>>>> I use the following modules:
>>>>>>>>
>>>>>>>> $ grep loadmodule opensips.cfg
>>>>>>>> loadmodule "signaling.so"
>>>>>>>> loadmodule "sl.so"
>>>>>>>> loadmodule "tm.so"
>>>>>>>> loadmodule "maxfwd.so"
>>>>>>>> loadmodule "rr.so"
>>>>>>>> loadmodule "textops.so"
>>>>>>>> loadmodule "localcache.so"
>>>>>>>> loadmodule "mi_fifo.so"
>>>>>>>> loadmodule "mi_datagram.so"
>>>>>>>> loadmodule "db_flatstore.so"
>>>>>>>> loadmodule "siptrace.so"
>>>>>>>> loadmodule "db_text.so"
>>>>>>>> loadmodule "db_postgres.so"
>>>>>>>> loadmodule "db_http.so"
>>>>>>>> loadmodule "usrloc.so"
>>>>>>>> loadmodule "registrar.so"
>>>>>>>> loadmodule "auth.so"
>>>>>>>> loadmodule "auth_db.so"
>>>>>>>> loadmodule "uri.so"
>>>>>>>> loadmodule "domain.so"
>>>>>>>> loadmodule "dialog.so"
>>>>>>>> loadmodule "mediaproxy.so"
>>>>>>>> loadmodule "nat_traversal.so"
>>>>>>>> loadmodule "permissions.so"
>>>>>>>> loadmodule "alias_db.so"
>>>>>>>> loadmodule "avpops.so"
>>>>>>>> loadmodule "mi_xmlrpc.so"
>>>>>>>> loadmodule "dialplan.so"
>>>>>>>> loadmodule "pike.so"
>>>>>>>> loadmodule "pua.so"
>>>>>>>> loadmodule "pua_usrloc.so"
>>>>>>>>
>>>>>>>> --
>>>>>>>> WBR, John Khvatov
>>>>>> --
>>>>>> WBR, John Khvatov
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
More information about the Users
mailing list