[OpenSIPS-Users] Opensips crashing due to out of memory error

John Khvatov ivaxer at gmail.com
Wed Apr 20 17:51:20 CEST 2011 

Hi Anca,

On 20.04.2011, at 13:24, Anca Vamanu wrote:

> Hi John,
> 
> Thanks for the logs.
> Do you get the memory statistics from opensips statistics or from top?
> In the logs I don't see a shared memory leak..

Hm... Maybe the memory leak in the system libraries?

Memory usage is calculated as follows:

# ps aux | head -n 1
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# ps aux | grep -E "/usr/sbin/opensips -[P]" 
opensips  8389  0.0  0.0 451444  9880 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8395  0.2  0.0 451448  5216 ?        S    Apr19   2:06 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8396  0.0  0.0 451448  5184 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8398  0.0  0.0 451452  5160 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8399  0.0  0.1 451684 12732 ?        S    Apr19   0:10 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8400  0.0  0.1 451704 12708 ?        S    Apr19   0:09 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8402  0.0  0.1 451708 12644 ?        S    Apr19   0:09 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8403  0.0  0.1 451708 12640 ?        S    Apr19   0:10 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8405  0.0  0.0 451448  5088 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8406  0.0  0.0 451448  5088 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8409  0.0  0.0 451448  5088 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8410  0.0  0.0 451448  5088 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8411  0.0  0.0 450644  3148 ?        S    Apr19   0:07 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8412  0.0  0.1 451448 11572 ?        S    Apr19   0:11 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8413  0.0  0.0 451444  5304 ?        S    Apr19   0:05 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8414  0.0  0.0 451452  9660 ?        S    Apr19   0:01 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8418  0.0  0.0 451452  9336 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8426  0.0  0.0 451452  9684 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8428  0.0  0.0 451452  5920 ?        S    Apr19   0:00 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
opensips  8429  0.0  0.0 451416  6544 ?        S    Apr19   0:01 /usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 256 -u opensips -g opensips
# ps aux | grep -E "/usr/sbin/opensips -[P]" | awk '{a += $6} END {print a}'
157684

i.e. sum of RSS value, where RSS (man ps)
RSS - resident set size, the non-swapped physical memory that a task has used (in kiloBytes). (alias rssize, rsz).

How can I help?


> On 04/20/2011 10:18 AM, John Khvatov wrote:
>> Hello Anca,
>> 
>> On 19.04.2011, at 13:21, Anca Vamanu wrote:
>> 
>>> Hi John,
>>> 
>>> That looks very bad...
>>> Please help me investigate this.
>>> 
>>> What you can do is to compile with memory debugging support - in Makefile.defs search -DDBG_QM_MALLOC line, uncomment it, move it above -DF_MALLOC line and comment this line with -DF_MALLOC. Then recompile and reinstall all. Also set mem_dump=1 in your config.
>> I've built OpenSIPS from latest SVN head (branch 1.6) as you said.
>> 
>>> After a couple of hours, when you see that the memory is high - you can either shutdown opensips or ask it to dump the memory by calling 'kill -SIGUSR1 lowest_pid' , where lowest_pid is the smallest pid when doing 'ps aux | grep opensips'. It will dump lots of info in the log file(this why is better to do this quite soon - maybe after 1-2 hours in the rhythm the memory increases for you). In the logs you will see all the chunks of memory and where they were allocated. Look from the end if there is a function repeated  many times. You can also send the part of the log with the dump to me ( not on the list, because it will be quite large).
>> Done.
>> 
>> SIGUSR1:
>> http://dev.sgu.ru/pub/memdump.log.gz
>> 
>> SIGTERM:
>> http://dev.sgu.ru/pub/memdump.log.shutdown.gz
>> 
>> 
>>> On 04/19/2011 10:30 AM, John Khvatov wrote:
>>>> Hello Anca,
>>>> 
>>>> Sorry, the memory leak is still here:
>>>> 
>>>> http://dev.sgu.ru/pub/opensips_memory_usage.2.png
>>>> 
>>>> BTW, dialog module was not used in my script. I've removed loadmodule before update.
>>>> 
>>>> 
>>>> On 12.04.2011, at 0:00, John Khvatov wrote:
>>>> 
>>>>> Hello Anca,
>>>>> 
>>>>> Thank you for  the advice. I built OpenSIPS from rev7863.  BTW, rev7824 was used in production.
>>>>> 
>>>>> I'll let you know the result within a few days.
>>>>> 
>>>>> 
>>>>> On 11.04.2011, at 17:48, Anca Vamanu wrote:
>>>>> 
>>>>>> Hi John,
>>>>>> 
>>>>>> Can you please take the pua and pua_usrloc and dialog modules from svn? There were some problems discovered in there and fixed.
>>>>>> The svn version of these modules ( 1.6 branch) is stable, we are using it in production.
>>>>>> 
>>>>>> Regards,
>>>>>> -- 
>>>>>> Anca Vamanu
>>>>>> OpenSIPS Developer
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 04/11/2011 12:45 AM, John Khvatov wrote:
>>>>>>> On 10.04.2011, at 8:35, Ronald Cepres wrote:
>>>>>>> 
>>>>>>>> Have you tried increasing the amount of allocated memory for OpenSIPS?
>>>>>>> I have same problem with OpenSIPS since 1.6.0 release. OpenSIPS has huge memory leak somewhere. Increasing the amount of allocated memory helps only delay crash.
>>>>>>> 
>>>>>>> Memory usage graph (1.6.4, KkB is MB):
>>>>>>> http://dev.sgu.ru/pub/opensips_memory_usage.png
>>>>>>> 
>>>>>>> I use the following modules:
>>>>>>> 
>>>>>>> $ grep loadmodule opensips.cfg
>>>>>>> loadmodule "signaling.so"
>>>>>>> loadmodule "sl.so"
>>>>>>> loadmodule "tm.so"
>>>>>>> loadmodule "maxfwd.so"
>>>>>>> loadmodule "rr.so"
>>>>>>> loadmodule "textops.so"
>>>>>>> loadmodule "localcache.so"
>>>>>>> loadmodule "mi_fifo.so"
>>>>>>> loadmodule "mi_datagram.so"
>>>>>>> loadmodule "db_flatstore.so"
>>>>>>> loadmodule "siptrace.so"
>>>>>>> loadmodule "db_text.so"
>>>>>>> loadmodule "db_postgres.so"
>>>>>>> loadmodule "db_http.so"
>>>>>>> loadmodule "usrloc.so"
>>>>>>> loadmodule "registrar.so"
>>>>>>> loadmodule "auth.so"
>>>>>>> loadmodule "auth_db.so"
>>>>>>> loadmodule "uri.so"
>>>>>>> loadmodule "domain.so"
>>>>>>> loadmodule "dialog.so"
>>>>>>> loadmodule "mediaproxy.so"
>>>>>>> loadmodule "nat_traversal.so"
>>>>>>> loadmodule "permissions.so"
>>>>>>> loadmodule "alias_db.so"
>>>>>>> loadmodule "avpops.so"
>>>>>>> loadmodule "mi_xmlrpc.so"
>>>>>>> loadmodule "dialplan.so"
>>>>>>> loadmodule "pike.so"
>>>>>>> loadmodule "pua.so"
>>>>>>> loadmodule "pua_usrloc.so"
>>>>>>> 
>>>>>>> -- 
>>>>>>> WBR, John Khvatov
>>>>> -- 
>>>>> WBR, John Khvatov
> 
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users

-- 
WBR, John Khvatov

More information about the Users mailing list