[OpenSIPS-Devel] Finding a memory leak

[Liviu Chircu]

Hi John,

Memory leak troubleshooting is more difficult on OpenSIPS versions up 
to, and including, 2.1. Starting from version 2.2, we have improved the 
output summary in such a way that leaks are immediately visible, with no 
additional work on behalf of the user!

Now, it's still possible to spot the leak with your current version, we 
only need to work a bit more. Useful steps:

* if needed, make sure any syslog rate-limiting is disabled. These dumps 
can be quite large!

* use "opensipsctl fifo ps" to find the PID of each OpenSIPS process you 
may want to send the "kill -SIGUSR1" command

* "pv_add_extra(4480)" points to an allocation at line 4480, of the 
"pv_add_extra()" function

* if there were a leak at "pv_add_extra(4480)", then periodical dumps of 
a single process will contain more and more lines indicating 
"pv_add_extra(4480)". You may need to do some output text manipulation 
magic in order to sum up all this information, for each dump.

Best regards,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 15.12.2016 18:20, John Quick wrote:
> Please can someone provide advice on interpreting OpenSIPS memory dumps and
> identifying the cause of a memory leak.
>
> A Proxy application I wrote, running v2.1.4 with DBG_QM_MALLOC enabled, is
> leaking package memory so badly that I have to restart the service once a
> week.
>
> When I use this command:
>          opensipsctl fifo get_statistics pkmem: | grep max
> ...it lists 57 lines of output. Most of these show a similar value from day
> to day, but there are three instances near the middle of the list that
> increase from one day to the next. For example:
> pkmem:0-max_used_size:: 1359072
> pkmem:1-max_used_size:: 1427568
> pkmem:2-max_used_size:: 1463256
> ...
> pkmem:23-max_used_size:: 3013848
> pkmem:24-max_used_size:: 2541240
> pkmem:25-max_used_size:: 3799672
> ...
> pkmem:55-max_used_size:: 1420112
> pkmem:56-max_used_size:: 1422104
> pkmem:57-max_used_size:: 1396384
>
> When I trigger a memory report (dump) using the command
>          kill -SIGUSR1 971
> ...the resulting output in my opensips.log file looks almost the same from
> one day top the next.
>
> Do I need to use a different Process ID so I can get a memory dump for the
> pkmem:25 child thread?
> If so, how do I know which PID to use?  The command "ps ax | grep opensips"
> returns a list of 59 process ID's.
> Even if I get the right PID, what am I looking for in the memory dump?
> Typical memory dump output looks like this:
> 2016-12-15 15:54:44       41. N  address=0x7f4be8db9048 frag=0x7f4be8db9018
> size=48 used=1
> 2016-12-15 15:54:44              alloc'd from sr_module.c:
> register_module(151)
> 2016-12-15 15:54:44          start check=f0f0f0f0f0f0f0f0, end check=
> c0c0c0c0c0c0c0c0, abcdefedabcdefed
> 2016-12-15 15:54:44       42. N  address=0x7f4be8db90d8 frag=0x7f4be8db90a8
> size=128 used=1
> 2016-12-15 15:54:44              alloc'd from cfg.lex: addstr(925)
> 2016-12-15 15:54:44          start check=f0f0f0f0f0f0f0f0, end check=
> c0c0c0c0c0c0c0c0, abcdefedabcdefed
> ...
> 2016-12-15 15:54:44       63. N  address=0x7f4be8db9f70 frag=0x7f4be8db9f40
> size=80 used=1
> 2016-12-15 15:54:44              alloc'd from pvar.c: pv_add_extra(4480)
> 2016-12-15 15:54:44          start check=f0f0f0f0f0f0f0f0, end check=
> c0c0c0c0c0c0c0c0, abcdefedabcdefed
> 2016-12-15 15:54:44       64. N  address=0x7f4be8dba020 frag=0x7f4be8db9ff0
> size=80 used=1
> 2016-12-15 15:54:44              alloc'd from pvar.c: pv_add_extra(4480)
>
> I have not found any cases where the value for "frag size" exceeds 240. The
> line that starts "alloc'd from" never seems to have a value in parentheses
> exceeding 4480.
> I cannot see anything that looks like a smoking gun.  The list never goes
> beyond item 64.
>
> John Quick
> Smartvox Limited
> Tel:   +44-1727-221221
>
>
>
> _______________________________________________
> Devel mailing list
> Devel at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel