[OpenSIPS-Users] Memory leak

Thu Dec 16 07:43:50 EST 2021

Hi Bogdan

I think I found the issue, I recently added these lines of code,
because of a probing issue I was having, I just searched from my
previous tickets and I see that you have warned me about the
implications but for some reason I never read the message.

Here is the old ticket
https://www.mail-archive.com/users@lists.opensips.org/msg43301.html
the reason I'm using INVITE to probe is because I want the servers
that were probed not only to respond but also check if the database is
working, I did it this way because I had cases where mysql crashed but
my asterisk servers were still responding to the probe but all of the
calls just hung, so I do a invite and it does a DB lookup and it will
only return a positive message if it was able to query the DB, do you
have a better solution? at the time I set it up I couldn't run a query
on receipt of a OPTIONS but perhaps I didn't look good enough :-),
either way can I do anything to make sure this code doesn't leak
memory? this probing has worked for years until I needed the Contact
header.

local_route {
     if (is_method("INVITE")&& $fU=="pingTest"){
        $ru="sip:s@"+$rd ;
        append_hf("Contact: <sip:pingTest@$fd:5060>\r\n");
        exit;
     }
}

On Fri, Dec 10, 2021 at 2:16 PM Schneur Rosenberg
<rosenberg11219 at gmail.com> wrote:
>
> Hi Bogdan,
>
> I did it on a backup server, its also leaking memory but at a slower
> pace, I'm attaching the logs when running kill -SIGUSR1 on the pid
> that's growing in size, it still has available memory, I hop this will
> give you a clue.
>
> Here is a pastbin to the loggs https://pastebin.com/KJVb9Y75
>
> On Fri, Dec 10, 2021 at 11:00 AM Schneur Rosenberg
> <rosenberg11219 at gmail.com> wrote:
> >
> > Thank you, does this reduce performance? can I leave it enabled on a
> > production machine? I will wait for the memory leak to be apparent and
> > I'll post the result.
> >
> >
> > On Thu, Dec 9, 2021 at 12:31 PM Bogdan-Andrei Iancu <bogdan at opensips.org> wrote:
> > >
> > > Hi Schneur,
> > >
> > > Just follow the
> > > https://www.opensips.org/Documentation/TroubleShooting-OutOfMem and
> > > provide the dump. This is the only way to investigate this.
> > >
> > > Regards,
> > >
> > > Bogdan-Andrei Iancu
> > >
> > > OpenSIPS Founder and Developer
> > >    https://www.opensips-solutions.com
> > > OpenSIPS eBootcamp 2021
> > >    https://opensips.org/training/OpenSIPS_eBootcamp_2021/
> > >
> > > On 12/8/21 12:14 PM, Schneur Rosenberg wrote:
> > > > I just noticed that process 88 runs the timer handler, perhaps this
> > > > might shed light on whats going on.
> > > >
> > > > opensipsctl fifo ps
> > > > Process::  ID=88 PID=5327 Type=Timer handler
> > > >
> > > > On Wed, Dec 8, 2021 at 10:55 AM Schneur Rosenberg
> > > > <rosenberg11219 at gmail.com> wrote:
> > > >> Now a few hours later this is what I'm getting
> > > >> Dec  8 09:50:13 /sbin/opensips[21699]: ERROR:nathelper:nh_timer: out
> > > >> of pkg memory
> > > >> Dec  8 09:50:16 /sbin/opensips[21699]: WARNING:core:fm_malloc: not
> > > >> enough continuous free pkg memory (3024 bytes left, need 5128),
> > > >> attempting defragmentation... please increase the "-M" command line
> > > >> parameter!
> > > >> Dec  8 09:50:16 /sbin/opensips[21699]: ERROR:core:fm_malloc: not
> > > >> enough free pkg memory (3024 bytes left, need 5128), please increase
> > > >> the "-M" command line parameter!
> > > >>
> > > >> Here is the last 20 package memory max_used_size
> > > >> pkmem:70-max_used_size:: 1009584
> > > >> pkmem:71-max_used_size:: 1009584
> > > >> pkmem:72-max_used_size:: 1009584
> > > >> pkmem:73-max_used_size:: 1009584
> > > >> pkmem:74-max_used_size:: 1009584
> > > >> pkmem:75-max_used_size:: 1009584
> > > >> pkmem:76-max_used_size:: 1009584
> > > >> pkmem:77-max_used_size:: 1009584
> > > >> pkmem:78-max_used_size:: 1009584
> > > >> pkmem:79-max_used_size:: 1009584
> > > >> pkmem:80-max_used_size:: 1044752
> > > >> pkmem:81-max_used_size:: 1075552
> > > >> pkmem:82-max_used_size:: 1116848
> > > >> pkmem:83-max_used_size:: 1117456
> > > >> pkmem:84-max_used_size:: 1102640
> > > >> pkmem:85-max_used_size:: 1306992
> > > >> pkmem:86-max_used_size:: 1706304
> > > >> pkmem:87-max_used_size:: 2507000
> > > >> pkmem:88-max_used_size:: 4194264
> > > >> pkmem:89-max_used_size:: 1009584
> > > >>
> > > >> And here is the real used size, you can see that process 88 maxed out
> > > >> pkmem:69-real_used_size:: 975528
> > > >> pkmem:70-real_used_size:: 978016
> > > >> pkmem:71-real_used_size:: 989592
> > > >> pkmem:72-real_used_size:: 951416
> > > >> pkmem:73-real_used_size:: 982496
> > > >> pkmem:74-real_used_size:: 965744
> > > >> pkmem:75-real_used_size:: 959424
> > > >> pkmem:76-real_used_size:: 949472
> > > >> pkmem:77-real_used_size:: 983080
> > > >> pkmem:78-real_used_size:: 961400
> > > >> pkmem:79-real_used_size:: 977808
> > > >> pkmem:80-real_used_size:: 978928
> > > >> pkmem:81-real_used_size:: 1009936
> > > >> pkmem:82-real_used_size:: 1110760
> > > >> pkmem:83-real_used_size:: 1116720
> > > >> pkmem:84-real_used_size:: 1096568
> > > >> pkmem:85-real_used_size:: 1300592
> > > >> pkmem:86-real_used_size:: 1699648
> > > >> pkmem:87-real_used_size:: 2501096
> > > >> pkmem:88-real_used_size:: 4191280
> > > >> pkmem:89-real_used_size:: 882528
> > > >>
> > > >> On Tue, Dec 7, 2021 at 7:53 PM Schneur Rosenberg
> > > >> <rosenberg11219 at gmail.com> wrote:
> > > >>> Hi, lately I'm getting  these errors in my logs.
> > > >>>
> > > >>> ERROR:core:fm_malloc: not enough free pkg memory (1792 bytes left,
> > > >>> need 2184), please increase the "-M" command line para
> > > >>> meter!
> > > >>>
> > > >>> CRITICAL:core:hostent_cpy: pkg memory allocation failure
> > > >>>
> > > >>> ERROR:nathelper:nh_timer: out of pkg memory
> > > >>>
> > > >>> ERROR:core:fm_malloc: not enough free pkg memory (5952 bytes left,
> > > >>> need 5408), please increase the "-M" command line para
> > > >>> meter!
> > > >>>
> > > >>> I was on version 2.4.8 and I upgraded to 2.4.11 and I'm monitoring the
> > > >>> max_used_size of the package memory, a few hours later I see that 2
> > > >>> processes keep on getting bigger, so far the rest are pretty stable, I
> > > >>> have 90 processes and 87 and 88 are growing.
> > > >>>
> > > >>> here you can see the last few processes, OpenSIPS set aside 4 mb per process.
> > > >>>
> > > >>> pkmem:80-max_used_size:: 1009584
> > > >>> pkmem:81-max_used_size:: 1009584
> > > >>> pkmem:82-max_used_size:: 1009584
> > > >>> pkmem:83-max_used_size:: 1009584
> > > >>> pkmem:84-max_used_size:: 1009584
> > > >>> pkmem:85-max_used_size:: 1009584
> > > >>> pkmem:86-max_used_size:: 1143608
> > > >>> pkmem:87-max_used_size:: 1323256
> > > >>> pkmem:88-max_used_size:: 1831928
> > > >>> pkmem:89-max_used_size:: 1009584
> > > >>>
> > > >>> Any hints where to start looking besides the solutions fund here.
> > > >>>
> > > >>> https://www.opensips.org/Documentation/TroubleShooting-OutOfMem
> > > >>>
> > > >>> thank you
> > > >>> Scott
> > > > _______________________________________________
> > > > Users mailing list
> > > > Users at lists.opensips.org
> > > > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > >