[OpenSIPS-Users] Memory leak
Bogdan-Andrei Iancu
bogdan at opensips.org
Fri Dec 17 10:35:52 EST 2021
Hi Schneur,
I suspect that the leaking mk_proxy is related to the changing of the
RURI in local route. Let me test your snippet. BTW, is that the whole
processing you do in local route? is the $rd (from LB) a FQDN or
straight IP ?
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS eBootcamp 2021
https://opensips.org/training/OpenSIPS_eBootcamp_2021/
On 12/16/21 9:43 AM, Schneur Rosenberg wrote:
> Hi Bogdan
>
> I think I found the issue, I recently added these lines of code,
> because of a probing issue I was having, I just searched from my
> previous tickets and I see that you have warned me about the
> implications but for some reason I never read the message.
>
> Here is the old ticket
> https://www.mail-archive.com/users@lists.opensips.org/msg43301.html
> the reason I'm using INVITE to probe is because I want the servers
> that were probed not only to respond but also check if the database is
> working, I did it this way because I had cases where mysql crashed but
> my asterisk servers were still responding to the probe but all of the
> calls just hung, so I do a invite and it does a DB lookup and it will
> only return a positive message if it was able to query the DB, do you
> have a better solution? at the time I set it up I couldn't run a query
> on receipt of a OPTIONS but perhaps I didn't look good enough :-),
> either way can I do anything to make sure this code doesn't leak
> memory? this probing has worked for years until I needed the Contact
> header.
>
> local_route {
> if (is_method("INVITE")&& $fU=="pingTest"){
> $ru="sip:s@"+$rd ;
> append_hf("Contact: <sip:pingTest@$fd:5060>\r\n");
> exit;
> }
> }
>
> On Fri, Dec 10, 2021 at 2:16 PM Schneur Rosenberg
> <rosenberg11219 at gmail.com> wrote:
>> Hi Bogdan,
>>
>> I did it on a backup server, its also leaking memory but at a slower
>> pace, I'm attaching the logs when running kill -SIGUSR1 on the pid
>> that's growing in size, it still has available memory, I hop this will
>> give you a clue.
>>
>> Here is a pastbin to the loggs https://pastebin.com/KJVb9Y75
>>
>> On Fri, Dec 10, 2021 at 11:00 AM Schneur Rosenberg
>> <rosenberg11219 at gmail.com> wrote:
>>> Thank you, does this reduce performance? can I leave it enabled on a
>>> production machine? I will wait for the memory leak to be apparent and
>>> I'll post the result.
>>>
>>>
>>> On Thu, Dec 9, 2021 at 12:31 PM Bogdan-Andrei Iancu <bogdan at opensips.org> wrote:
>>>> Hi Schneur,
>>>>
>>>> Just follow the
>>>> https://www.opensips.org/Documentation/TroubleShooting-OutOfMem and
>>>> provide the dump. This is the only way to investigate this.
>>>>
>>>> Regards,
>>>>
>>>> Bogdan-Andrei Iancu
>>>>
>>>> OpenSIPS Founder and Developer
>>>> https://www.opensips-solutions.com
>>>> OpenSIPS eBootcamp 2021
>>>> https://opensips.org/training/OpenSIPS_eBootcamp_2021/
>>>>
>>>> On 12/8/21 12:14 PM, Schneur Rosenberg wrote:
>>>>> I just noticed that process 88 runs the timer handler, perhaps this
>>>>> might shed light on whats going on.
>>>>>
>>>>> opensipsctl fifo ps
>>>>> Process:: ID=88 PID=5327 Type=Timer handler
>>>>>
>>>>> On Wed, Dec 8, 2021 at 10:55 AM Schneur Rosenberg
>>>>> <rosenberg11219 at gmail.com> wrote:
>>>>>> Now a few hours later this is what I'm getting
>>>>>> Dec 8 09:50:13 /sbin/opensips[21699]: ERROR:nathelper:nh_timer: out
>>>>>> of pkg memory
>>>>>> Dec 8 09:50:16 /sbin/opensips[21699]: WARNING:core:fm_malloc: not
>>>>>> enough continuous free pkg memory (3024 bytes left, need 5128),
>>>>>> attempting defragmentation... please increase the "-M" command line
>>>>>> parameter!
>>>>>> Dec 8 09:50:16 /sbin/opensips[21699]: ERROR:core:fm_malloc: not
>>>>>> enough free pkg memory (3024 bytes left, need 5128), please increase
>>>>>> the "-M" command line parameter!
>>>>>>
>>>>>> Here is the last 20 package memory max_used_size
>>>>>> pkmem:70-max_used_size:: 1009584
>>>>>> pkmem:71-max_used_size:: 1009584
>>>>>> pkmem:72-max_used_size:: 1009584
>>>>>> pkmem:73-max_used_size:: 1009584
>>>>>> pkmem:74-max_used_size:: 1009584
>>>>>> pkmem:75-max_used_size:: 1009584
>>>>>> pkmem:76-max_used_size:: 1009584
>>>>>> pkmem:77-max_used_size:: 1009584
>>>>>> pkmem:78-max_used_size:: 1009584
>>>>>> pkmem:79-max_used_size:: 1009584
>>>>>> pkmem:80-max_used_size:: 1044752
>>>>>> pkmem:81-max_used_size:: 1075552
>>>>>> pkmem:82-max_used_size:: 1116848
>>>>>> pkmem:83-max_used_size:: 1117456
>>>>>> pkmem:84-max_used_size:: 1102640
>>>>>> pkmem:85-max_used_size:: 1306992
>>>>>> pkmem:86-max_used_size:: 1706304
>>>>>> pkmem:87-max_used_size:: 2507000
>>>>>> pkmem:88-max_used_size:: 4194264
>>>>>> pkmem:89-max_used_size:: 1009584
>>>>>>
>>>>>> And here is the real used size, you can see that process 88 maxed out
>>>>>> pkmem:69-real_used_size:: 975528
>>>>>> pkmem:70-real_used_size:: 978016
>>>>>> pkmem:71-real_used_size:: 989592
>>>>>> pkmem:72-real_used_size:: 951416
>>>>>> pkmem:73-real_used_size:: 982496
>>>>>> pkmem:74-real_used_size:: 965744
>>>>>> pkmem:75-real_used_size:: 959424
>>>>>> pkmem:76-real_used_size:: 949472
>>>>>> pkmem:77-real_used_size:: 983080
>>>>>> pkmem:78-real_used_size:: 961400
>>>>>> pkmem:79-real_used_size:: 977808
>>>>>> pkmem:80-real_used_size:: 978928
>>>>>> pkmem:81-real_used_size:: 1009936
>>>>>> pkmem:82-real_used_size:: 1110760
>>>>>> pkmem:83-real_used_size:: 1116720
>>>>>> pkmem:84-real_used_size:: 1096568
>>>>>> pkmem:85-real_used_size:: 1300592
>>>>>> pkmem:86-real_used_size:: 1699648
>>>>>> pkmem:87-real_used_size:: 2501096
>>>>>> pkmem:88-real_used_size:: 4191280
>>>>>> pkmem:89-real_used_size:: 882528
>>>>>>
>>>>>> On Tue, Dec 7, 2021 at 7:53 PM Schneur Rosenberg
>>>>>> <rosenberg11219 at gmail.com> wrote:
>>>>>>> Hi, lately I'm getting these errors in my logs.
>>>>>>>
>>>>>>> ERROR:core:fm_malloc: not enough free pkg memory (1792 bytes left,
>>>>>>> need 2184), please increase the "-M" command line para
>>>>>>> meter!
>>>>>>>
>>>>>>> CRITICAL:core:hostent_cpy: pkg memory allocation failure
>>>>>>>
>>>>>>> ERROR:nathelper:nh_timer: out of pkg memory
>>>>>>>
>>>>>>> ERROR:core:fm_malloc: not enough free pkg memory (5952 bytes left,
>>>>>>> need 5408), please increase the "-M" command line para
>>>>>>> meter!
>>>>>>>
>>>>>>> I was on version 2.4.8 and I upgraded to 2.4.11 and I'm monitoring the
>>>>>>> max_used_size of the package memory, a few hours later I see that 2
>>>>>>> processes keep on getting bigger, so far the rest are pretty stable, I
>>>>>>> have 90 processes and 87 and 88 are growing.
>>>>>>>
>>>>>>> here you can see the last few processes, OpenSIPS set aside 4 mb per process.
>>>>>>>
>>>>>>> pkmem:80-max_used_size:: 1009584
>>>>>>> pkmem:81-max_used_size:: 1009584
>>>>>>> pkmem:82-max_used_size:: 1009584
>>>>>>> pkmem:83-max_used_size:: 1009584
>>>>>>> pkmem:84-max_used_size:: 1009584
>>>>>>> pkmem:85-max_used_size:: 1009584
>>>>>>> pkmem:86-max_used_size:: 1143608
>>>>>>> pkmem:87-max_used_size:: 1323256
>>>>>>> pkmem:88-max_used_size:: 1831928
>>>>>>> pkmem:89-max_used_size:: 1009584
>>>>>>>
>>>>>>> Any hints where to start looking besides the solutions fund here.
>>>>>>>
>>>>>>> https://www.opensips.org/Documentation/TroubleShooting-OutOfMem
>>>>>>>
>>>>>>> thank you
>>>>>>> Scott
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opensips.org
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
More information about the Users
mailing list