[OpenSIPS-Users] Chasing down unreleased memory

[Jock McKechnie]

Thank you very much, Răzvan;

We are using the compression module for mc_compact(). We use SIP
headers for passing around data (don't ask) and the compact headers
are mandatory or else we hit MTU/fragmentation problems. We're on the
cusp of moving to SIP/TCP but haven't gotten there quite yet.
The mc_compact() lives in the main route[1] block and is simply this, in whole:
    mc_compact("P-Asserted-Identity|Remote-Party-ID|X-UCID");

And my apologies for not including the shm dump segment - please find it below.
https://pastebin.com/itSTER7Y

Let me know what else I can provide, this is fairly easy to replicate
(although it takes about an hour of runtime pummeling it to get nice
easy to read dumps). Thank you, again, very much.

 - Jock

On Fri, Sep 21, 2018 at 3:01 AM, Răzvan Crainea <razvan at opensips.org> wrote:
> Hi, Jock!
>
> I think you are actually heating two memory leaks: one in private memory and
> one in shared memory.
> In the memory dump you sent I can spot a pkg mem related to (I believe) the
> compression module. Can you tell me what functions of the compression module
> you are using and what flags are you using for them?
>
> Besides that, you claim there is a shared memory leak as well. Can you send
> us the logs for the shared memory? They are dumped by the first process
> (attendant) and should start with the string "Memory status (shm):". Can you
> post those logs as well?
>
> Best regards,
> Razvan
>
>
> On 9/20/18 9:43 PM, Jock McKechnie wrote:
>>
>> Greetings OpenSIPS gurus;
>>
>> I've run into an issue that I'm not making any real headway with on my
>> own. I have an OpenSIPS 2.4.2 running on CentOS7 that does not appear
>> to be releasing the bulk of its shared memory when calls die off.
>> Eventually OpenSIPS runs itself out of shared memory and goes beserk
>> when it can't reclaim memory for new transactions, or I've also seen
>> the kernel oomkiller take it down.
>>
>> The setup is, as mentioned, a 2.4.2 which has a chunk of perl living
>> on its back that handles some API directed call decisions. Because of
>> the inevitable delay in the API lookups I am running a large number of
>> threads (100) to ensure that I don't start dropping packets under high
>> levels of load (50 CPS+ - until I bumped children counts up I was
>> seeing packets being ignored which got the End User very excitable). I
>> appreciate having lots of children amplifies memory usage but my
>> belief is during regular load on our side (~6000 calls at ~30 - 40CPS)
>> I should have a full day's runtime easily w/o running into problems.
>> The issue is that because the memory is not released overnight it
>> starts the next day with half its shared memory gone, and then things
>> go to pot the next day when it runs through all its remaining shared
>> memory.
>>
>> I have it configured for a very large (I think) amount of memory to
>> attempt to compensate short-term - 2048MB shared memory, 128MB package
>> memory. This is not a well thought-out number, just me throwing RAM at
>> the problem. (For reference this is on a VM running with 4 cores and
>> 4096MB of RAM)
>>
>> I have graphed the memory stats out of the FIFO, after enabling
>> memstats and DBG_MALLOC/etc. The graphs can be found here:
>> https://imgur.com/a/oapKYJW
>>
>> You'll see that the first graph, the main shmem:, shows the usage gets
>> tied up and remains after calls die off (these examples run for 20
>> minutes after the calls stopped - but I can leave it overnight and the
>> shmem: usage won't drop). This being said, the other graphed modules
>> show that it releases memory correctly after calls quit like they
>> should. None of the other modules graphed (eg, _all_ of the modules I
>> was using in OpenSIPS) showed no significant usage and I've omitted
>> them.
>>
>> Interesting that the 'perl' module shows no stats ever which makes me
>> wonder if the stats on this module works. I am not storing anything
>> inside perl, besides lots of local session variables, and all channel
>> variables are pushed back into OpenSIPS AVPs, but I assume there must
>> be some memory overhead, although it may not significantly change.
>>
>> When I shut OpenSIPS down I get child dumps like the following: (I've
>> included only one for brevity, but they're all much of a muchness)
>> https://pastebin.com/KG2qxxBa
>>
>> If I'm reading this right (and there's a strong possibility I'm
>> misunderstanding what I'm looking at) the memory is tied up in io_wait
>> and the parser. I'm not sure what this signifies, however.
>>
>> So I guess what I'm asking is:
>> Am I understanding what I'm looking at correctly and that OpenSIPS is
>> not releasing its shared memory which causes it to run out of memory
>> quicker the following day? (Which lines up with the symptoms of
>> shorter runtimes before failure)
>>
>> Based on the dumps where should I look for the source of this issue?
>>
>> Finally, and unrelated to the above, I attempted to mail the official
>> OpenSIPS peeps to talk about paid support but none of the 'contact us'
>> submits appear to work - when I click on the submit button it changes
>> shade but does not appear to act. Certainly I never received any
>> response. If there was a direct eMail address I could send to that
>> would be preferable.
>>
>> As always, my thanks for your time.
>>
>>   - Jock
>>
>> PS -
>> List of modules I'm using in this config:loadmodule "signaling.so"
>> loadmodule "sl.so"
>> loadmodule "tm.so"
>> loadmodule "rr.so"
>> loadmodule "maxfwd.so"
>> loadmodule "textops.so"
>> loadmodule "uri.so"
>> loadmodule "mi_fifo.so"
>> loadmodule "siptrace.so"
>> loadmodule "sipmsgops.so"
>> loadmodule "dialog.so"
>> loadmodule "uac_auth.so"
>> loadmodule "uac.so"
>> loadmodule "perl.so"
>> loadmodule "acc.so"
>> loadmodule "proto_hep.so"
>> loadmodule "proto_udp.so"
>> loadmodule "compression.so"
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>
> --
> Răzvan Crainea
> OpenSIPS Core Developer
>   http://www.opensips-solutions.com
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users