[OpenSIPS-Users] Quest to find memory leak

Mon Mar 6 13:09:30 EST 2017

here is another trace
http://pastebin.com/9Ge2NEVQ

I see lot of alloc request but no free.

On Mon, Mar 6, 2017 at 6:57 PM, John Nash <john.nash778 at gmail.com> wrote:

> Ok will try that. Is it possible that wrong usage of drouting may cause
> this to happen instead of actual leak?... What are the things private
> memory is used for?
>
> On Mon, Mar 6, 2017 at 6:48 PM, Răzvan Crainea <razvan at opensips.org>
> wrote:
>
>> Hi, John!
>>
>> From the dump you sent, I don't see any leaks. Perhaps some of those
>> fragments increase over time. Can you make a memory dump after the server
>> runs some time, like after it gets 100 messages?
>>
>> Best regards,
>>
>> Răzvan Crainea
>> OpenSIPS Solutionswww.opensips-solutions.com
>>
>> On 03/06/2017 03:02 PM, John Nash wrote:
>>
>> Here is the dump
>> http://pastebin.com/DTEHF5Vc
>>
>> On Mon, Mar 6, 2017 at 6:20 PM, Răzvan Crainea <razvan at opensips.org>
>> wrote:
>>
>>> None of the "actions" you are talking about have big impact on private
>>> memory, but the shared one. Better do the dump and send it over to point
>>> out what is "eating" memory.
>>>
>>> Best regards,
>>>
>>> Răzvan Crainea
>>> OpenSIPS Solutionswww.opensips-solutions.com
>>>
>>> On 03/06/2017 02:39 PM, John Nash wrote:
>>>
>>> with every call attempt it decreases. I tried some changes by rejecting
>>> invite before drouting call (That means after auth , dispatcher) and found
>>> memory is stable but when drouting sends Invite to external gateway and
>>> external gateway rejects it. Then this issue happens.
>>>
>>> Inuse transactions and active dialogs also 0. Somthing wrong happening
>>> in handling of failure replies. But apart from use_next_gw and setting
>>> some avps for CDR not much going on there.
>>>
>>> On Mon, Mar 6, 2017 at 5:54 PM, Răzvan Crainea <razvan at opensips.org>
>>> wrote:
>>>
>>>> Ok, so it is the first listener for the private IP that leaks. Next, is
>>>> the memory stabilizing in time? Or it is continously decreasing?
>>>> Yes, that's how you should make the dump.
>>>>
>>>> Best regards,
>>>>
>>>> Răzvan Crainea
>>>> OpenSIPS Solutionswww.opensips-solutions.com
>>>>
>>>> On 03/06/2017 10:57 AM, John Nash wrote:
>>>>
>>>> Dear Razvan,
>>>>
>>>> Below is the info on my processes
>>>> Process::  ID=0 PID=17351 Type=attendant
>>>> Process::  ID=1 PID=17352 Type=MI FIFO
>>>> Process::  ID=2 PID=17353 Type=MI Datagram
>>>> Process::  ID=3 PID=17354 Type=time_keeper
>>>> Process::  ID=4 PID=17355 Type=timer
>>>> Process::  ID=5 PID=17356 Type=SIP receiver udp:1.1.1.1:9094
>>>> Process::  ID=6 PID=17357 Type=SIP receiver udp:1.1.1.1:5060
>>>> Process::  ID=7 PID=17358 Type=SIP receiver udp:192.168.45.5:5064
>>>> Process::  ID=8 PID=17359 Type=Timer handler
>>>>
>>>> 1.1.1.1 is public IP (I changed). The decrease in memory I see is for
>>>> Process::  ID=7 PID=17358 mainly. My call flow is as following
>>>>
>>>> - New Invite hits the opensips on 1.1.1.1:9094
>>>> - Apart from message validity checks I query DB to check if its a valid
>>>> user (Using local cache also there)
>>>> - Create dialog, Topology_hiding functions are called along with some
>>>> avp population
>>>> - Using dispatcher ds_select_domain Call sent to udp:192.168.45.2:7060
>>>> (using force socket). This 192.168.45.2:7060 is actually freeswitch
>>>> - Call again comes back to opensips on udp:192.168.45.5:5064
>>>> - New dialog is created and topology_hiding is called
>>>> - Drouting function do_routing is called which tries one gateway and
>>>> fails
>>>>
>>>>
>>>> Dump i need to create with memlog=4 memdump=1 right?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Mar 6, 2017 at 2:05 PM, Răzvan Crainea <razvan at opensips.org>
>>>> wrote:
>>>>
>>>>> Hi, John!
>>>>>
>>>>> Transactions are stored in shared memory, not in the private one. So
>>>>> the possible leak you are facing its not related to transactions.
>>>>> During runtime, OpenSIPS might resize some internal structures, which
>>>>> may lead to increase memory usage. However, after a while, these
>>>>> allocations should stabilize .
>>>>> Can you post the output of the kill -SIGUSR1 on pastebin so we can
>>>>> take a look? Also, what type of process is the one you are seeing the leak
>>>>> into? You can find out using the 'opensipsctl ps' command.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Răzvan Crainea
>>>>> OpenSIPS Solutionswww.opensips-solutions.com
>>>>>
>>>>> On 03/06/2017 09:55 AM, John Nash wrote:
>>>>>
>>>>> I am using OpenSIPS (2.1.5 (x86_64/linux)) in production. I observed
>>>>> private memory is decreasing constantly for one process mainly and
>>>>> ultimately leading to memory errors and crash.
>>>>>
>>>>> To debug this issue I prepared a test server and compiled opensips as
>>>>> per https://www.opensips.org/Documentation/TroubleShooting-OutOfMem
>>>>>
>>>>> I made only one single call (which was rejected by opensips as it was
>>>>> not authorized user) and I saw private free memory decreased. I was hoping
>>>>> since transaction is done ideally it should release memory and should show
>>>>> me same memory as startup but it did not. I verified this with many call
>>>>> attempts and i see free memory is always decreasing slowly.
>>>>>
>>>>> I used kill -SIGUSR1 <lowest pid> to create memory dump. But i am
>>>>> unable to make sense of it. It shows log like ...
>>>>>
>>>>> r  6 07:29:19 Server3021 opensips[13276]: Memory status (pkg):
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]: qm_status (0x7f5b8ebba010):
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]:  heap size= 4194304
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]:  used= 346768,
>>>>> used+overhead=848792, free=3345512
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]:  max used (+overhead)=
>>>>> 931920
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]: dumping all alloc'ed.
>>>>> fragments:
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]:       0. N
>>>>>  address=0x7f5b8ebef528 frag=0x7f5b8ebef4f8 size=40 used=1
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]:             alloc'd from
>>>>> script_cb.c: add_callback(60)
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]:         start
>>>>> check=f0f0f0f0f0f0f0f0, end check= c0c0c0c0c0c0c0c0, abcdefedabcdefed
>>>>> Mar  6 07:29:19 Server3021 opensips[13276]:       1. N
>>>>>  address=0x7f5b8ebef5b0
>>>>>
>>>>> I pasted only few lines in this mail. What should be my next
>>>>> step?...How can i really trace what is wrong in my script or any other
>>>>> memory leak?
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>> _______________________________________________ Users mailing list
>>>>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>>>>> bin/mailman/listinfo/users
>>>>
>>>> _______________________________________________
>>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>> _______________________________________________ Users mailing list
>>>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>>>> bin/mailman/listinfo/users
>>>
>>> _______________________________________________
>>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>> _______________________________________________ Users mailing list
>>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>>> bin/mailman/listinfo/users
>>
>> _______________________________________________
>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20170306/5c61d433/attachment-0001.html>