Sorry for the confusion. Orginally when I was testing my opensips config with the dialoginfo_set() function used in the script I was running into opensips memory (pk and sh) running out. This is most likely due to the fact that a ton of records were building up in the PUA table because opensips was generating pua message because of the dialoginfo_set() even though I had commented out everything else in my opensips script that had to do with pua and presence. I am hoping that when I am done with stress testing and enable all my presence and pua stuff the dialoginfo_set() functions will work as normal and not place a ton of records in the pua table (something I will need to verify later on).
<br />
<br />I think where I was getting confused was that when I did the top command I would see that my opensips processes were building up memory under the %MEM column and that number under %MEM was never decreasing after I waited 20 minutes. Yet last night after doing more research via the mailing list I ran across the command
<br />
<br />opensipsctl fifo get_statistics pkmem: shmem:
<br />
<br />When I look at the info from that commands output I see that SHMEM does get memory back instead of just steadily running out. I also see that with PKMEM all my processes are using about 1M of memory with 3M left over and that extra free memory doesn't get touched. So that leads me to believe that my opensips processes have also stabilized and aren't just gobbling up memory until it runs out. So even though the TOP command appears to show that the opensips processes aren't releasing that %MEM I can see with the opensipsctl fifo get_statistics command that pkmem and shmem isn't growing and being overloaded.
<br />
<br />Does that make more since? Everything for now appears to be good. I can run my SIPP test for a very long time and OpenSIPS doesn't bomb out on me with memory issues.
<br />
<br />On Apr 6, 2011 9:10am, Bogdan-Andrei Iancu <bogdan@opensips.org> wrote:
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> Duane,
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> trying to follow what you are saying :) - the mem leak you observe
<br />>
<br />> is for system memory (when you look with top) or is opensips memory
<br />>
<br />> (private or shared) ?
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> Regards,
<br />>
<br />>
<br />> Bogdan
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> On 04/06/2011 06:18 AM, Duane Larson wrote:
<br />>
<br />>
<br />>
<br />> OK after more testing it looks like my current test config
<br />>
<br />> with dialoginfo_set(); commented out is stable. My opensips
<br />>
<br />> processes grow to around 8.1 and 7.9 via the top command and
<br />>
<br />> stay there without growing anymore. Also I just found the
<br />>
<br />> following command
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> opensipsctl fifo get_statistics pkmem: shmem:
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> Didn't know about it. After looking at that I see that I
<br />>
<br />> have a good amount of free memory with all the pkmem's and the
<br />>
<br />> shmem.
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> As for all the records in the PUA table that is probably
<br />>
<br />> because I have edited my script so much just to test calls per
<br />>
<br />> second. I had disabled all my presence processing but forgot
<br />>
<br />> to comment out the dialoginfo_set() function. Hopefully when I
<br />>
<br />> am done stress testing the PUA stuff will not be an issue.
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> On Tue, Apr 5, 2011 at 6:02 PM, Duane
<br />>
<br />> Larson duane.larson@gmail.com>
<br />>
<br />> wrote:
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> Thanks for the reply. I actually have been following the
<br />>
<br />> directions on
<br />>
<br />>
<br />>
<br />>
<br />> http://www.opensips.org/Resources/DocsTsMem
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> Perhaps my memory is building up because I am receiving a
<br />>
<br />> ton of the following in my PUA database table
<br />>
<br />>
<br />> mysql> select * from pua;
<br />>
<br />>
<br />> +--------+------------------+----------------+-------+---------+-----------------+------+------+----------+-------------+--------+---------+--------+----------+------+--------------+---------+----------------+---------+---------------+
<br />>
<br />>
<br />> | id | pres_uri | pres_id | event |
<br />>
<br />> expires | desired_expires | flag | etag | tuple_id |
<br />>
<br />> watcher_uri | to_uri | call_id | to_tag | from_tag | cseq |
<br />>
<br />> record_route | contact | remote_contact | version |
<br />>
<br />> extra_headers |
<br />>
<br />>
<br />> +--------+------------------+----------------+-------+---------+-----------------+------+------+----------+-------------+--------+---------+--------+----------+------+--------------+---------+----------------+---------+---------------+
<br />>
<br />>
<br />> | 641966 | sip:102@test.com
<br />>
<br />> | DIALOG_PUBLISH | 64 | 0 | 1302067600 | 1024
<br />>
<br />> | | | | | |
<br />>
<br />> | | 0 | | |
<br />>
<br />> | 0 | |
<br />>
<br />>
<br />> | 642014 | sip:102@test.com
<br />>
<br />> | DIALOG_PUBLISH | 64 | 0 | 1302067599 | 1024
<br />>
<br />> | | | | | |
<br />>
<br />> | | 0 | | |
<br />>
<br />> | 0 | |
<br />>
<br />>
<br />> This is because of the following function in my
<br />>
<br />> opensips.cfg script
<br />>
<br />>
<br />> dialoginfo_set(); (using this so I can have the BLF
<br />>
<br />> feature)
<br />>
<br />>
<br />> When I comment this out my script appears to run fine when
<br />>
<br />> I stress test it with SIPP. I have the following
<br />>
<br />>
<br />> children = 10
<br />>
<br />>
<br />> Server has 1 Gig of RAM
<br />>
<br />>
<br />> I configured my config.h to be "define PKG_MEM_POOL_SIZE
<br />>
<br />> 4*1024*1024"
<br />>
<br />>
<br />> I execute the Opensips service with 128M of memory
<br />>
<br />>
<br />> I run my SIPP test for about 5 minutes and 30 seconds and
<br />>
<br />> get the following outcome
<br />>
<br />>
<br />> Counter Name | Periodic value |
<br />>
<br />> Cumulative value
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Elapsed Time | 00:00:00:000 |
<br />>
<br />> 00:05:32:211
<br />>
<br />>
<br />> Call Rate | 0.000 cps |
<br />>
<br />> 18.455 cps
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Incoming call created | 0
<br />>
<br />> | 0
<br />>
<br />>
<br />> OutGoing call created | 0 |
<br />>
<br />> 6131
<br />>
<br />>
<br />> Total Call created | |
<br />>
<br />> 6131
<br />>
<br />>
<br />> Current Call | 0
<br />>
<br />> |
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Counter 5 | 0 |
<br />>
<br />> 6130
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Successful call | 0 |
<br />>
<br />> 6130
<br />>
<br />>
<br />> Failed call | 0
<br />>
<br />> | 1
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> With this test I have nothing in my PUA table. Also at the
<br />>
<br />> end of this test I see that I have the folllowing when I
<br />>
<br />> execute the "top" command
<br />>
<br />>
<br />> Mem: 1022536k total, 785916k used, 236620k free,
<br />>
<br />> 29708k buffers
<br />>
<br />>
<br />> Swap: 2097148k total, 0k used, 2097148k free,
<br />>
<br />> 457508k cached
<br />>
<br />>
<br />> PID USER PR NI VIRT RES SHR S %CPU %MEM
<br />>
<br />> TIME+ COMMAND
<br />>
<br />>
<br />> 15327 mysql 20 0 333m 183m 10m S 1 18.4 9:24.24
<br />>
<br />> mysqld
<br />>
<br />>
<br />> 1084 root 20 0 63012 15m 2788 S 0 1.5 0:00.28
<br />>
<br />> opensips-mi-pro
<br />>
<br />>
<br />> 1096 root 20 0 73296 14m 3128 S 0 1.5 0:00.89
<br />>
<br />> media-dispatche
<br />>
<br />>
<br />> 18654 opensips 20 0 283m 11m 9404 S 0 1.2 0:00.74
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18643 opensips 20 0 281m 11m 9464 S 0 1.2 0:01.07
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18647 opensips 20 0 281m 11m 9420 S 0 1.2 0:01.10
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18646 opensips 20 0 281m 11m 9376 S 0 1.2 0:01.09
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18649 opensips 20 0 281m 11m 9320 S 0 1.2 0:01.10
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18650 opensips 20 0 281m 11m 9284 S 0 1.1 0:01.12
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18651 opensips 20 0 281m 11m 9276 S 0 1.1 0:01.10
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18645 opensips 20 0 281m 11m 9272 S 0 1.1 0:01.10
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18644 opensips 20 0 281m 11m 9140 S 0 1.1 0:01.10
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18648 opensips 20 0 281m 11m 9076 S 0 1.1 0:01.08
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18652 opensips 20 0 281m 11m 8972 S 0 1.1 0:01.11
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18639 opensips 20 0 281m 8360 6096 S 0 0.8 0:00.09
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18640 opensips 20 0 283m 4120 1500 S 0 0.4 0:00.08
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18656 opensips 20 0 283m 4044 1760 S 0 0.4 0:00.12
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18641 opensips 20 0 281m 3448 1144 S 0 0.3 0:00.00
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18642 opensips 20 0 281m 3224 960 S 0 0.3 0:00.00
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18655 opensips 20 0 281m 3208 940 S 0 0.3 0:00.07
<br />>
<br />> opensips
<br />>
<br />>
<br />> 18653 opensips 20 0 281m 2320 88 S 0 0.2 0:00.51
<br />>
<br />> opensips
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> Then I run a second test with SIPP and get the following
<br />>
<br />> Counter Name | Periodic value |
<br />>
<br />> Cumulative value
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Elapsed Time | 00:00:00:000 |
<br />>
<br />> 00:13:35:590
<br />>
<br />>
<br />> Call Rate | 0.000 cps |
<br />>
<br />> 77.586 cps
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Incoming call created | 0
<br />>
<br />> | 0
<br />>
<br />>
<br />> OutGoing call created | 0 |
<br />>
<br />> 63278
<br />>
<br />>
<br />> Total Call created | |
<br />>
<br />> 63278
<br />>
<br />>
<br />> Current Call | 0
<br />>
<br />> |
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Counter 5 | 0 |
<br />>
<br />> 63201
<br />>
<br />>
<br />> -------------------------+---------------------------+--------------------------
<br />>
<br />>
<br />> Successful call | 0 |
<br />>
<br />> 63201
<br />>
<br />>
<br />> Failed call | 0 |
<br />>
<br />> 77
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> And then when I do the "top" command again I see the
<br />>
<br />> following
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> Mem: 1022536k total, 675176k used, 347360k
<br />>
<br />> free, 30956k buffers
<br />>
<br />>
<br />> Swap: 2097148k total, 0k used, 2097148k free,
<br />>
<br />> 345944k cached
<br />>
<br />>
<br />> PID USER PR NI VIRT RES SHR S %CPU %MEM
<br />>
<br />> TIME+ COMMAND
<br />>
<br />>
<br />> 15327 mysql 20 0 333m 183m 10m S 1 18.4
<br />>
<br />> 12:29.06 mysqld
<br />>
<br />>
<br />> 18643 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.51 opensips
<br />>
<br />>
<br />> 18644 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.26 opensips
<br />>
<br />>
<br />> 18647 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.42 opensips
<br />>
<br />>
<br />> 18646 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.34 opensips
<br />>
<br />>
<br />> 18648 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.41 opensips
<br />>
<br />>
<br />> 18649 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.27 opensips
<br />>
<br />>
<br />> 18650 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.32 opensips
<br />>
<br />>
<br />> 18651 opensips 20 0 281m 28m 26m S 0 2.9
<br />>
<br />> 0:12.28 opensips
<br />>
<br />>
<br />> 18645 opensips 20 0 281m 28m 26m S 0 2.8
<br />>
<br />> 0:12.35 opensips
<br />>
<br />>
<br />> 18652 opensips 20 0 281m 28m 26m S 0 2.8
<br />>
<br />> 0:12.32 opensips
<br />>
<br />>
<br />> 18654 opensips 20 0 283m 26m 24m S 0 2.7
<br />>
<br />> 0:02.51 opensips
<br />>
<br />>
<br />> 1084 root 20 0 63012 15m 2788 S 0 1.5
<br />>
<br />> 0:00.28 opensips-mi-pro
<br />>
<br />>
<br />> 1096 root 20 0 73296 14m 3128 S 0 1.5
<br />>
<br />> 0:00.89 media-dispatche
<br />>
<br />>
<br />> 18642 opensips 20 0 281m 9860 7404 S 0 1.0
<br />>
<br />> 0:00.15 opensips
<br />>
<br />>
<br />> 18639 opensips 20 0 281m 8360 6096 S 0 0.8
<br />>
<br />> 0:00.09 opensips
<br />>
<br />>
<br />> 18640 opensips 20 0 283m 4192 1572 S 0 0.4
<br />>
<br />> 0:00.15 opensips
<br />>
<br />>
<br />> 18656 opensips 20 0 283m 4084 1800 S 0 0.4
<br />>
<br />> 0:00.25 opensips
<br />>
<br />>
<br />> 18641 opensips 20 0 281m 3448 1144 S 0 0.3
<br />>
<br />> 0:00.00 opensips
<br />>
<br />>
<br />> 18655 opensips 20 0 281m 3208 940 S 0 0.3
<br />>
<br />> 0:00.15 opensips
<br />>
<br />>
<br />> 18653 opensips 20 0 281m 2320 88 S 0 0.2
<br />>
<br />> 0:01.06 opensips
<br />>
<br />>
<br />>
<br />>
<br />>
<br />> So a lot of the opensips processes have about 2.9%
<br />>
<br />> memory. I wait more than 20 minutes to see if the
<br />>
<br />> memory would be released but it didn't.
<br />>
<br />>
<br />> Not sure if this is a bad thing or not. I could let my
<br />>
<br />> SIPP test just keep running to see if OpenSIPS crashes
<br />>
<br />> from no memory if the memory keeps creeping up or I
<br />>
<br />> could do a "kill -SIGUSR1 18643" to see what is
<br />>
<br />> currently using the memory? Any thoughts or is this not
<br />>
<br />> an issue at all?
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br />>
<br