[OpenSIPS-Devel] [ opensips-Bugs-3125596 ] mem leak

SourceForge.net noreply at sourceforge.net
Tue Jun 28 10:08:40 CEST 2011


Bugs item #3125596, was opened at 2010-12-02 17:39
Message generated for change (Comment added) made by bogdan_iancu
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3125596&group_id=232389

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: core
Group: 1.6.x
Status: Open
Resolution: Accepted
Priority: 7
Private: No
Submitted By: thrillerbee (thr1ll3rb33)
Assigned to: Bogdan-Andrei Iancu (bogdan_iancu)
Summary: mem leak 

Initial Comment:
Errors in the syslog this time right before the crash:
/usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left
kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000]

I do use the flat_rotate MI command with the acc module for cdr collection.  The backtrace is attached.

mem dumps are here:
http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz
http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz

----------------------------------------------------------------------

>Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2011-06-28 11:08

Message:
Hi, 

What is the status on this ?

Regards,
Bogdan

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2011-01-07 21:15

Message:
Please let me know if I can provide any further information.  This is a
critical issue for my network.  Thanks.

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-29 23:53

Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101229_2.gz

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-29 16:51

Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101229_1.gz

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-29 01:14

Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101228_3.gz

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-28 23:10

Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101228_2.zip

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-28 20:41

Message:
Bogdan,

Here's a new memdump after sending SIGUSR1 to opensips while it was out of
memory and failing to route calls (so, during runtime):
http://www.starviewconnect.com/tmp/mem_dump_20101228.zip

Thanks!

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-28 17:33

Message:
a shutdown dump is not relevant - try to make a runtime dump by sending
SIGUSR1 signal to opensips - wait for couple of hours of run and make the
dump.

Thanks and regards,
Bogdan

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-22 16:18

Message:
The server crashed (so, on shutdown).

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-22 13:48

Message:
Is this a runtime dump or a shutdown one ?

Regards,
Bogdan

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-21 21:41

Message:
my apologies for the previous comment; here it is cleaned up:

 Memory status (shm):
 qm_status (0x7f1c48c82000):
  heap size= 536870912
  used= 13760, used+overhead=21459048, free=515411864
  max used (+overhead)= 75080016  
 dumping all alloc'ed. fragments:
       0. N  address=0x7f1c48cb7518 frag=0x7f1c48cb74e8 size=8 used=1
             alloc'd from mem/shm_mem.c: shm_mem_init_mallocs(196)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     396. N  address=0x7f1c48f682b0 frag=0x7f1c48f68280 size=40 used=1
             alloc'd from lb_data.c: load_lb_data(52)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     536. N  address=0x7f1c48f6d708 frag=0x7f1c48f6d6d8 size=6192 used=1
             alloc'd from hash.c: new_subnet_table(276)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     537. N  address=0x7f1c48f6ef98 frag=0x7f1c48f6ef68 size=6192 used=1
             alloc'd from hash.c: new_subnet_table(276)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     538. N  address=0x7f1c48f70828 frag=0x7f1c48f707f8 size=8 used=1
             alloc'd from address.c: init_address(320)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     539. N  address=0x7f1c48f70890 frag=0x7f1c48f70860 size=48 used=1
             alloc'd from hash.c: subnet_table_insert(318)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     540. N  address=0x7f1c48f70920 frag=0x7f1c48f708f0 size=8 used=1
             alloc'd from hash.c: subnet_table_insert(329)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     586. N  address=0x7f1c48f72ce8 frag=0x7f1c48f72cb8 size=960 used=1
             alloc'd from core_stats.c: init_pkg_stats(182)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     587. N  address=0x7f1c48f73108 frag=0x7f1c48f730d8 size=160 used=1
             alloc'd from core_stats.c: init_pkg_stats(183)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     81089. N  address=0x7f1c4a0cdc60 frag=0x7f1c4a0cdc30 size=24 used=1
             alloc'd from proxy.c: hostent_shm_cpy(83)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     128616. N  address=0x7f1c4b2f1c90 frag=0x7f1c4b2f1c60 size=72 used=1
             alloc'd from t_fwd.c: shm_clone_proxy(233)
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     176525. N  address=0x7f1c4c8317e8 frag=0x7f1c4c8317b8 size=16 used=1
             alloc'd from mem/shm_mem.c: _shm_resize(112) 
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     201764. N  address=0x7f1c4cdffb50 frag=0x7f1c4cdffb20 size=16 used=1
             alloc'd from mem/shm_mem.c: _shm_resize(112) 
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     214714. N  address=0x7f1c4d06ed58 frag=0x7f1c4d06ed28 size=16 used=1
             alloc'd from mem/shm_mem.c: _shm_resize(112) 
         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-21 21:39

Message:
/usr/local/sbin/opensips[13045]: Memory status (shm):
/usr/local/sbin/opensips[13045]: qm_status (0x7f1c48c82000):
/usr/local/sbin/opensips[13045]:  heap size= 536870912  
/usr/local/sbin/opensips[13045]:  used= 13760, used+overhead=21459048,
free=515411864
/usr/local/sbin/opensips[13045]:  max used (+overhead)= 75080016 
/usr/local/sbin/opensips[13045]: dumping all alloc'ed. fragments:
/usr/local/sbin/opensips[13045]:       0. N  address=0x7f1c48cb7518
frag=0x7f1c48cb74e8 size=8 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from mem/shm_mem.c:
shm_mem_init_mallocs(196)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     396. N  address=0x7f1c48f682b0
frag=0x7f1c48f68280 size=40 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from lb_data.c:
load_lb_data(52)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     536. N  address=0x7f1c48f6d708
frag=0x7f1c48f6d6d8 size=6192 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from hash.c:
new_subnet_table(276) 
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     537. N  address=0x7f1c48f6ef98
frag=0x7f1c48f6ef68 size=6192 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from hash.c:
new_subnet_table(276) 
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     538. N  address=0x7f1c48f70828
frag=0x7f1c48f707f8 size=8 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from address.c:
init_address(320)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     539. N  address=0x7f1c48f70890
frag=0x7f1c48f70860 size=48 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from hash.c:
subnet_table_insert(318)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     540. N  address=0x7f1c48f70920
frag=0x7f1c48f708f0 size=8 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from hash.c:
subnet_table_insert(329)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     586. N  address=0x7f1c48f72ce8
frag=0x7f1c48f72cb8 size=960 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from core_stats.c:
init_pkg_stats(182)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     587. N  address=0x7f1c48f73108
frag=0x7f1c48f730d8 size=160 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from core_stats.c:
init_pkg_stats(183)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     81089. N  address=0x7f1c4a0cdc60
frag=0x7f1c4a0cdc30 size=24 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from proxy.c:
hostent_shm_cpy(83)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     128616. N  address=0x7f1c4b2f1c90
frag=0x7f1c4b2f1c60 size=72 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from t_fwd.c:
shm_clone_proxy(233)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     176525. N  address=0x7f1c4c8317e8
frag=0x7f1c4c8317b8 size=16 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from mem/shm_mem.c:
_shm_resize(112)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     201764. N  address=0x7f1c4cdffb50
frag=0x7f1c4cdffb20 size=16 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from mem/shm_mem.c:
_shm_resize(112)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]:     214714. N  address=0x7f1c4d06ed58
frag=0x7f1c4d06ed28 size=16 used=1
/usr/local/sbin/opensips[13045]:             alloc'd from mem/shm_mem.c:
_shm_resize(112)
/usr/local/sbin/opensips[13045]:         start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-14 18:19

Message:
your makfile.defs was not correct (makefile syntax errors) - use the
attached one (I fixed the file you uploaded).

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-14 18:06

Message:
Makefile.defs is attached

opensips -V
version: opensips 1.6.3-notls (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
SHM_MMAP, PKG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
svnrevision: 2:7290M
@(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
main.c compiled on 22:47:42 Dec  8 2010 with gcc 4.3.2

Relevant config:
debug=2
memlog=10
memdump=1

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-14 16:29

Message:
Your opensips is still not compiled with mem debugging :(...The dump from
memory debugger looks like:
     0(17665) Memory status (shm):
     0(17665) qm_status (0xb5a7e000):
     0(17665)  heap size= 33554432
     0(17665)  used= 1592952, used+overhead=1811564, free=31742868
     0(17665)  max used (+overhead)= 1811564
     0(17665) dumping all alloc'ed. fragments:
     0(17665)       0. N  address=0xb5ab240c frag=0xb5ab23f4 size=4
used=1
     0(17665)             alloc'd from mem/shm_mem.c:
shm_mem_init_mallocs(199)
     0(17665)         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
     0(17665)       1. N  address=0xb5ab2440 frag=0xb5ab2428 size=4 used=1

     0(17665)             alloc'd from timer.c: init_timer(52)
     0(17665)         start check=f0f0f0f0, end check= c0c0c0c0, abcdefed

Just to check the flags you use, please upload your Makefile.defs and post
the output of "opensips  -V" .

Thanks and regards,
Bogdan

----------------------------------------------------------------------

Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-13 16:56

Message:
The requested mem dump can be downloaded here:
http://www.starviewconnect.com/tmp/mem_dump_20101213.zip

Thanks.

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-03 14:28

Message:
Hi,

Indeed, the logs show a leak and I suspect it is from db_flatstore module
- unfortunately you do not have properly compiled the memory debugging
support, so it hard to pinpoint the guilty code.
I tried to reproduce ACC via flatstore, but not able to get a leak at
all...

So, please compile the memory debugger as per
http://www.opensips.org/Resources/DocsTsMem  (be sure you put DBG_QM_MALLOC
and take out F_MALLOC ) and recompile + reinstall +  restart opensips.

In your cfg use:
memlog=10
memdump=1
This will generate only a memory dump without any additional memory logs
at rutime.

You can take a mem dump at runtime by sending a SIGUSR1 signal to an
opensips process,
Regards,
Bogdan

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3125596&group_id=232389



More information about the Devel mailing list