[OpenSIPS-Devel] [ opensips-Bugs-3125596 ] mem leak
SourceForge.net
noreply at sourceforge.net
Tue Jun 28 10:08:40 CEST 2011
Bugs item #3125596, was opened at 2010-12-02 17:39
Message generated for change (Comment added) made by bogdan_iancu
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3125596&group_id=232389
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: core
Group: 1.6.x
Status: Open
Resolution: Accepted
Priority: 7
Private: No
Submitted By: thrillerbee (thr1ll3rb33)
Assigned to: Bogdan-Andrei Iancu (bogdan_iancu)
Summary: mem leak
Initial Comment:
Errors in the syslog this time right before the crash:
/usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left
kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000]
I do use the flat_rotate MI command with the acc module for cdr collection. The backtrace is attached.
mem dumps are here:
http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz
http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz
----------------------------------------------------------------------
>Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2011-06-28 11:08
Message:
Hi,
What is the status on this ?
Regards,
Bogdan
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2011-01-07 21:15
Message:
Please let me know if I can provide any further information. This is a
critical issue for my network. Thanks.
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-29 23:53
Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101229_2.gz
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-29 16:51
Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101229_1.gz
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-29 01:14
Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101228_3.gz
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-28 23:10
Message:
another memdump:
http://www.starviewconnect.com/tmp/mem_dump_20101228_2.zip
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-28 20:41
Message:
Bogdan,
Here's a new memdump after sending SIGUSR1 to opensips while it was out of
memory and failing to route calls (so, during runtime):
http://www.starviewconnect.com/tmp/mem_dump_20101228.zip
Thanks!
----------------------------------------------------------------------
Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-28 17:33
Message:
a shutdown dump is not relevant - try to make a runtime dump by sending
SIGUSR1 signal to opensips - wait for couple of hours of run and make the
dump.
Thanks and regards,
Bogdan
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-22 16:18
Message:
The server crashed (so, on shutdown).
----------------------------------------------------------------------
Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-22 13:48
Message:
Is this a runtime dump or a shutdown one ?
Regards,
Bogdan
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-21 21:41
Message:
my apologies for the previous comment; here it is cleaned up:
Memory status (shm):
qm_status (0x7f1c48c82000):
heap size= 536870912
used= 13760, used+overhead=21459048, free=515411864
max used (+overhead)= 75080016
dumping all alloc'ed. fragments:
0. N address=0x7f1c48cb7518 frag=0x7f1c48cb74e8 size=8 used=1
alloc'd from mem/shm_mem.c: shm_mem_init_mallocs(196)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
396. N address=0x7f1c48f682b0 frag=0x7f1c48f68280 size=40 used=1
alloc'd from lb_data.c: load_lb_data(52)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
536. N address=0x7f1c48f6d708 frag=0x7f1c48f6d6d8 size=6192 used=1
alloc'd from hash.c: new_subnet_table(276)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
537. N address=0x7f1c48f6ef98 frag=0x7f1c48f6ef68 size=6192 used=1
alloc'd from hash.c: new_subnet_table(276)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
538. N address=0x7f1c48f70828 frag=0x7f1c48f707f8 size=8 used=1
alloc'd from address.c: init_address(320)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
539. N address=0x7f1c48f70890 frag=0x7f1c48f70860 size=48 used=1
alloc'd from hash.c: subnet_table_insert(318)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
540. N address=0x7f1c48f70920 frag=0x7f1c48f708f0 size=8 used=1
alloc'd from hash.c: subnet_table_insert(329)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
586. N address=0x7f1c48f72ce8 frag=0x7f1c48f72cb8 size=960 used=1
alloc'd from core_stats.c: init_pkg_stats(182)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
587. N address=0x7f1c48f73108 frag=0x7f1c48f730d8 size=160 used=1
alloc'd from core_stats.c: init_pkg_stats(183)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
81089. N address=0x7f1c4a0cdc60 frag=0x7f1c4a0cdc30 size=24 used=1
alloc'd from proxy.c: hostent_shm_cpy(83)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
128616. N address=0x7f1c4b2f1c90 frag=0x7f1c4b2f1c60 size=72 used=1
alloc'd from t_fwd.c: shm_clone_proxy(233)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
176525. N address=0x7f1c4c8317e8 frag=0x7f1c4c8317b8 size=16 used=1
alloc'd from mem/shm_mem.c: _shm_resize(112)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
201764. N address=0x7f1c4cdffb50 frag=0x7f1c4cdffb20 size=16 used=1
alloc'd from mem/shm_mem.c: _shm_resize(112)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
214714. N address=0x7f1c4d06ed58 frag=0x7f1c4d06ed28 size=16 used=1
alloc'd from mem/shm_mem.c: _shm_resize(112)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-21 21:39
Message:
/usr/local/sbin/opensips[13045]: Memory status (shm):
/usr/local/sbin/opensips[13045]: qm_status (0x7f1c48c82000):
/usr/local/sbin/opensips[13045]: heap size= 536870912
/usr/local/sbin/opensips[13045]: used= 13760, used+overhead=21459048,
free=515411864
/usr/local/sbin/opensips[13045]: max used (+overhead)= 75080016
/usr/local/sbin/opensips[13045]: dumping all alloc'ed. fragments:
/usr/local/sbin/opensips[13045]: 0. N address=0x7f1c48cb7518
frag=0x7f1c48cb74e8 size=8 used=1
/usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c:
shm_mem_init_mallocs(196)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 396. N address=0x7f1c48f682b0
frag=0x7f1c48f68280 size=40 used=1
/usr/local/sbin/opensips[13045]: alloc'd from lb_data.c:
load_lb_data(52)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 536. N address=0x7f1c48f6d708
frag=0x7f1c48f6d6d8 size=6192 used=1
/usr/local/sbin/opensips[13045]: alloc'd from hash.c:
new_subnet_table(276)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 537. N address=0x7f1c48f6ef98
frag=0x7f1c48f6ef68 size=6192 used=1
/usr/local/sbin/opensips[13045]: alloc'd from hash.c:
new_subnet_table(276)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 538. N address=0x7f1c48f70828
frag=0x7f1c48f707f8 size=8 used=1
/usr/local/sbin/opensips[13045]: alloc'd from address.c:
init_address(320)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 539. N address=0x7f1c48f70890
frag=0x7f1c48f70860 size=48 used=1
/usr/local/sbin/opensips[13045]: alloc'd from hash.c:
subnet_table_insert(318)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 540. N address=0x7f1c48f70920
frag=0x7f1c48f708f0 size=8 used=1
/usr/local/sbin/opensips[13045]: alloc'd from hash.c:
subnet_table_insert(329)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 586. N address=0x7f1c48f72ce8
frag=0x7f1c48f72cb8 size=960 used=1
/usr/local/sbin/opensips[13045]: alloc'd from core_stats.c:
init_pkg_stats(182)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 587. N address=0x7f1c48f73108
frag=0x7f1c48f730d8 size=160 used=1
/usr/local/sbin/opensips[13045]: alloc'd from core_stats.c:
init_pkg_stats(183)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 81089. N address=0x7f1c4a0cdc60
frag=0x7f1c4a0cdc30 size=24 used=1
/usr/local/sbin/opensips[13045]: alloc'd from proxy.c:
hostent_shm_cpy(83)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 128616. N address=0x7f1c4b2f1c90
frag=0x7f1c4b2f1c60 size=72 used=1
/usr/local/sbin/opensips[13045]: alloc'd from t_fwd.c:
shm_clone_proxy(233)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 176525. N address=0x7f1c4c8317e8
frag=0x7f1c4c8317b8 size=16 used=1
/usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c:
_shm_resize(112)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 201764. N address=0x7f1c4cdffb50
frag=0x7f1c4cdffb20 size=16 used=1
/usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c:
_shm_resize(112)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
/usr/local/sbin/opensips[13045]: 214714. N address=0x7f1c4d06ed58
frag=0x7f1c4d06ed28 size=16 used=1
/usr/local/sbin/opensips[13045]: alloc'd from mem/shm_mem.c:
_shm_resize(112)
/usr/local/sbin/opensips[13045]: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
----------------------------------------------------------------------
Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-14 18:19
Message:
your makfile.defs was not correct (makefile syntax errors) - use the
attached one (I fixed the file you uploaded).
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-14 18:06
Message:
Makefile.defs is attached
opensips -V
version: opensips 1.6.3-notls (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM,
SHM_MMAP, PKG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
svnrevision: 2:7290M
@(#) $Id: main.c 6169 2009-09-22 12:48:37Z bogdan_iancu $
main.c compiled on 22:47:42 Dec 8 2010 with gcc 4.3.2
Relevant config:
debug=2
memlog=10
memdump=1
----------------------------------------------------------------------
Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-14 16:29
Message:
Your opensips is still not compiled with mem debugging :(...The dump from
memory debugger looks like:
0(17665) Memory status (shm):
0(17665) qm_status (0xb5a7e000):
0(17665) heap size= 33554432
0(17665) used= 1592952, used+overhead=1811564, free=31742868
0(17665) max used (+overhead)= 1811564
0(17665) dumping all alloc'ed. fragments:
0(17665) 0. N address=0xb5ab240c frag=0xb5ab23f4 size=4
used=1
0(17665) alloc'd from mem/shm_mem.c:
shm_mem_init_mallocs(199)
0(17665) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
0(17665) 1. N address=0xb5ab2440 frag=0xb5ab2428 size=4 used=1
0(17665) alloc'd from timer.c: init_timer(52)
0(17665) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Just to check the flags you use, please upload your Makefile.defs and post
the output of "opensips -V" .
Thanks and regards,
Bogdan
----------------------------------------------------------------------
Comment By: thrillerbee (thr1ll3rb33)
Date: 2010-12-13 16:56
Message:
The requested mem dump can be downloaded here:
http://www.starviewconnect.com/tmp/mem_dump_20101213.zip
Thanks.
----------------------------------------------------------------------
Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2010-12-03 14:28
Message:
Hi,
Indeed, the logs show a leak and I suspect it is from db_flatstore module
- unfortunately you do not have properly compiled the memory debugging
support, so it hard to pinpoint the guilty code.
I tried to reproduce ACC via flatstore, but not able to get a leak at
all...
So, please compile the memory debugger as per
http://www.opensips.org/Resources/DocsTsMem (be sure you put DBG_QM_MALLOC
and take out F_MALLOC ) and recompile + reinstall + restart opensips.
In your cfg use:
memlog=10
memdump=1
This will generate only a memory dump without any additional memory logs
at rutime.
You can take a mem dump at runtime by sending a SIGUSR1 signal to an
opensips process,
Regards,
Bogdan
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3125596&group_id=232389
More information about the Devel
mailing list