[OpenSIPS-Devel] [ opensips-Bugs-3557213 ] Stuck Transactions/Memory Leak with B2B top hiding

SourceForge.net noreply at sourceforge.net
Mon Sep 24 18:27:06 CEST 2012


Bugs item #3557213, was opened at 2012-08-13 22:38
Message generated for change (Comment added) made by rrb3942
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3557213&group_id=232389

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: modules
Group: 1.8.x
Status: Open
Resolution: None
Priority: 8
Private: No
Submitted By: Ryan Bullock (rrb3942)
Assigned to: Bogdan-Andrei Iancu (bogdan_iancu)
Summary: Stuck Transactions/Memory Leak with B2B top hiding

Initial Comment:
I am experiencing a memory leak when using the B2B modules with the internal "top hiding" scenario. It appears to be related to struck/un-freed transactions, as the opensips mi will report many thousands of inuse_transactions when the reality is only a handful. An example is we currently show 15000+ inuse_transactions while only a handful of calls are actually in progress. These transactions appear to build up over time.

Opensips information:
version: opensips 1.8.0-notls (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
svnrevision: 2:9164M
@(#) $Id: main.c 8772 2012-03-08 11:16:13Z bogdan_iancu $
main.c compiled on 12:24:36 Aug  3 2012 with gcc 4.4.6

I currently have the memdump option enabled and am waiting for another buildup of transactions to gather the memdump information. I can supply access to the dump and the opensips configuration file via email.

----------------------------------------------------------------------

>Comment By: Ryan Bullock (rrb3942)
Date: 2012-09-24 09:27

Message:
Hey Bogdan,

I tried the patch, but the compiler complained that tm might be
uninitialized and it caused a segfault when I tried to run it. I have the
core dump if you would like a backtrace.

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2012-09-19 08:50

Message:
Just uploaded a patch here - it is not a fix, but rather something to help
me with the fix - trying to collect more info.

Please apply this patch, run your b2b and send me logs (debug 3 is ok). I'm
looking for messages on CRITICAL level  with "Transaction not replied" +
any previous messages from the same process.

Thanks and regards,

----------------------------------------------------------------------

Comment By: Jim OBrien (jimdoesvoip)
Date: 2012-08-20 01:31

Message:
I may be seeing this same behavior as well.  I had thought this was related
to TLS / many client sessions all connecting at once as things failed over
from one server to another.  Are you both running TCP?  I can see that Ryan
is not running TLS from the description above.  We use opensips as a proxy;
not b2bua, it is doing far end NAT...

The symptom that I see is that as tm:UAS_transactions increases, so does
tm:inuse_transactions, following it very closely.  Eventually when this
happens, the processes run out of memory / things go bad.



----------------------------------------------------------------------

Comment By: Ryan Bullock (rrb3942)
Date: 2012-08-14 21:44

Message:
Config file was sent to both Vlad and Bogdan via email earlier today and I
just sent login information to obtain the memdump as well. If you did not
receive the emails please let me know and I can try re-sending.

Thanks for the help.

Regards,

Ryan

----------------------------------------------------------------------

Comment By: Nick Altmann (nikbyte)
Date: 2012-08-14 09:56

Message:
> We use the 'b2bl_from_spec_param'.
I use it too.

> We are storing the b2b states in a database.
I store it in pgsql too.

> We do modify some headers in the local_route.
I also modify headers in local_route

> We do sequential routing through the b2b, as in we have a proxy that
hunts and sends its requests through the b2b2.
I'm too.


----------------------------------------------------------------------

Comment By: Ryan Bullock (rrb3942)
Date: 2012-08-14 09:42

Message:
Only differences I can think of:
We use the 'b2bl_from_spec_param'.
We are storing the b2b states in a database.
We do modify some headers in the local_route.
We do sequential routing through the b2b, as in we have a proxy that hunts
and sends its requests through the b2b2.

I will send you the configuration shortly, it is pretty straight forward.

I will not have the memdump until tonight, when I can force the dump out.
Once I have it I will post it someplace where I can get you access and send
that via email.

Is their anything else I can get that would help with troubleshooting this?

----------------------------------------------------------------------

Comment By: Nick Altmann (nikbyte)
Date: 2012-08-14 08:18

Message:
Cannot attach file. Look it here:
http://img39.imageshack.us/img39/5275/memorys.png

Also just simple topo hiding, nothing special.

----------------------------------------------------------------------

Comment By: Nick Altmann (nikbyte)
Date: 2012-08-14 08:13

Message:
Vlad,

I also expect problems with memory. Look at screenshot.
I cannot collect statistics now, but memory leak is there.
Every day I restart opensips to free memory.

----------------------------------------------------------------------

Comment By: Vladut-Stefan Paiu (vladut-paiu)
Date: 2012-08-14 07:47

Message:
Hello Ryan,

I cannot seem to reproduce your topology hiding bug.
Can you privately send me your OpenSIPS configuration file, via email ?
Also, access to the memory dump would be useful.

Regards,
Vlad

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2012-08-14 02:03

Message:
Ryan,

I will try to see if I can reproduce this. You can simple topo hiding
scenario, with successful calls, nothing special, right ?

Regards,
Bogdan

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=3557213&group_id=232389



More information about the Devel mailing list