[OpenSIPS-Devel] Debugging crash

Alex Massover alex at jajah.com
Thu Jan 14 20:23:59 CET 2010


Hi!

I use DNS. As far as I understand the reason could be that libradiusclient use gethostbyaddr() which is not reentrant and I call to libradiusclient in threads.

So I recompiled libradiusclient with -D_REENTRANT.

Afterwards it crashed also in libradiusclient in free() - no idea meanwhile how it happened. Just to be sure I added one more if() before sending it to free().

The strange thing is that crash happens only after several hours of stress test, after processing thousands of calls.
AFAIU it looks like memory corruption.

Now I recompiled everything with -O0 and run with _MALLOCK_CHECK=1, meanwhile it's ok.

Can you think about any special complication of opening threads to make accounting non-blocking?


--
Sincerely yours,
Alex Massover
VoIP R&D TL
Jajah Inc.

> -----Original Message-----
> From: devel-bounces at lists.opensips.org [mailto:devel-
> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
> Sent: Thursday, January 14, 2010 8:06 PM
> To: OpenSIPS devel mailling list
> Subject: Re: [OpenSIPS-Devel] Debugging crash
> 
> Hi Alex,
> 
> The crash looks to be in the libradiusclient stuff....and not in the
> opensips part.
> 
> Looking at the backtrace, it seams the lib tries to resolve the name of
> the acc server - do you use DNS or IP for it?
> 
> Regards,
> Bogdan
> 
> Alex Massover wrote:
> >
> > Hmmm... -D_REENTRANT supposed to help here?
> >
> > --
> >
> > Best Regards,
> >
> > Alex Massover
> >
> > VoIP R&D TL
> >
> > Jajah Inc.
> >
> > *From:* devel-bounces at lists.opensips.org
> > [mailto:devel-bounces at lists.opensips.org] *On Behalf Of *Alex
> Massover
> > *Sent:* Monday, January 11, 2010 12:21 PM
> > *To:* OpenSIPS devel mailling list
> > *Subject:* Re: [OpenSIPS-Devel] Debugging crash
> >
> > Here's a backtrace:
> >
> > #0 0xb7fcf424 in __kernel_vsyscall ()
> >
> > (gdb) bt
> >
> > #0 0xb7fcf424 in __kernel_vsyscall ()
> >
> > #1 0xb7e7c640 in raise () from /lib/i686/cmov/libc.so.6
> >
> > #2 0xb7e7e018 in abort () from /lib/i686/cmov/libc.so.6
> >
> > #3 0xb7eb93dd in ?? () from /lib/i686/cmov/libc.so.6
> >
> > #4 0xb7ebf6b4 in ?? () from /lib/i686/cmov/libc.so.6
> >
> > #5 0xb7ec18b6 in free () from /lib/i686/cmov/libc.so.6
> >
> > #6 0xb7ebb69d in _IO_file_seekoff () from /lib/i686/cmov/libc.so.6
> >
> > #7 0xb7eb1900 in ?? () from /lib/i686/cmov/libc.so.6
> >
> > #8 0xb7eb7ea8 in rewind () from /lib/i686/cmov/libc.so.6
> >
> > #9 0xb74c7f2e in ?? () from /lib/i686/cmov/libnss_files.so.2
> >
> > #10 0xb74c86b8 in _nss_files_gethostbyaddr_r () from
> > /lib/i686/cmov/libnss_files.so.2
> >
> > #11 0xb7f4a5ec in gethostbyaddr_r () from /lib/i686/cmov/libc.so.6
> >
> > #12 0xb7f4a3dc in gethostbyaddr () from /lib/i686/cmov/libc.so.6
> >
> > #13 0xb7ae75f5 in rc_ip_hostname () from /usr/lib/libradiusclient-
> ng.so.2
> >
> > #14 0xb7ae3b5f in rc_send_server () from /usr/lib/libradiusclient-
> ng.so.2
> >
> > #15 0xb7ae2b3d in rc_acct () from /usr/lib/libradiusclient-ng.so.2
> >
> > #16 0xb7da60cf in rad_send_message () from
> > /usr/lib/opensips/modules/aaa_radius.so
> >
> > #17 0xb7ac6fed in thread_rc_acct () from
> /usr/lib/opensips/modules/acc.so
> >
> > #18 0xb7ba54c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
> >
> > #19 0xb7f3161e in clone () from /lib/i686/cmov/libc.so.6
> >
> > --
> >
> > Best Regards,
> >
> > Alex Massover
> >
> > VoIP R&D TL
> >
> > Jajah Inc.
> >
> > *From:* devel-bounces at lists.opensips.org
> > [mailto:devel-bounces at lists.opensips.org] *On Behalf Of *Alex
> Massover
> > *Sent:* Monday, January 11, 2010 12:04 PM
> > *To:* OpenSIPS devel mailling list
> > *Subject:* [OpenSIPS-Devel] Debugging crash
> >
> > Hi!
> >
> > We're running a stress test and after some time (tens of minutes and
> > hundreds of processes calls) OpenSIPS 1.6.0 crashes.
> >
> > Below is an output of gdb. OpenSIPS is slightly tuned for
> performance:
> >
> > a) RADIUS accounting is sent in detached thread
> >
> > b) Libradius-ng uses shared memory & atomic memory access instead of
> > seqfile
> >
> > Any tip where/how to dig please?
> >
> > (gdb) core core.opensips.sig6.11596
> >
> > [New Thread 30240]
> >
> > [New Thread 30238]
> >
> > [New Thread 30231]
> >
> > [New Thread 30229]
> >
> > [New Thread 30203]
> >
> > [New Thread 30191]
> >
> > [New Thread 30183]
> >
> > [New Thread 30165]
> >
> > [New Thread 30159]
> >
> > [New Thread 30155]
> >
> > [New Thread 30142]
> >
> > [New Thread 30130]
> >
> > [New Thread 30126]
> >
> > [New Thread 30103]
> >
> > [New Thread 30070]
> >
> > [New Thread 30037]
> >
> > [New Thread 30010]
> >
> > [New Thread 29966]
> >
> > [New Thread 29912]
> >
> > [New Thread 29862]
> >
> > [New Thread 29818]
> >
> > [New Thread 29742]
> >
> > [New Thread 29715]
> >
> > [New Thread 29682]
> >
> > [New Thread 29628]
> >
> > [New Thread 29597]
> >
> > [New Thread 29586]
> >
> > [New Thread 29561]
> >
> > [New Thread 29491]
> >
> > [New Thread 29396]
> >
> > [New Thread 29366]
> >
> > [New Thread 29359]
> >
> > [New Thread 29316]
> >
> > [New Thread 29305]
> >
> > [New Thread 29278]
> >
> > [New Thread 29264]
> >
> > [New Thread 29260]
> >
> > [New Thread 29238]
> >
> > [New Thread 29235]
> >
> > [New Thread 29210]
> >
> > [New Thread 29184]
> >
> > [New Thread 29136]
> >
> > [New Thread 29132]
> >
> > [New Thread 11596]
> >
> > Core was generated by `/usr/sbin/opensips -P
> > /var/run/opensips/opensips.pid -m 32 -u opensips -g opens'.
> >
> > Program terminated with signal 6, Aborted.
> >
> > #0 0xb7fcf424 in __kernel_vsyscall ()
> >
> > --
> >
> > Best Regards,
> >
> > Alex Massover
> >
> > VoIP R&D TL
> >
> > Jajah Inc.
> >
> >
> >
> > This mail was sent via Mail-SeCure System.
> >
> >
> > This mail was received via Mail-SeCure System.
> >
> >
> >
> > This mail was sent via Mail-SeCure System.
> >
> >
> > This mail was received via Mail-SeCure System.
> >
> >
> >
> > This mail was sent via Mail-SeCure System.
> > ---------------------------------------------------------------------
> ---
> >
> > _______________________________________________
> > Devel mailing list
> > Devel at lists.opensips.org
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
> >
> 
> 
> --
> Bogdan-Andrei Iancu
> www.voice-system.ro
> 
> 
> _______________________________________________
> Devel mailing list
> Devel at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
> 
> This mail was received via Mail-SeCure System.
> 


This mail was sent via Mail-SeCure System.





More information about the Devel mailing list