[OpenSIPS-Devel] Debugging crash

[Alex Massover]

Yes, you right, I should have done it in the libradiusclient rather than hacking acc module.
Now it runs already for 18 hours - so I suppose it's OK for now, before the new design is available.

Thanks and looking forward to switch to the new opensips architecture! :)
--
Sincerely yours,
Alex Massover
VoIP R&D TL
Jajah Inc.

> -----Original Message-----
> From: devel-bounces at lists.opensips.org [mailto:devel-
> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
> Sent: Friday, January 15, 2010 1:48 PM
> To: OpenSIPS devel mailling list
> Subject: Re: [OpenSIPS-Devel] Debugging crash
> 
> Hi Alex,
> 
> I see - I guess you did some extension to simulate non blocking radius
> acc and this triggers the problem in the lib.
> 
> Regarding the question - to make the acc non-blocking is something to
> be
> normally done in the lib radius (like the net functions do allow you to
> use them into a non-blocking way). Trying to emulate non-blocking in
> opensips for blocking function is a kind of ugly and bullsxxt stuff.
> 
> Anyhow, we are preparing the new design for opensips - probably it will
> be published next week or so. It will solve all these problems.
> 
> Regards,
> Bogdan
> 
> Regards,
> Bogdan
> 
> Alex Massover wrote:
> > Hi!
> >
> > I use DNS. As far as I understand the reason could be that
> libradiusclient use gethostbyaddr() which is not reentrant and I call
> to libradiusclient in threads.
> >
> > So I recompiled libradiusclient with -D_REENTRANT.
> >
> > Afterwards it crashed also in libradiusclient in free() - no idea
> meanwhile how it happened. Just to be sure I added one more if() before
> sending it to free().
> >
> > The strange thing is that crash happens only after several hours of
> stress test, after processing thousands of calls.
> > AFAIU it looks like memory corruption.
> >
> > Now I recompiled everything with -O0 and run with _MALLOCK_CHECK=1,
> meanwhile it's ok.
> >
> > Can you think about any special complication of opening threads to
> make accounting non-blocking?
> >
> >
> > --
> > Sincerely yours,
> > Alex Massover
> > VoIP R&D TL
> > Jajah Inc.
> >
> >
> >> -----Original Message-----
> >> From: devel-bounces at lists.opensips.org [mailto:devel-
> >> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
> >> Sent: Thursday, January 14, 2010 8:06 PM
> >> To: OpenSIPS devel mailling list
> >> Subject: Re: [OpenSIPS-Devel] Debugging crash
> >>
> >> Hi Alex,
> >>
> >> The crash looks to be in the libradiusclient stuff....and not in the
> >> opensips part.
> >>
> >> Looking at the backtrace, it seams the lib tries to resolve the name
> of
> >> the acc server - do you use DNS or IP for it?
> >>
> >> Regards,
> >> Bogdan
> >>
> >> Alex Massover wrote:
> >>
> >>> Hmmm... -D_REENTRANT supposed to help here?
> >>>
> >>> --
> >>>
> >>> Best Regards,
> >>>
> >>> Alex Massover
> >>>
> >>> VoIP R&D TL
> >>>
> >>> Jajah Inc.
> >>>
> >>> *From:* devel-bounces at lists.opensips.org
> >>> [mailto:devel-bounces at lists.opensips.org] *On Behalf Of *Alex
> >>>
> >> Massover
> >>
> >>> *Sent:* Monday, January 11, 2010 12:21 PM
> >>> *To:* OpenSIPS devel mailling list
> >>> *Subject:* Re: [OpenSIPS-Devel] Debugging crash
> >>>
> >>> Here's a backtrace:
> >>>
> >>> #0 0xb7fcf424 in __kernel_vsyscall ()
> >>>
> >>> (gdb) bt
> >>>
> >>> #0 0xb7fcf424 in __kernel_vsyscall ()
> >>>
> >>> #1 0xb7e7c640 in raise () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #2 0xb7e7e018 in abort () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #3 0xb7eb93dd in ?? () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #4 0xb7ebf6b4 in ?? () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #5 0xb7ec18b6 in free () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #6 0xb7ebb69d in _IO_file_seekoff () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #7 0xb7eb1900 in ?? () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #8 0xb7eb7ea8 in rewind () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #9 0xb74c7f2e in ?? () from /lib/i686/cmov/libnss_files.so.2
> >>>
> >>> #10 0xb74c86b8 in _nss_files_gethostbyaddr_r () from
> >>> /lib/i686/cmov/libnss_files.so.2
> >>>
> >>> #11 0xb7f4a5ec in gethostbyaddr_r () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #12 0xb7f4a3dc in gethostbyaddr () from /lib/i686/cmov/libc.so.6
> >>>
> >>> #13 0xb7ae75f5 in rc_ip_hostname () from /usr/lib/libradiusclient-
> >>>
> >> ng.so.2
> >>
> >>> #14 0xb7ae3b5f in rc_send_server () from /usr/lib/libradiusclient-
> >>>
> >> ng.so.2
> >>
> >>> #15 0xb7ae2b3d in rc_acct () from /usr/lib/libradiusclient-ng.so.2
> >>>
> >>> #16 0xb7da60cf in rad_send_message () from
> >>> /usr/lib/opensips/modules/aaa_radius.so
> >>>
> >>> #17 0xb7ac6fed in thread_rc_acct () from
> >>>
> >> /usr/lib/opensips/modules/acc.so
> >>
> >>> #18 0xb7ba54c0 in start_thread () from
> /lib/i686/cmov/libpthread.so.0
> >>>
> >>> #19 0xb7f3161e in clone () from /lib/i686/cmov/libc.so.6
> >>>
> >>> --
> >>>
> >>> Best Regards,
> >>>
> >>> Alex Massover
> >>>
> >>> VoIP R&D TL
> >>>
> >>> Jajah Inc.
> >>>
> >>> *From:* devel-bounces at lists.opensips.org
> >>> [mailto:devel-bounces at lists.opensips.org] *On Behalf Of *Alex
> >>>
> >> Massover
> >>
> >>> *Sent:* Monday, January 11, 2010 12:04 PM
> >>> *To:* OpenSIPS devel mailling list
> >>> *Subject:* [OpenSIPS-Devel] Debugging crash
> >>>
> >>> Hi!
> >>>
> >>> We're running a stress test and after some time (tens of minutes
> and
> >>> hundreds of processes calls) OpenSIPS 1.6.0 crashes.
> >>>
> >>> Below is an output of gdb. OpenSIPS is slightly tuned for
> >>>
> >> performance:
> >>
> >>> a) RADIUS accounting is sent in detached thread
> >>>
> >>> b) Libradius-ng uses shared memory & atomic memory access instead
> of
> >>> seqfile
> >>>
> >>> Any tip where/how to dig please?
> >>>
> >>> (gdb) core core.opensips.sig6.11596
> >>>
> >>> [New Thread 30240]
> >>>
> >>> [New Thread 30238]
> >>>
> >>> [New Thread 30231]
> >>>
> >>> [New Thread 30229]
> >>>
> >>> [New Thread 30203]
> >>>
> >>> [New Thread 30191]
> >>>
> >>> [New Thread 30183]
> >>>
> >>> [New Thread 30165]
> >>>
> >>> [New Thread 30159]
> >>>
> >>> [New Thread 30155]
> >>>
> >>> [New Thread 30142]
> >>>
> >>> [New Thread 30130]
> >>>
> >>> [New Thread 30126]
> >>>
> >>> [New Thread 30103]
> >>>
> >>> [New Thread 30070]
> >>>
> >>> [New Thread 30037]
> >>>
> >>> [New Thread 30010]
> >>>
> >>> [New Thread 29966]
> >>>
> >>> [New Thread 29912]
> >>>
> >>> [New Thread 29862]
> >>>
> >>> [New Thread 29818]
> >>>
> >>> [New Thread 29742]
> >>>
> >>> [New Thread 29715]
> >>>
> >>> [New Thread 29682]
> >>>
> >>> [New Thread 29628]
> >>>
> >>> [New Thread 29597]
> >>>
> >>> [New Thread 29586]
> >>>
> >>> [New Thread 29561]
> >>>
> >>> [New Thread 29491]
> >>>
> >>> [New Thread 29396]
> >>>
> >>> [New Thread 29366]
> >>>
> >>> [New Thread 29359]
> >>>
> >>> [New Thread 29316]
> >>>
> >>> [New Thread 29305]
> >>>
> >>> [New Thread 29278]
> >>>
> >>> [New Thread 29264]
> >>>
> >>> [New Thread 29260]
> >>>
> >>> [New Thread 29238]
> >>>
> >>> [New Thread 29235]
> >>>
> >>> [New Thread 29210]
> >>>
> >>> [New Thread 29184]
> >>>
> >>> [New Thread 29136]
> >>>
> >>> [New Thread 29132]
> >>>
> >>> [New Thread 11596]
> >>>
> >>> Core was generated by `/usr/sbin/opensips -P
> >>> /var/run/opensips/opensips.pid -m 32 -u opensips -g opens'.
> >>>
> >>> Program terminated with signal 6, Aborted.
> >>>
> >>> #0 0xb7fcf424 in __kernel_vsyscall ()
> >>>
> >>> --
> >>>
> 
> 
> _______________________________________________
> Devel mailing list
> Devel at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/devel
> 
> This mail was received via Mail-SeCure System.
> 


This mail was sent via Mail-SeCure System.