[OpenSIPS-Devel] Debugging crash

Bogdan-Andrei Iancu bogdan at voice-system.ro
Fri Jan 15 12:48:16 CET 2010


Hi Alex,

I see - I guess you did some extension to simulate non blocking radius 
acc and this triggers the problem in the lib.

Regarding the question - to make the acc non-blocking is something to be 
normally done in the lib radius (like the net functions do allow you to 
use them into a non-blocking way). Trying to emulate non-blocking in 
opensips for blocking function is a kind of ugly and bullsxxt stuff.

Anyhow, we are preparing the new design for opensips - probably it will 
be published next week or so. It will solve all these problems.

Regards,
Bogdan

Regards,
Bogdan

Alex Massover wrote:
> Hi!
>
> I use DNS. As far as I understand the reason could be that libradiusclient use gethostbyaddr() which is not reentrant and I call to libradiusclient in threads.
>
> So I recompiled libradiusclient with -D_REENTRANT.
>
> Afterwards it crashed also in libradiusclient in free() - no idea meanwhile how it happened. Just to be sure I added one more if() before sending it to free().
>
> The strange thing is that crash happens only after several hours of stress test, after processing thousands of calls.
> AFAIU it looks like memory corruption.
>
> Now I recompiled everything with -O0 and run with _MALLOCK_CHECK=1, meanwhile it's ok.
>
> Can you think about any special complication of opening threads to make accounting non-blocking?
>
>
> --
> Sincerely yours,
> Alex Massover
> VoIP R&D TL
> Jajah Inc.
>
>   
>> -----Original Message-----
>> From: devel-bounces at lists.opensips.org [mailto:devel-
>> bounces at lists.opensips.org] On Behalf Of Bogdan-Andrei Iancu
>> Sent: Thursday, January 14, 2010 8:06 PM
>> To: OpenSIPS devel mailling list
>> Subject: Re: [OpenSIPS-Devel] Debugging crash
>>
>> Hi Alex,
>>
>> The crash looks to be in the libradiusclient stuff....and not in the
>> opensips part.
>>
>> Looking at the backtrace, it seams the lib tries to resolve the name of
>> the acc server - do you use DNS or IP for it?
>>
>> Regards,
>> Bogdan
>>
>> Alex Massover wrote:
>>     
>>> Hmmm... -D_REENTRANT supposed to help here?
>>>
>>> --
>>>
>>> Best Regards,
>>>
>>> Alex Massover
>>>
>>> VoIP R&D TL
>>>
>>> Jajah Inc.
>>>
>>> *From:* devel-bounces at lists.opensips.org
>>> [mailto:devel-bounces at lists.opensips.org] *On Behalf Of *Alex
>>>       
>> Massover
>>     
>>> *Sent:* Monday, January 11, 2010 12:21 PM
>>> *To:* OpenSIPS devel mailling list
>>> *Subject:* Re: [OpenSIPS-Devel] Debugging crash
>>>
>>> Here's a backtrace:
>>>
>>> #0 0xb7fcf424 in __kernel_vsyscall ()
>>>
>>> (gdb) bt
>>>
>>> #0 0xb7fcf424 in __kernel_vsyscall ()
>>>
>>> #1 0xb7e7c640 in raise () from /lib/i686/cmov/libc.so.6
>>>
>>> #2 0xb7e7e018 in abort () from /lib/i686/cmov/libc.so.6
>>>
>>> #3 0xb7eb93dd in ?? () from /lib/i686/cmov/libc.so.6
>>>
>>> #4 0xb7ebf6b4 in ?? () from /lib/i686/cmov/libc.so.6
>>>
>>> #5 0xb7ec18b6 in free () from /lib/i686/cmov/libc.so.6
>>>
>>> #6 0xb7ebb69d in _IO_file_seekoff () from /lib/i686/cmov/libc.so.6
>>>
>>> #7 0xb7eb1900 in ?? () from /lib/i686/cmov/libc.so.6
>>>
>>> #8 0xb7eb7ea8 in rewind () from /lib/i686/cmov/libc.so.6
>>>
>>> #9 0xb74c7f2e in ?? () from /lib/i686/cmov/libnss_files.so.2
>>>
>>> #10 0xb74c86b8 in _nss_files_gethostbyaddr_r () from
>>> /lib/i686/cmov/libnss_files.so.2
>>>
>>> #11 0xb7f4a5ec in gethostbyaddr_r () from /lib/i686/cmov/libc.so.6
>>>
>>> #12 0xb7f4a3dc in gethostbyaddr () from /lib/i686/cmov/libc.so.6
>>>
>>> #13 0xb7ae75f5 in rc_ip_hostname () from /usr/lib/libradiusclient-
>>>       
>> ng.so.2
>>     
>>> #14 0xb7ae3b5f in rc_send_server () from /usr/lib/libradiusclient-
>>>       
>> ng.so.2
>>     
>>> #15 0xb7ae2b3d in rc_acct () from /usr/lib/libradiusclient-ng.so.2
>>>
>>> #16 0xb7da60cf in rad_send_message () from
>>> /usr/lib/opensips/modules/aaa_radius.so
>>>
>>> #17 0xb7ac6fed in thread_rc_acct () from
>>>       
>> /usr/lib/opensips/modules/acc.so
>>     
>>> #18 0xb7ba54c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
>>>
>>> #19 0xb7f3161e in clone () from /lib/i686/cmov/libc.so.6
>>>
>>> --
>>>
>>> Best Regards,
>>>
>>> Alex Massover
>>>
>>> VoIP R&D TL
>>>
>>> Jajah Inc.
>>>
>>> *From:* devel-bounces at lists.opensips.org
>>> [mailto:devel-bounces at lists.opensips.org] *On Behalf Of *Alex
>>>       
>> Massover
>>     
>>> *Sent:* Monday, January 11, 2010 12:04 PM
>>> *To:* OpenSIPS devel mailling list
>>> *Subject:* [OpenSIPS-Devel] Debugging crash
>>>
>>> Hi!
>>>
>>> We're running a stress test and after some time (tens of minutes and
>>> hundreds of processes calls) OpenSIPS 1.6.0 crashes.
>>>
>>> Below is an output of gdb. OpenSIPS is slightly tuned for
>>>       
>> performance:
>>     
>>> a) RADIUS accounting is sent in detached thread
>>>
>>> b) Libradius-ng uses shared memory & atomic memory access instead of
>>> seqfile
>>>
>>> Any tip where/how to dig please?
>>>
>>> (gdb) core core.opensips.sig6.11596
>>>
>>> [New Thread 30240]
>>>
>>> [New Thread 30238]
>>>
>>> [New Thread 30231]
>>>
>>> [New Thread 30229]
>>>
>>> [New Thread 30203]
>>>
>>> [New Thread 30191]
>>>
>>> [New Thread 30183]
>>>
>>> [New Thread 30165]
>>>
>>> [New Thread 30159]
>>>
>>> [New Thread 30155]
>>>
>>> [New Thread 30142]
>>>
>>> [New Thread 30130]
>>>
>>> [New Thread 30126]
>>>
>>> [New Thread 30103]
>>>
>>> [New Thread 30070]
>>>
>>> [New Thread 30037]
>>>
>>> [New Thread 30010]
>>>
>>> [New Thread 29966]
>>>
>>> [New Thread 29912]
>>>
>>> [New Thread 29862]
>>>
>>> [New Thread 29818]
>>>
>>> [New Thread 29742]
>>>
>>> [New Thread 29715]
>>>
>>> [New Thread 29682]
>>>
>>> [New Thread 29628]
>>>
>>> [New Thread 29597]
>>>
>>> [New Thread 29586]
>>>
>>> [New Thread 29561]
>>>
>>> [New Thread 29491]
>>>
>>> [New Thread 29396]
>>>
>>> [New Thread 29366]
>>>
>>> [New Thread 29359]
>>>
>>> [New Thread 29316]
>>>
>>> [New Thread 29305]
>>>
>>> [New Thread 29278]
>>>
>>> [New Thread 29264]
>>>
>>> [New Thread 29260]
>>>
>>> [New Thread 29238]
>>>
>>> [New Thread 29235]
>>>
>>> [New Thread 29210]
>>>
>>> [New Thread 29184]
>>>
>>> [New Thread 29136]
>>>
>>> [New Thread 29132]
>>>
>>> [New Thread 11596]
>>>
>>> Core was generated by `/usr/sbin/opensips -P
>>> /var/run/opensips/opensips.pid -m 32 -u opensips -g opens'.
>>>
>>> Program terminated with signal 6, Aborted.
>>>
>>> #0 0xb7fcf424 in __kernel_vsyscall ()
>>>
>>> --
>>>       




More information about the Devel mailing list