[OpenSIPS-Devel] [ opensips-Bugs-2274274 ] seg fault in db_postgres

Mon Apr 13 10:08:58 CEST 2009

Bugs item #2274274, was opened at 2008-11-13 07:25
Message generated for change (Comment added) made by bogdan_iancu
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=2274274&group_id=232389

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: modules
Group: trunk
>Status: Closed
>Resolution: Out of Date
Priority: 2
Private: No
Submitted By: Brent (brent_thomson)
Assigned to: Bogdan-Andrei Iancu (bogdan_iancu)
Summary: seg fault in db_postgres

Initial Comment:
Partial output:

*** glibc detected *** opensips: free(): invalid next size (fast): 0x0000000001166c70 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2b92149b8684]
/lib64/libc.so.6(cfree+0x8c)[0x2b92149bbccc]
/usr/lib64/libpq.so.5(PQsendQuery+0x69)[0x2b921642a819]
/usr/lib64/opensips/modules/db_postgres.so[0x2b921620f0df]
opensips(db_do_query+0x335)[0x4b0da1]
/usr/lib64/opensips/modules/db_postgres.so(db_postgres_query+0x8a)[0x2b921620fd59]
/usr/lib64/opensips/modules/auth_db.so[0x2b92181ae5f1]
/usr/lib64/opensips/modules/auth_db.so[0x2b92181ae0f1]
/usr/lib64/opensips/modules/auth_db.so(proxy_authorize+0x2a)[0x2b92181adec8]
opensips(do_action+0x24d4)[0x410de0]
opensips(run_action_list+0x32)[0x40e1fa]
opensips[0x44d7aa]
opensips(eval_expr+0xc8)[0x451633]
opensips(eval_expr+0x1ac)[0x451717]
opensips(eval_expr+0x1e0)[0x45174b]
opensips(do_action+0x1caf)[0x4105bb]
opensips(run_action_list+0x32)[0x40e1fa]
opensips(do_action+0x1de0)[0x4106ec]
opensips(run_action_list+0x32)[0x40e1fa]
opensips[0x40e46e]
opensips(do_action+0x103b)[0x40f947]
opensips(run_action_list+0x32)[0x40e1fa]
opensips(do_action+0x1de0)[0x4106ec]
opensips(run_action_list+0x32)[0x40e1fa]
opensips[0x40e46e]
opensips(run_top_route+0x61)[0x40e526]
opensips(receive_msg+0x51f)[0x445da6]
opensips(udp_rcv_loop+0x45c)[0x47af5b]
opensips[0x4219ad]
opensips(main+0x1c74)[0x423ccf]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b92149648b4]
opensips[0x40e119]

CentOS 5.2
OpenSIPS: latest from svn (about an hour ago)
PostgreSQL libs: 8.3.4 (built against the same)
DB in writeback mode

Causes OpenSIPS to crash every time within a few minutes of starting. I can reproduce the crash by repeatedly reregistering a hard or soft phone, making a call from a successfully registered device, or anything else that touches the database. Reference build and config is OpenSER 1.2, which does not have the problem.

The core dump is too large to attach so I've posted it here:

http://www.sendspace.com/file/1pv989

----------------------------------------------------------------------

>Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2009-04-13 11:08

Message:
All fixes that were done on postgres for 1.5.0 were also ported  on 1.4
branch, so if you update to 1.4.5 (latest on 1.4 branch) it should be also
fixed.

Regards,
Bogdan

----------------------------------------------------------------------

Comment By: Brent (brent_thomson)
Date: 2009-04-12 07:15

Message:
I've been running 1.5 with the setup that produced the crash consistently
before for about an hour now without incident. That's not long, but 1.4 was
crashing within a few minutes.

I'll follow up if the problem surfaces again.

----------------------------------------------------------------------

Comment By: Brent (brent_thomson)
Date: 2009-03-16 20:52

Message:
I haven't tried it in a while--just been running 1.3.2. I'll give the
latest version a try when I have a second.

----------------------------------------------------------------------

Comment By: Bogdan-Andrei Iancu (bogdan_iancu)
Date: 2009-03-16 20:47

Message:
hi brent,

there were some fixes in 1.5 over the postgres module - can you update and
test ?

or, is this still happening ?

regards,
bogdan

----------------------------------------------------------------------

Comment By: Brent (brent_thomson)
Date: 2009-02-13 23:47

Message:
Doesn't really sound like a fix ... and 70 connections isn't acceptable. I
think I'll look for things that have changed since openser 1.3.2, which
doesn't have this issue.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-02-13 23:02

Message:
Brent, do you feel like getting the patch from issue 2593088 and to try
with it? It seems like the same issue.

Keep in mind that it will make a ton of connections to the database (in my
case they were 70).

----------------------------------------------------------------------

Comment By: Brent (brent_thomson)
Date: 2009-02-13 21:58

Message:
Hmmm ... That was some time ago. Let me see if I can reproduce it and put
up a log file.

----------------------------------------------------------------------

Comment By: Sergio Gutierrez (saguti)
Date: 2009-02-10 07:10

Message:
Hello Brent.

I was reviewing your report. Please, could you post OpenSIPS log before
Segmentation Fault?

Regards.

Sergio.

----------------------------------------------------------------------

Comment By: Brent (brent_thomson)
Date: 2008-11-18 20:05

Message:
Is this what you need?

(gdb) bt
#0  0x00002b9e74dc7155 in raise () from /lib64/libc.so.6
#1  0x00002b9e74dc8bf0 in abort () from /lib64/libc.so.6
#2  0x00002b9e74e013db in __libc_message () from /lib64/libc.so.6
#3  0x00002b9e74e08684 in _int_free () from /lib64/libc.so.6
#4  0x00002b9e74e0bccc in free () from /lib64/libc.so.6
#5  0x00002b9e768797e9 in PQclear () from /usr/lib64/libpq.so.5
#6  0x00002b9e7665fbe9 in free_query (_con=0x779fe8) at dbase.c:320
#7  0x00002b9e7666037d in db_postgres_store_result (_con=0x779fe8,
_r=0x7fff368d58b8)
    at dbase.c:475
#8  0x00000000004b0634 in db_do_query ()
#9  0x00002b9e7665fcc9 in db_postgres_query (_h=0x779fe8,
_k=0x7fff368d5830, _op=0x0, 
    _v=0x7fff368d57f0, _c=0x7802e0, _n=1, _nc=2, _o=0x0,
_r=0x7fff368d58b8) at dbase.c:363
#10 0x00002b9e785fd5b1 in get_ha1 (_username=0x780948,
_domain=0x7fff368d58d0, 
    _table=0x7fff368d58c0, _ha1=0x7fff368d58f0 "", res=0x7fff368d58b8) at
authorize.c:101
#11 0x00002b9e785fd0b1 in authorize (_m=0x77f328, _realm=0x779f30,
_table=0x7737a8 "subscriber", 
    _hftype=HDR_AUTHORIZATION_T) at authorize.c:244
#12 0x00002b9e785fd9fd in www_authorize (_m=0x77f328, _realm=0x779f30
"\001", 
    _table=0x7737a8 "subscriber") at authorize.c:287
#13 0x00000000004105f0 in do_action ()
#14 0x000000000040da0a in run_action_list ()
#15 0x000000000044cfba in eval_elem ()
#16 0x0000000000450e43 in eval_expr ()
#17 0x0000000000450f27 in eval_expr ()
#18 0x0000000000450f5b in eval_expr ()
#19 0x000000000040fdcb in do_action ()
#20 0x000000000040da0a in run_action_list ()
#21 0x000000000040dc7e in run_actions ()
#22 0x000000000040f157 in do_action ()
#23 0x000000000040da0a in run_action_list ()
#24 0x000000000044cfba in eval_elem ()
#25 0x0000000000450e43 in eval_expr ()
#26 0x0000000000450f5b in eval_expr ()
#27 0x000000000040fdcb in do_action ()
#28 0x000000000040da0a in run_action_list ()
#29 0x000000000040fefc in do_action ()
#30 0x000000000040da0a in run_action_list ()
#31 0x000000000040dc7e in run_actions ()
#32 0x000000000040dd36 in run_top_route ()
#33 0x00000000004455b6 in receive_msg ()
#34 0x000000000047a76b in udp_rcv_loop ()
#35 0x0000000000421501 in main_loop ()
#36 0x00000000004234df in main ()

----------------------------------------------------------------------

Comment By: Brent (brent_thomson)
Date: 2008-11-18 19:17

Message:
That backtrace is the output from opensips itself, with debugging set to 9.
I'm not a gdb jedi. Can you give me instructions or point me to what I need
to do to generate what you're asking for? Is it just a matter of running
the binary in the debugger and then grabbing the output when it crashes?

----------------------------------------------------------------------

Comment By: Sergio Gutierrez (saguti)
Date: 2008-11-18 02:19

Message:
Hello Brent.

Is that backtrace taken with gdb? The syntax does not loke familiar.
In case it does not, could you take the backtrace with gdb?

Regards.

Sergio.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2008-11-18 01:59

Message:
After fiddling with this some more, it appears that I can only reproduce
this consistently if fork=yes in opensips.cfg. If I don't daemonize
opensips (run it bare, rather than through the init script) and let all the
output go to the console, it no longer crashes.

Please let me know if there's any other useful information that I can
provide.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1086410&aid=2274274&group_id=232389