[OpenSIPS-Users] OpenSIPS core dumps
Bogdan-Andrei Iancu
bogdan at voice-system.ro
Mon Nov 8 13:07:07 CET 2010
Hi,
strange if you do not have any errors :(....
I just made a fix on both trunk and 1.6 to extend some checks in
flatstore and prevent crashing (even if the DB op will not be executed).
Could you update from SVN and see if stops crashing ?
Regards,
Bogdan
thrillerbee wrote:
> Bogdan,
>
> I am not seeing any other errors in the logs. Is there anything else
> I can look at? Both proxies are crashing everyday.
>
> Thanks.
>
> On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
> <bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>> wrote:
>
> I got some clue about what is happening - as you are using
> flatstore for
> acc, the acc module does not check the success of the "use_table" DB
> operation - in 90% of the cases (for mysql, postgres, etc) this
> function
> cannot fail, but it seams that for flatstore can. And if it fails, the
> h->tail is set to NULL, leading to crash.
>
> Now, before the crash itself, do you see any other ERROR messages
> (even
> long before the crash) related to flatstore module ? Try:
> grep "ERROR" opensips_log_file | grep "flatstore"
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > One more detail that may help - I added another OpenSIPS proxy in
> > parallel with this one (& load balancing between the two) and
> now both
> > OpenSIPS proxies crash at the same time (within a couple seconds).
> >
> > I can provide more core dumps if it will help.
> >
> > Thanks.
> >
> >
> > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
> <thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>>
> wrote:
> >
> > Bogdan,
> >
> > Below is the info requested:
> > (gdb) frame 0
> > #0 0x00007f51999f221f in flat_db_insert (h=0x7f0978,
> > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
> > 165 f = CON_FILE(h);
> > (gdb) print h
> > $1 = (const db_con_t *) 0x7f0978
> > (gdb) print h->tail
> > $2 = 0
> > (gdb) print ((struct flat_con*)(h->tail))->file
> > Cannot access memory at address 0x10
> >
> > Before each of the crashes yesterday, I saw these in the logs:
> > Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
> > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> > db_flatstore.so[7f51999ef000+5000]
> > Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
> > segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
> > db_flatstore.so[7f726cb98000+5000]
> >
> > Are there any compiler flags I should use for debugging?
> > (gdb) info locals
> > f = <value optimized out>
> > i = <value optimized out>
> > l = <value optimized out>
> > p = <value optimized out>
> > __FUNCTION__ = "flat_db_insert"
> >
> > Thanks again.
> >
> >
> > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> > <bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>
> <mailto:bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>>>
> wrote:
> >
> > Hi,
> >
> > in frame 0, could you print:
> > h
> > h->tail
> > ((struct flat_con*)(h->tail))->file
> >
> > Also, before crash, do you see in the logs any errors
> from the
> > db_flatstore module ?
> >
> > Regards,
> > Bogdan
> >
> > thrillerbee wrote:
> > > Bogdan,
> > >
> > > It crashed again tonight. I've attached the backtrace.
> > >
> > > Thanks.
> > >
> > > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> > <thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>
> <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>>> wrote:
> > >
> > > Bogdan,
> > >
> > > Yes, I've attached 2 to my response - one was a crash
> > from 10/29.
> > > The other occurred a few minutes ago.
> > >
> > > Thanks.
> > >
> > > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
> > > <bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro> <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>>
> > <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>
> > <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>>>> wrote:
> > >
> > > Any chance with the backtraces ?
> > >
> > > Regards,
> > > Bogdan
> > >
> > > Anca Vamanu wrote:
> > > > Hi,
> > > >
> > > > You need to inspect them with gdb, run: gdb
> > > > path_to_opensips_executable
> path_to_corefile, and
> > then run
> > > 'bt full'
> > > > and send the output.
> > > >
> > > > Regards,
> > > > --
> > > > Anca Vamanu
> > > > www.voice-system.ro
> <http://www.voice-system.ro> <http://www.voice-system.ro>
> > <http://www.voice-system.ro>
> > > >
> > > >
> > > > On 10/14/2010 10:12 PM, thrillerbee wrote:
> > > >> I have this info from dmesg:
> > > >>
> > > >> [1985853.285221] opensips[30865]: segfault
> at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >> [1985856.379671] opensips[30858]: segfault
> at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >> [1985896.961279] opensips[30868]: segfault
> at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >>
> > > >> [2000131.245512] opensips[17672]: segfault
> at 10 ip
> > > 7fd0f21fb21f sp
> > > >> 7fff3a3b4f00 error 4 in
> > db_flatstore.so[7fd0f21f8000+5000]
> > > >> [2000161.735962] opensips[17668]: segfault
> at 10 ip
> > > 7fd0f21fb21f sp
> > > >> 7fff3a3b4e40 error 4 in
> > db_flatstore.so[7fd0f21f8000+5000]
> > > >> [2000167.299402] opensips[17670]: segfault
> at 10 ip
> > > 7fd0f21fb21f sp
> > > >> 7fff3a3b4f00 error 4 in
> > db_flatstore.so[7fd0f21f8000+5000]
> > > >>
> > > >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
> > > <thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>>
> > > >> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>>>> wrote:
> > > >>
> > > >> When OpenSIPS crashes, three corefiles are
> > generated
> > > that are
> > > >> 2.1GB in size. How do I use these files to
> > understand
> > > what's
> > > >> causing the crash?
> > > >>
> > > >> Thanks.
> > > >>
> >
>
--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro
More information about the Users
mailing list