[OpenSIPS-Users] OpenSIPS core dumps
Bogdan-Andrei Iancu
bogdan at voice-system.ro
Wed Nov 3 10:21:36 CET 2010
I got some clue about what is happening - as you are using flatstore for
acc, the acc module does not check the success of the "use_table" DB
operation - in 90% of the cases (for mysql, postgres, etc) this function
cannot fail, but it seams that for flatstore can. And if it fails, the
h->tail is set to NULL, leading to crash.
Now, before the crash itself, do you see any other ERROR messages (even
long before the crash) related to flatstore module ? Try:
grep "ERROR" opensips_log_file | grep "flatstore"
Regards,
Bogdan
thrillerbee wrote:
> Bogdan,
>
> One more detail that may help - I added another OpenSIPS proxy in
> parallel with this one (& load balancing between the two) and now both
> OpenSIPS proxies crash at the same time (within a couple seconds).
>
> I can provide more core dumps if it will help.
>
> Thanks.
>
>
> On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee <thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>> wrote:
>
> Bogdan,
>
> Below is the info requested:
> (gdb) frame 0
> #0 0x00007f51999f221f in flat_db_insert (h=0x7f0978,
> k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
> 165 f = CON_FILE(h);
> (gdb) print h
> $1 = (const db_con_t *) 0x7f0978
> (gdb) print h->tail
> $2 = 0
> (gdb) print ((struct flat_con*)(h->tail))->file
> Cannot access memory at address 0x10
>
> Before each of the crashes yesterday, I saw these in the logs:
> Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
> segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> db_flatstore.so[7f51999ef000+5000]
> Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
> segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
> db_flatstore.so[7f726cb98000+5000]
>
> Are there any compiler flags I should use for debugging?
> (gdb) info locals
> f = <value optimized out>
> i = <value optimized out>
> l = <value optimized out>
> p = <value optimized out>
> __FUNCTION__ = "flat_db_insert"
>
> Thanks again.
>
>
> On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> <bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>> wrote:
>
> Hi,
>
> in frame 0, could you print:
> h
> h->tail
> ((struct flat_con*)(h->tail))->file
>
> Also, before crash, do you see in the logs any errors from the
> db_flatstore module ?
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > It crashed again tonight. I've attached the backtrace.
> >
> > Thanks.
> >
> > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> <thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>> wrote:
> >
> > Bogdan,
> >
> > Yes, I've attached 2 to my response - one was a crash
> from 10/29.
> > The other occurred a few minutes ago.
> >
> > Thanks.
> >
> > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
> > <bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>
> <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>>> wrote:
> >
> > Any chance with the backtraces ?
> >
> > Regards,
> > Bogdan
> >
> > Anca Vamanu wrote:
> > > Hi,
> > >
> > > You need to inspect them with gdb, run: gdb
> > > path_to_opensips_executable path_to_corefile, and
> then run
> > 'bt full'
> > > and send the output.
> > >
> > > Regards,
> > > --
> > > Anca Vamanu
> > > www.voice-system.ro <http://www.voice-system.ro>
> <http://www.voice-system.ro>
> > >
> > >
> > > On 10/14/2010 10:12 PM, thrillerbee wrote:
> > >> I have this info from dmesg:
> > >>
> > >> [1985853.285221] opensips[30865]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in
> db_flatstore.so[7f43899cb000+5000]
> > >> [1985856.379671] opensips[30858]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in
> db_flatstore.so[7f43899cb000+5000]
> > >> [1985896.961279] opensips[30868]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in
> db_flatstore.so[7f43899cb000+5000]
> > >>
> > >> [2000131.245512] opensips[17672]: segfault at 10 ip
> > 7fd0f21fb21f sp
> > >> 7fff3a3b4f00 error 4 in
> db_flatstore.so[7fd0f21f8000+5000]
> > >> [2000161.735962] opensips[17668]: segfault at 10 ip
> > 7fd0f21fb21f sp
> > >> 7fff3a3b4e40 error 4 in
> db_flatstore.so[7fd0f21f8000+5000]
> > >> [2000167.299402] opensips[17670]: segfault at 10 ip
> > 7fd0f21fb21f sp
> > >> 7fff3a3b4f00 error 4 in
> db_flatstore.so[7fd0f21f8000+5000]
> > >>
> > >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
> > <thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>
> > >> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>>> wrote:
> > >>
> > >> When OpenSIPS crashes, three corefiles are
> generated
> > that are
> > >> 2.1GB in size. How do I use these files to
> understand
> > what's
> > >> causing the crash?
> > >>
> > >> Thanks.
> > >>
> > >>
> > >
> >
> ------------------------------------------------------------------------
> > >
> > > _______________________________________________
> > > Users mailing list
> > > Users at lists.opensips.org
> <mailto:Users at lists.opensips.org>
> <mailto:Users at lists.opensips.org
> <mailto:Users at lists.opensips.org>>
> > >
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > >
> >
> >
> > --
> > Bogdan-Andrei Iancu
> > OpenSIPS Bootcamp
> > 15 - 19 November 2010, Edison, New Jersey, USA
> > www.voice-system.ro <http://www.voice-system.ro>
> <http://www.voice-system.ro>
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users at lists.opensips.org
> <mailto:Users at lists.opensips.org>
> <mailto:Users at lists.opensips.org
> <mailto:Users at lists.opensips.org>>
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >
> >
> >
> >
> ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Users mailing list
> > Users at lists.opensips.org <mailto:Users at lists.opensips.org>
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> --
> Bogdan-Andrei Iancu
> OpenSIPS Bootcamp
> 15 - 19 November 2010, Edison, New Jersey, USA
> www.voice-system.ro <http://www.voice-system.ro>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro
More information about the Users
mailing list