[OpenSIPS-Users] OpenSIPS core dumps
Bogdan-Andrei Iancu
bogdan at voice-system.ro
Wed Nov 10 11:04:59 CET 2010
Hi,
opensipsctl takes care that each command takes a separate fifo reply, so
here it should be no problem. But the problem may be when comes with
sending multiple commands (via FIFO) in the same time - this translates
into parallel writes to the same file and depends on the atomicity of
the write op.
But in the worst case, a mixture at the FIFO level may lead to bogus
command and not in any kind of crash....Do you use the "flat_rotate" MI
command ?
Regards,
Bogdan
thrillerbee wrote:
> Bogdan,
>
> It seems the issue is with 'opensipsctl fifo' - it's very sensitive to
> simultaneous calls. Basically, I've combined all my scripts to
> prevent 'opensipsctl fifo' from being called too frequently and that
> seems (so far) to have mitigated the issue. Is there anything one
> should know about how (not) to use /opensipsctl/?
>
> Thanks.
>
> On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
> <bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>> wrote:
>
> Hi,
>
> strange if you do not have any errors :(....
>
> I just made a fix on both trunk and 1.6 to extend some checks in
> flatstore and prevent crashing (even if the DB op will not be
> executed).
>
> Could you update from SVN and see if stops crashing ?
>
> Regards,
> Bogdan
>
>
> thrillerbee wrote:
> > Bogdan,
> >
> > I am not seeing any other errors in the logs. Is there anything
> else
> > I can look at? Both proxies are crashing everyday.
> >
> > Thanks.
> >
> > On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
> > <bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>
> <mailto:bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>>>
> wrote:
> >
> > I got some clue about what is happening - as you are using
> > flatstore for
> > acc, the acc module does not check the success of the
> "use_table" DB
> > operation - in 90% of the cases (for mysql, postgres, etc) this
> > function
> > cannot fail, but it seams that for flatstore can. And if it
> fails, the
> > h->tail is set to NULL, leading to crash.
> >
> > Now, before the crash itself, do you see any other ERROR
> messages
> > (even
> > long before the crash) related to flatstore module ? Try:
> > grep "ERROR" opensips_log_file | grep "flatstore"
> >
> > Regards,
> > Bogdan
> >
> > thrillerbee wrote:
> > > Bogdan,
> > >
> > > One more detail that may help - I added another OpenSIPS
> proxy in
> > > parallel with this one (& load balancing between the two) and
> > now both
> > > OpenSIPS proxies crash at the same time (within a couple
> seconds).
> > >
> > > I can provide more core dumps if it will help.
> > >
> > > Thanks.
> > >
> > >
> > > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
> > <thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>
> <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>>>
> > wrote:
> > >
> > > Bogdan,
> > >
> > > Below is the info requested:
> > > (gdb) frame 0
> > > #0 0x00007f51999f221f in flat_db_insert (h=0x7f0978,
> > > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at
> flatstore.c:165
> > > 165 f = CON_FILE(h);
> > > (gdb) print h
> > > $1 = (const db_con_t *) 0x7f0978
> > > (gdb) print h->tail
> > > $2 = 0
> > > (gdb) print ((struct flat_con*)(h->tail))->file
> > > Cannot access memory at address 0x10
> > >
> > > Before each of the crashes yesterday, I saw these in
> the logs:
> > > Nov 1 14:17:40 core1 kernel: [4287745.452111]
> opensips[22141]:
> > > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> > > db_flatstore.so[7f51999ef000+5000]
> > > Nov 1 23:52:58 core1 kernel: [4348562.990735]
> opensips[26978]:
> > > segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in
> > > db_flatstore.so[7f726cb98000+5000]
> > >
> > > Are there any compiler flags I should use for debugging?
> > > (gdb) info locals
> > > f = <value optimized out>
> > > i = <value optimized out>
> > > l = <value optimized out>
> > > p = <value optimized out>
> > > __FUNCTION__ = "flat_db_insert"
> > >
> > > Thanks again.
> > >
> > >
> > > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> > > <bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro> <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>>
> > <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro> <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>>>>
> > wrote:
> > >
> > > Hi,
> > >
> > > in frame 0, could you print:
> > > h
> > > h->tail
> > > ((struct flat_con*)(h->tail))->file
> > >
> > > Also, before crash, do you see in the logs any errors
> > from the
> > > db_flatstore module ?
> > >
> > > Regards,
> > > Bogdan
> > >
> > > thrillerbee wrote:
> > > > Bogdan,
> > > >
> > > > It crashed again tonight. I've attached the
> backtrace.
> > > >
> > > > Thanks.
> > > >
> > > > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> > > <thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>
> <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>>
> > > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>>>> wrote:
> > > >
> > > > Bogdan,
> > > >
> > > > Yes, I've attached 2 to my response - one
> was a crash
> > > from 10/29.
> > > > The other occurred a few minutes ago.
> > > >
> > > > Thanks.
> > > >
> > > > On Fri, Oct 22, 2010 at 11:41 AM,
> Bogdan-Andrei Iancu
> > > > <bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>
> > <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>> <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>
> > <mailto:bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>>>
> > > <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>
> > <mailto:bogdan at voice-system.ro <mailto:bogdan at voice-system.ro>>
> > > <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>
> > <mailto:bogdan at voice-system.ro
> <mailto:bogdan at voice-system.ro>>>>> wrote:
> > > >
> > > > Any chance with the backtraces ?
> > > >
> > > > Regards,
> > > > Bogdan
> > > >
> > > > Anca Vamanu wrote:
> > > > > Hi,
> > > > >
> > > > > You need to inspect them with gdb,
> run: gdb
> > > > > path_to_opensips_executable
> > path_to_corefile, and
> > > then run
> > > > 'bt full'
> > > > > and send the output.
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Anca Vamanu
> > > > > www.voice-system.ro
> <http://www.voice-system.ro>
> > <http://www.voice-system.ro> <http://www.voice-system.ro>
> > > <http://www.voice-system.ro>
> > > > >
> > > > >
> > > > > On 10/14/2010 10:12 PM, thrillerbee wrote:
> > > > >> I have this info from dmesg:
> > > > >>
> > > > >> [1985853.285221] opensips[30865]:
> segfault
> > at 10 ip
> > > > 7f43899ce21f sp
> > > > >> 7fff8de1cf40 error 4 in
> > > db_flatstore.so[7f43899cb000+5000]
> > > > >> [1985856.379671] opensips[30858]:
> segfault
> > at 10 ip
> > > > 7f43899ce21f sp
> > > > >> 7fff8de1cf40 error 4 in
> > > db_flatstore.so[7f43899cb000+5000]
> > > > >> [1985896.961279] opensips[30868]:
> segfault
> > at 10 ip
> > > > 7f43899ce21f sp
> > > > >> 7fff8de1cf40 error 4 in
> > > db_flatstore.so[7f43899cb000+5000]
> > > > >>
> > > > >> [2000131.245512] opensips[17672]:
> segfault
> > at 10 ip
> > > > 7fd0f21fb21f sp
> > > > >> 7fff3a3b4f00 error 4 in
> > > db_flatstore.so[7fd0f21f8000+5000]
> > > > >> [2000161.735962] opensips[17668]:
> segfault
> > at 10 ip
> > > > 7fd0f21fb21f sp
> > > > >> 7fff3a3b4e40 error 4 in
> > > db_flatstore.so[7fd0f21f8000+5000]
> > > > >> [2000167.299402] opensips[17670]:
> segfault
> > at 10 ip
> > > > 7fd0f21fb21f sp
> > > > >> 7fff3a3b4f00 error 4 in
> > > db_flatstore.so[7fd0f21f8000+5000]
> > > > >>
> > > > >> On Thu, Oct 14, 2010 at 1:57 PM,
> thrillerbee
> > > > <thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>>>
> > > > >> <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>>
> > > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com <mailto:thrillerbee at gmail.com>>
> > > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>
> > <mailto:thrillerbee at gmail.com
> <mailto:thrillerbee at gmail.com>>>>>> wrote:
> > > > >>
> > > > >> When OpenSIPS crashes, three
> corefiles are
> > > generated
> > > > that are
> > > > >> 2.1GB in size. How do I use
> these files to
> > > understand
> > > > what's
> > > > >> causing the crash?
> > > > >>
> > > > >> Thanks.
> > > > >>
> > >
> >
>
>
> --
> Bogdan-Andrei Iancu
> OpenSIPS Bootcamp
> 15 - 19 November 2010, Edison, New Jersey, USA
> www.voice-system.ro <http://www.voice-system.ro>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org <mailto:Users at lists.opensips.org>
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro
More information about the Users
mailing list