Bogdan,<div><br></div><div>I am not seeing any other errors in the logs. Is there anything else I can look at? Both proxies are crashing everyday.</div><div><br></div><div>Thanks.</div><div><br><div class="gmail_quote">On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">I got some clue about what is happening - as you are using flatstore for<br>
acc, the acc module does not check the success of the "use_table" DB<br>
operation - in 90% of the cases (for mysql, postgres, etc) this function<br>
cannot fail, but it seams that for flatstore can. And if it fails, the<br>
h->tail is set to NULL, leading to crash.<br>
<br>
Now, before the crash itself, do you see any other ERROR messages (even<br>
long before the crash) related to flatstore module ? Try:<br>
grep "ERROR" opensips_log_file | grep "flatstore"<br>
<div class="im"><br>
Regards,<br>
Bogdan<br>
<br>
thrillerbee wrote:<br>
> Bogdan,<br>
><br>
</div><div class="im">> One more detail that may help - I added another OpenSIPS proxy in<br>
> parallel with this one (& load balancing between the two) and now both<br>
> OpenSIPS proxies crash at the same time (within a couple seconds).<br>
><br>
> I can provide more core dumps if it will help.<br>
><br>
> Thanks.<br>
><br>
><br>
> On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee <<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
</div><div class="im">> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>> wrote:<br>
><br>
> Bogdan,<br>
><br>
</div><div><div></div><div class="h5">> Below is the info requested:<br>
> (gdb) frame 0<br>
> #0 0x00007f51999f221f in flat_db_insert (h=0x7f0978,<br>
> k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165<br>
> 165 f = CON_FILE(h);<br>
> (gdb) print h<br>
> $1 = (const db_con_t *) 0x7f0978<br>
> (gdb) print h->tail<br>
> $2 = 0<br>
> (gdb) print ((struct flat_con*)(h->tail))->file<br>
> Cannot access memory at address 0x10<br>
><br>
> Before each of the crashes yesterday, I saw these in the logs:<br>
> Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:<br>
> segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in<br>
> db_flatstore.so[7f51999ef000+5000]<br>
> Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:<br>
> segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in<br>
> db_flatstore.so[7f726cb98000+5000]<br>
><br>
> Are there any compiler flags I should use for debugging?<br>
> (gdb) info locals<br>
> f = <value optimized out><br>
> i = <value optimized out><br>
> l = <value optimized out><br>
> p = <value optimized out><br>
> __FUNCTION__ = "flat_db_insert"<br>
><br>
> Thanks again.<br>
><br>
><br>
> On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu<br>
</div></div><div class="im">> <<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>>> wrote:<br>
><br>
</div><div class="im">> Hi,<br>
><br>
> in frame 0, could you print:<br>
> h<br>
> h->tail<br>
> ((struct flat_con*)(h->tail))->file<br>
><br>
> Also, before crash, do you see in the logs any errors from the<br>
> db_flatstore module ?<br>
><br>
> Regards,<br>
> Bogdan<br>
><br>
> thrillerbee wrote:<br>
> > Bogdan,<br>
> ><br>
> > It crashed again tonight. I've attached the backtrace.<br>
> ><br>
> > Thanks.<br>
> ><br>
> > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee<br>
> <<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
</div><div class="im">> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>>> wrote:<br>
> ><br>
</div><div><div></div><div class="h5">> > Bogdan,<br>
> ><br>
> > Yes, I've attached 2 to my response - one was a crash<br>
> from 10/29.<br>
> > The other occurred a few minutes ago.<br>
> ><br>
> > Thanks.<br>
> ><br>
> > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu<br>
> > <<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>><br>
> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a><br>
> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>>>> wrote:<br>
> ><br>
> > Any chance with the backtraces ?<br>
> ><br>
> > Regards,<br>
> > Bogdan<br>
> ><br>
> > Anca Vamanu wrote:<br>
> > > Hi,<br>
> > ><br>
> > > You need to inspect them with gdb, run: gdb<br>
> > > path_to_opensips_executable path_to_corefile, and<br>
> then run<br>
> > 'bt full'<br>
> > > and send the output.<br>
> > ><br>
> > > Regards,<br>
> > > --<br>
> > > Anca Vamanu<br>
> > > <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
> > ><br>
> > ><br>
> > > On 10/14/2010 10:12 PM, thrillerbee wrote:<br>
> > >> I have this info from dmesg:<br>
> > >><br>
> > >> [1985853.285221] opensips[30865]: segfault at 10 ip<br>
> > 7f43899ce21f sp<br>
> > >> 7fff8de1cf40 error 4 in<br>
> db_flatstore.so[7f43899cb000+5000]<br>
> > >> [1985856.379671] opensips[30858]: segfault at 10 ip<br>
> > 7f43899ce21f sp<br>
> > >> 7fff8de1cf40 error 4 in<br>
> db_flatstore.so[7f43899cb000+5000]<br>
> > >> [1985896.961279] opensips[30868]: segfault at 10 ip<br>
> > 7f43899ce21f sp<br>
> > >> 7fff8de1cf40 error 4 in<br>
> db_flatstore.so[7f43899cb000+5000]<br>
> > >><br>
> > >> [2000131.245512] opensips[17672]: segfault at 10 ip<br>
> > 7fd0f21fb21f sp<br>
> > >> 7fff3a3b4f00 error 4 in<br>
> db_flatstore.so[7fd0f21f8000+5000]<br>
> > >> [2000161.735962] opensips[17668]: segfault at 10 ip<br>
> > 7fd0f21fb21f sp<br>
> > >> 7fff3a3b4e40 error 4 in<br>
> db_flatstore.so[7fd0f21f8000+5000]<br>
> > >> [2000167.299402] opensips[17670]: segfault at 10 ip<br>
> > 7fd0f21fb21f sp<br>
> > >> 7fff3a3b4f00 error 4 in<br>
> db_flatstore.so[7fd0f21f8000+5000]<br>
> > >><br>
> > >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee<br>
> > <<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>><br>
> > >> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>>>> wrote:<br>
> > >><br>
> > >> When OpenSIPS crashes, three corefiles are<br>
> generated<br>
> > that are<br>
> > >> 2.1GB in size. How do I use these files to<br>
> understand<br>
> > what's<br>
> > >> causing the crash?<br>
> > >><br>
> > >> Thanks.<br>
> > >><br>
> > >><br>
> > ><br>
> ><br>
> ------------------------------------------------------------------------<br>
> > ><br>
> > > _______________________________________________<br>
> > > Users mailing list<br>
> > > <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>><br>
> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>>><br>
> > ><br>
> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Bogdan-Andrei Iancu<br>
> > OpenSIPS Bootcamp<br>
> > 15 - 19 November 2010, Edison, New Jersey, USA<br>
> > <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
> > <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>><br>
> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>>><br>
> > <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ------------------------------------------------------------------------<br>
> ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
> > <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>><br>
> > <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
><br>
><br>
> --<br>
> Bogdan-Andrei Iancu<br>
> OpenSIPS Bootcamp<br>
> 15 - 19 November 2010, Edison, New Jersey, USA<br>
> <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>><br>
> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
><br>
><br>
><br>
> ------------------------------------------------------------------------<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
><br>
<br>
<br>
--<br>
Bogdan-Andrei Iancu<br>
OpenSIPS Bootcamp<br>
15 - 19 November 2010, Edison, New Jersey, USA<br>
<a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>