Bogdan,<div><br></div><div>One more detail that may help - I added another OpenSIPS proxy in parallel with this one (& load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds).</div>
<div><br></div><div>I can provide more core dumps if it will help.</div><div><br></div><div>Thanks.</div><div><br><br><div class="gmail_quote">On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee <span dir="ltr"><<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Bogdan,<div><br></div><div>Below is the info requested:<br><div><span style="font-family:'courier new', monospace"><div>
(gdb) frame 0</div><div>#0 0x00007f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165</div>
<div>165 f = CON_FILE(h);</div></span></div><div><span style="font-family:'courier new', monospace">(gdb) print h</span></div><div><div><font face="'courier new', monospace">$1 = (const db_con_t *) 0x7f0978</font></div>
<div><font face="'courier new', monospace">(gdb) print h->tail</font></div><div><font face="'courier new', monospace">$2 = 0</font></div><div><font face="'courier new', monospace">(gdb) print ((struct flat_con*)(h->tail))->file</font></div>
<div><font face="'courier new', monospace">Cannot access memory at address 0x10</font></div></div><div><br></div><div>Before each of the crashes yesterday, I saw these in the logs:</div><div>
<div><font face="'courier new', monospace">Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000]</font></div>
<div><font face="'courier new', monospace">Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000]</font></div>
</div><div><font face="'courier new', monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">Are there any compiler flags I should use for debugging?</font></div>
<div><font face="arial, helvetica, sans-serif"><div><font face="'courier new', monospace">(gdb) info locals</font></div><div><font face="'courier new', monospace">f = <value optimized out></font></div>
<div><font face="'courier new', monospace">i = <value optimized out></font></div><div><font face="'courier new', monospace">l = <value optimized out></font></div>
<div><font face="'courier new', monospace">p = <value optimized out></font></div><div><font face="'courier new', monospace">__FUNCTION__ = "flat_db_insert"</font></div>
</font></div><div><br></div><div>Thanks again.</div><div><div></div><div class="h5"><div><br></div><div><br><div class="gmail_quote">On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
in frame 0, could you print:<br>
h<br>
h->tail<br>
((struct flat_con*)(h->tail))->file<br>
<br>
Also, before crash, do you see in the logs any errors from the<br>
db_flatstore module ?<br>
<br>
Regards,<br>
Bogdan<br>
<div><br>
thrillerbee wrote:<br>
> Bogdan,<br>
><br>
> It crashed again tonight. I've attached the backtrace.<br>
><br>
> Thanks.<br>
><br>
> On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee <<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a><br>
</div><div>> <mailto:<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a>>> wrote:<br>
><br>
</div><div>> Bogdan,<br>
><br>
> Yes, I've attached 2 to my response - one was a crash from 10/29.<br>
> The other occurred a few minutes ago.<br>
><br>
> Thanks.<br>
><br>
> On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu<br>
</div><div>> <<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>>> wrote:<br>
><br>
> Any chance with the backtraces ?<br>
><br>
> Regards,<br>
> Bogdan<br>
><br>
> Anca Vamanu wrote:<br>
> > Hi,<br>
> ><br>
> > You need to inspect them with gdb, run: gdb<br>
> > path_to_opensips_executable path_to_corefile, and then run<br>
> 'bt full'<br>
> > and send the output.<br>
> ><br>
> > Regards,<br>
> > --<br>
> > Anca Vamanu<br>
</div>> > <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
<div>> ><br>
> ><br>
> > On 10/14/2010 10:12 PM, thrillerbee wrote:<br>
> >> I have this info from dmesg:<br>
> >><br>
> >> [1985853.285221] opensips[30865]: segfault at 10 ip<br>
> 7f43899ce21f sp<br>
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]<br>
> >> [1985856.379671] opensips[30858]: segfault at 10 ip<br>
> 7f43899ce21f sp<br>
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]<br>
> >> [1985896.961279] opensips[30868]: segfault at 10 ip<br>
> 7f43899ce21f sp<br>
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]<br>
> >><br>
> >> [2000131.245512] opensips[17672]: segfault at 10 ip<br>
> 7fd0f21fb21f sp<br>
> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]<br>
> >> [2000161.735962] opensips[17668]: segfault at 10 ip<br>
> 7fd0f21fb21f sp<br>
> >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]<br>
> >> [2000167.299402] opensips[17670]: segfault at 10 ip<br>
> 7fd0f21fb21f sp<br>
> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]<br>
> >><br>
> >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee<br>
> <<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a> <mailto:<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a>><br>
</div>> >> <mailto:<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a><br>
<div>> <mailto:<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a>>>> wrote:<br>
> >><br>
> >> When OpenSIPS crashes, three corefiles are generated<br>
> that are<br>
> >> 2.1GB in size. How do I use these files to understand<br>
> what's<br>
> >> causing the crash?<br>
> >><br>
> >> Thanks.<br>
> >><br>
> >><br>
> ><br>
> ------------------------------------------------------------------------<br>
> ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
</div>> > <a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a> <mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>><br>
<div>> > <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
> ><br>
><br>
><br>
> --<br>
> Bogdan-Andrei Iancu<br>
> OpenSIPS Bootcamp<br>
> 15 - 19 November 2010, Edison, New Jersey, USA<br>
</div>> <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a> <mailto:<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>><br>
<div><div></div><div>> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
><br>
><br>
><br>
> ------------------------------------------------------------------------<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
<br>
--<br>
Bogdan-Andrei Iancu<br>
OpenSIPS Bootcamp<br>
15 - 19 November 2010, Edison, New Jersey, USA<br>
<a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>