Bogdan,<div><br></div><div>It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls. Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue. Is there anything one should know about how (not) to use <i>opensipsctl</i>?</div>
<div><br></div><div>Thanks.<br><br><div class="gmail_quote">On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi,<br>
<br>
strange if you do not have any errors :(....<br>
<br>
I just made a fix on both trunk and 1.6 to extend some checks in<br>
flatstore and prevent crashing (even if the DB op will not be executed).<br>
<br>
Could you update from SVN and see if stops crashing ?<br>
<div class="im"><br>
Regards,<br>
Bogdan<br>
<br>
<br>
thrillerbee wrote:<br>
> Bogdan,<br>
><br>
</div><div class="im">> I am not seeing any other errors in the logs. Is there anything else<br>
> I can look at? Both proxies are crashing everyday.<br>
><br>
> Thanks.<br>
><br>
> On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu<br>
</div><div class="im">> <<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>>> wrote:<br>
><br>
</div><div><div></div><div class="h5">> I got some clue about what is happening - as you are using<br>
> flatstore for<br>
> acc, the acc module does not check the success of the "use_table" DB<br>
> operation - in 90% of the cases (for mysql, postgres, etc) this<br>
> function<br>
> cannot fail, but it seams that for flatstore can. And if it fails, the<br>
> h->tail is set to NULL, leading to crash.<br>
><br>
> Now, before the crash itself, do you see any other ERROR messages<br>
> (even<br>
> long before the crash) related to flatstore module ? Try:<br>
> grep "ERROR" opensips_log_file | grep "flatstore"<br>
><br>
> Regards,<br>
> Bogdan<br>
><br>
> thrillerbee wrote:<br>
> > Bogdan,<br>
> ><br>
> > One more detail that may help - I added another OpenSIPS proxy in<br>
> > parallel with this one (& load balancing between the two) and<br>
> now both<br>
> > OpenSIPS proxies crash at the same time (within a couple seconds).<br>
> ><br>
> > I can provide more core dumps if it will help.<br>
> ><br>
> > Thanks.<br>
> ><br>
> ><br>
> > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee<br>
> <<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
</div></div><div class="im">> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>>><br>
> wrote:<br>
> ><br>
> > Bogdan,<br>
> ><br>
</div><div><div></div><div class="h5">> > Below is the info requested:<br>
> > (gdb) frame 0<br>
> > #0 0x00007f51999f221f in flat_db_insert (h=0x7f0978,<br>
> > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165<br>
> > 165 f = CON_FILE(h);<br>
> > (gdb) print h<br>
> > $1 = (const db_con_t *) 0x7f0978<br>
> > (gdb) print h->tail<br>
> > $2 = 0<br>
> > (gdb) print ((struct flat_con*)(h->tail))->file<br>
> > Cannot access memory at address 0x10<br>
> ><br>
> > Before each of the crashes yesterday, I saw these in the logs:<br>
> > Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:<br>
> > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in<br>
> > db_flatstore.so[7f51999ef000+5000]<br>
> > Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:<br>
> > segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in<br>
> > db_flatstore.so[7f726cb98000+5000]<br>
> ><br>
> > Are there any compiler flags I should use for debugging?<br>
> > (gdb) info locals<br>
> > f = <value optimized out><br>
> > i = <value optimized out><br>
> > l = <value optimized out><br>
> > p = <value optimized out><br>
> > __FUNCTION__ = "flat_db_insert"<br>
> ><br>
> > Thanks again.<br>
> ><br>
> ><br>
> > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu<br>
> > <<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>><br>
</div></div><div class="im">> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>>>><br>
> wrote:<br>
> ><br>
</div><div class="im">> > Hi,<br>
> ><br>
> > in frame 0, could you print:<br>
> > h<br>
> > h->tail<br>
> > ((struct flat_con*)(h->tail))->file<br>
> ><br>
> > Also, before crash, do you see in the logs any errors<br>
> from the<br>
> > db_flatstore module ?<br>
> ><br>
> > Regards,<br>
> > Bogdan<br>
> ><br>
> > thrillerbee wrote:<br>
> > > Bogdan,<br>
> > ><br>
> > > It crashed again tonight. I've attached the backtrace.<br>
> > ><br>
> > > Thanks.<br>
> > ><br>
> > > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee<br>
> > <<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>><br>
</div><div class="im">> > > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>>>> wrote:<br>
> > ><br>
</div><div><div></div><div class="h5">> > > Bogdan,<br>
> > ><br>
> > > Yes, I've attached 2 to my response - one was a crash<br>
> > from 10/29.<br>
> > > The other occurred a few minutes ago.<br>
> > ><br>
> > > Thanks.<br>
> > ><br>
> > > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu<br>
> > > <<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a><br>
> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a><br>
> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>>><br>
> > <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a><br>
> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>><br>
> > <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a><br>
> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>>>>> wrote:<br>
> > ><br>
> > > Any chance with the backtraces ?<br>
> > ><br>
> > > Regards,<br>
> > > Bogdan<br>
> > ><br>
> > > Anca Vamanu wrote:<br>
> > > > Hi,<br>
> > > ><br>
> > > > You need to inspect them with gdb, run: gdb<br>
> > > > path_to_opensips_executable<br>
> path_to_corefile, and<br>
> > then run<br>
> > > 'bt full'<br>
> > > > and send the output.<br>
> > > ><br>
> > > > Regards,<br>
> > > > --<br>
> > > > Anca Vamanu<br>
> > > > <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a><br>
> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>> <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
> > <<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>><br>
> > > ><br>
> > > ><br>
> > > > On 10/14/2010 10:12 PM, thrillerbee wrote:<br>
> > > >> I have this info from dmesg:<br>
> > > >><br>
> > > >> [1985853.285221] opensips[30865]: segfault<br>
> at 10 ip<br>
> > > 7f43899ce21f sp<br>
> > > >> 7fff8de1cf40 error 4 in<br>
> > db_flatstore.so[7f43899cb000+5000]<br>
> > > >> [1985856.379671] opensips[30858]: segfault<br>
> at 10 ip<br>
> > > 7f43899ce21f sp<br>
> > > >> 7fff8de1cf40 error 4 in<br>
> > db_flatstore.so[7f43899cb000+5000]<br>
> > > >> [1985896.961279] opensips[30868]: segfault<br>
> at 10 ip<br>
> > > 7f43899ce21f sp<br>
> > > >> 7fff8de1cf40 error 4 in<br>
> > db_flatstore.so[7f43899cb000+5000]<br>
> > > >><br>
> > > >> [2000131.245512] opensips[17672]: segfault<br>
> at 10 ip<br>
> > > 7fd0f21fb21f sp<br>
> > > >> 7fff3a3b4f00 error 4 in<br>
> > db_flatstore.so[7fd0f21f8000+5000]<br>
> > > >> [2000161.735962] opensips[17668]: segfault<br>
> at 10 ip<br>
> > > 7fd0f21fb21f sp<br>
> > > >> 7fff3a3b4e40 error 4 in<br>
> > db_flatstore.so[7fd0f21f8000+5000]<br>
> > > >> [2000167.299402] opensips[17670]: segfault<br>
> at 10 ip<br>
> > > 7fd0f21fb21f sp<br>
> > > >> 7fff3a3b4f00 error 4 in<br>
> > db_flatstore.so[7fd0f21f8000+5000]<br>
> > > >><br>
> > > >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee<br>
> > > <<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>>><br>
> > > >> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>><br>
> > > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>><br>
> > <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
> <mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>>>>>> wrote:<br>
> > > >><br>
> > > >> When OpenSIPS crashes, three corefiles are<br>
> > generated<br>
> > > that are<br>
> > > >> 2.1GB in size. How do I use these files to<br>
> > understand<br>
> > > what's<br>
> > > >> causing the crash?<br>
> > > >><br>
> > > >> Thanks.<br>
> > > >><br>
> ><br>
><br>
<br>
<br>
--<br>
</div></div><div><div></div><div class="h5">Bogdan-Andrei Iancu<br>
OpenSIPS Bootcamp<br>
15 - 19 November 2010, Edison, New Jersey, USA<br>
<a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>