Bogdan,<div><br></div><div>I am not seeing any other errors in the logs.  Is there anything else I can look at?  Both proxies are crashing everyday.</div><div><br></div><div>Thanks.</div><div><br><div class="gmail_quote">On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu <span dir="ltr">&lt;<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">I got some clue about what is happening - as you are using flatstore for<br>
acc, the acc module does not check the success of the &quot;use_table&quot; DB<br>
operation - in 90% of the cases (for mysql, postgres, etc) this function<br>
cannot fail, but it seams that for flatstore can. And if it fails, the<br>
h-&gt;tail is set to NULL, leading to crash.<br>
<br>
Now, before the crash itself, do you see any other ERROR messages (even<br>
long before the crash) related to flatstore module ? Try:<br>
    grep &quot;ERROR&quot; opensips_log_file | grep &quot;flatstore&quot;<br>
<div class="im"><br>
Regards,<br>
Bogdan<br>
<br>
thrillerbee wrote:<br>
&gt; Bogdan,<br>
&gt;<br>
</div><div class="im">&gt; One more detail that may help - I added another OpenSIPS proxy in<br>
&gt; parallel with this one (&amp; load balancing between the two) and now both<br>
&gt; OpenSIPS proxies crash at the same time (within a couple seconds).<br>
&gt;<br>
&gt; I can provide more core dumps if it will help.<br>
&gt;<br>
&gt; Thanks.<br>
&gt;<br>
&gt;<br>
&gt; On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee &lt;<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
</div><div class="im">&gt; &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>&gt;&gt; wrote:<br>
&gt;<br>
&gt;     Bogdan,<br>
&gt;<br>
</div><div><div></div><div class="h5">&gt;     Below is the info requested:<br>
&gt;     (gdb) frame 0<br>
&gt;     #0  0x00007f51999f221f in flat_db_insert (h=0x7f0978,<br>
&gt;     k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165<br>
&gt;     165             f = CON_FILE(h);<br>
&gt;     (gdb) print h<br>
&gt;     $1 = (const db_con_t *) 0x7f0978<br>
&gt;     (gdb) print h-&gt;tail<br>
&gt;     $2 = 0<br>
&gt;     (gdb) print ((struct flat_con*)(h-&gt;tail))-&gt;file<br>
&gt;     Cannot access memory at address 0x10<br>
&gt;<br>
&gt;     Before each of the crashes yesterday, I saw these in the logs:<br>
&gt;     Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:<br>
&gt;     segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in<br>
&gt;     db_flatstore.so[7f51999ef000+5000]<br>
&gt;     Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:<br>
&gt;     segfault at 10 ip 7f726cb9b21f sp 7ffff083f6f0 error 4 in<br>
&gt;     db_flatstore.so[7f726cb98000+5000]<br>
&gt;<br>
&gt;     Are there any compiler flags I should use for debugging?<br>
&gt;     (gdb) info locals<br>
&gt;     f = &lt;value optimized out&gt;<br>
&gt;     i = &lt;value optimized out&gt;<br>
&gt;     l = &lt;value optimized out&gt;<br>
&gt;     p = &lt;value optimized out&gt;<br>
&gt;     __FUNCTION__ = &quot;flat_db_insert&quot;<br>
&gt;<br>
&gt;     Thanks again.<br>
&gt;<br>
&gt;<br>
&gt;     On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu<br>
</div></div><div class="im">&gt;     &lt;<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> &lt;mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>&gt;&gt; wrote:<br>
&gt;<br>
</div><div class="im">&gt;         Hi,<br>
&gt;<br>
&gt;         in frame 0, could you print:<br>
&gt;            h<br>
&gt;            h-&gt;tail<br>
&gt;            ((struct flat_con*)(h-&gt;tail))-&gt;file<br>
&gt;<br>
&gt;         Also, before crash, do you see in the logs any errors from the<br>
&gt;         db_flatstore module ?<br>
&gt;<br>
&gt;         Regards,<br>
&gt;         Bogdan<br>
&gt;<br>
&gt;         thrillerbee wrote:<br>
&gt;         &gt; Bogdan,<br>
&gt;         &gt;<br>
&gt;         &gt; It crashed again tonight.  I&#39;ve attached the backtrace.<br>
&gt;         &gt;<br>
&gt;         &gt; Thanks.<br>
&gt;         &gt;<br>
&gt;         &gt; On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee<br>
&gt;         &lt;<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a> &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>&gt;<br>
</div><div class="im">&gt;         &gt; &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
&gt;         &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>&gt;&gt;&gt; wrote:<br>
&gt;         &gt;<br>
</div><div><div></div><div class="h5">&gt;         &gt;     Bogdan,<br>
&gt;         &gt;<br>
&gt;         &gt;     Yes,  I&#39;ve attached 2 to my response - one was a crash<br>
&gt;         from 10/29.<br>
&gt;         &gt;      The other occurred a few minutes ago.<br>
&gt;         &gt;<br>
&gt;         &gt;     Thanks.<br>
&gt;         &gt;<br>
&gt;         &gt;     On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu<br>
&gt;         &gt;     &lt;<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> &lt;mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>&gt;<br>
&gt;         &lt;mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a><br>
&gt;         &lt;mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>&gt;&gt;&gt; wrote:<br>
&gt;         &gt;<br>
&gt;         &gt;         Any chance with the backtraces ?<br>
&gt;         &gt;<br>
&gt;         &gt;         Regards,<br>
&gt;         &gt;         Bogdan<br>
&gt;         &gt;<br>
&gt;         &gt;         Anca Vamanu wrote:<br>
&gt;         &gt;         &gt; Hi,<br>
&gt;         &gt;         &gt;<br>
&gt;         &gt;         &gt; You need to inspect them with gdb, run: gdb<br>
&gt;         &gt;         &gt; path_to_opensips_executable path_to_corefile, and<br>
&gt;         then run<br>
&gt;         &gt;         &#39;bt full&#39;<br>
&gt;         &gt;         &gt; and send the output.<br>
&gt;         &gt;         &gt;<br>
&gt;         &gt;         &gt; Regards,<br>
&gt;         &gt;         &gt; --<br>
&gt;         &gt;         &gt; Anca Vamanu<br>
&gt;         &gt;         &gt; <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> &lt;<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>&gt;<br>
&gt;         &lt;<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>&gt;<br>
&gt;         &gt;         &gt;<br>
&gt;         &gt;         &gt;<br>
&gt;         &gt;         &gt; On 10/14/2010 10:12 PM, thrillerbee wrote:<br>
&gt;         &gt;         &gt;&gt; I have this info from dmesg:<br>
&gt;         &gt;         &gt;&gt;<br>
&gt;         &gt;         &gt;&gt; [1985853.285221] opensips[30865]: segfault at 10 ip<br>
&gt;         &gt;         7f43899ce21f sp<br>
&gt;         &gt;         &gt;&gt; 7fff8de1cf40 error 4 in<br>
&gt;         db_flatstore.so[7f43899cb000+5000]<br>
&gt;         &gt;         &gt;&gt; [1985856.379671] opensips[30858]: segfault at 10 ip<br>
&gt;         &gt;         7f43899ce21f sp<br>
&gt;         &gt;         &gt;&gt; 7fff8de1cf40 error 4 in<br>
&gt;         db_flatstore.so[7f43899cb000+5000]<br>
&gt;         &gt;         &gt;&gt; [1985896.961279] opensips[30868]: segfault at 10 ip<br>
&gt;         &gt;         7f43899ce21f sp<br>
&gt;         &gt;         &gt;&gt; 7fff8de1cf40 error 4 in<br>
&gt;         db_flatstore.so[7f43899cb000+5000]<br>
&gt;         &gt;         &gt;&gt;<br>
&gt;         &gt;         &gt;&gt; [2000131.245512] opensips[17672]: segfault at 10 ip<br>
&gt;         &gt;         7fd0f21fb21f sp<br>
&gt;         &gt;         &gt;&gt; 7fff3a3b4f00 error 4 in<br>
&gt;         db_flatstore.so[7fd0f21f8000+5000]<br>
&gt;         &gt;         &gt;&gt; [2000161.735962] opensips[17668]: segfault at 10 ip<br>
&gt;         &gt;         7fd0f21fb21f sp<br>
&gt;         &gt;         &gt;&gt; 7fff3a3b4e40 error 4 in<br>
&gt;         db_flatstore.so[7fd0f21f8000+5000]<br>
&gt;         &gt;         &gt;&gt; [2000167.299402] opensips[17670]: segfault at 10 ip<br>
&gt;         &gt;         7fd0f21fb21f sp<br>
&gt;         &gt;         &gt;&gt; 7fff3a3b4f00 error 4 in<br>
&gt;         db_flatstore.so[7fd0f21f8000+5000]<br>
&gt;         &gt;         &gt;&gt;<br>
&gt;         &gt;         &gt;&gt; On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee<br>
&gt;         &gt;         &lt;<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
&gt;         &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>&gt; &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
&gt;         &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>&gt;&gt;<br>
&gt;         &gt;         &gt;&gt; &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
&gt;         &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>&gt;<br>
&gt;         &gt;         &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a><br>
&gt;         &lt;mailto:<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>&gt;&gt;&gt;&gt; wrote:<br>
&gt;         &gt;         &gt;&gt;<br>
&gt;         &gt;         &gt;&gt;     When OpenSIPS crashes, three corefiles are<br>
&gt;         generated<br>
&gt;         &gt;         that are<br>
&gt;         &gt;         &gt;&gt;     2.1GB in size.  How do I use these files to<br>
&gt;         understand<br>
&gt;         &gt;         what&#39;s<br>
&gt;         &gt;         &gt;&gt;     causing the crash?<br>
&gt;         &gt;         &gt;&gt;<br>
&gt;         &gt;         &gt;&gt;     Thanks.<br>
&gt;         &gt;         &gt;&gt;<br>
&gt;         &gt;         &gt;&gt;<br>
&gt;         &gt;         &gt;<br>
&gt;         &gt;<br>
&gt;         ------------------------------------------------------------------------<br>
&gt;         &gt;         &gt;<br>
&gt;         &gt;         &gt; _______________________________________________<br>
&gt;         &gt;         &gt; Users mailing list<br>
&gt;         &gt;         &gt; <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
&gt;         &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>&gt;<br>
&gt;         &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
&gt;         &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>&gt;&gt;<br>
&gt;         &gt;         &gt;<br>
&gt;         <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
&gt;         &gt;         &gt;<br>
&gt;         &gt;<br>
&gt;         &gt;<br>
&gt;         &gt;         --<br>
&gt;         &gt;         Bogdan-Andrei Iancu<br>
&gt;         &gt;         OpenSIPS Bootcamp<br>
&gt;         &gt;         15 - 19 November 2010, Edison, New Jersey, USA<br>
&gt;         &gt;         <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> &lt;<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>&gt;<br>
&gt;         &lt;<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>&gt;<br>
&gt;         &gt;<br>
&gt;         &gt;<br>
&gt;         &gt;         _______________________________________________<br>
&gt;         &gt;         Users mailing list<br>
&gt;         &gt;         <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
&gt;         &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>&gt;<br>
&gt;         &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
&gt;         &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>&gt;&gt;<br>
&gt;         &gt;         <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
&gt;         &gt;<br>
&gt;         &gt;<br>
&gt;         &gt;<br>
&gt;         &gt;<br>
&gt;         ------------------------------------------------------------------------<br>
&gt;         &gt;<br>
&gt;         &gt; _______________________________________________<br>
&gt;         &gt; Users mailing list<br>
&gt;         &gt; <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a> &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>&gt;<br>
&gt;         &gt; <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
&gt;<br>
&gt;<br>
&gt;         --<br>
&gt;         Bogdan-Andrei Iancu<br>
&gt;         OpenSIPS Bootcamp<br>
&gt;         15 - 19 November 2010, Edison, New Jersey, USA<br>
&gt;         <a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a> &lt;<a href="http://www.voice-system.ro" target="_blank">http://www.voice-system.ro</a>&gt;<br>
&gt;<br>
&gt;<br>
&gt;         _______________________________________________<br>
&gt;         Users mailing list<br>
&gt;         <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a> &lt;mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>&gt;<br>
&gt;         <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; ------------------------------------------------------------------------<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; Users mailing list<br>
&gt; <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
&gt; <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
&gt;<br>
<br>
<br>
--<br>
Bogdan-Andrei Iancu<br>
OpenSIPS Bootcamp<br>
15 - 19 November 2010, Edison, New Jersey, USA<br>
<a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>