Bogdan,<div><br></div><div>Are these memlogs useful? Please let me know if I can provide any other info.</div><div><br></div><div>Thanks again.<br><br><div class="gmail_quote">On Sat, Nov 13, 2010 at 4:20 PM, thrillerbee <span dir="ltr"><<a href="mailto:thrillerbee@gmail.com">thrillerbee@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Bogdan,<div><br></div><div>Since I made those changes, it doesn't core dump anymore - it just runs out of memory and stops processing packets. It happened on both of my proxies - the mem dump can be downloaded here:</div>
<div><span style="font-family:arial, sans-serif;font-size:13px;border-collapse:collapse"><div><a href="http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz" style="color:rgb(0, 101, 204)" target="_blank">http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz</a></div>
<div><a href="http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz" style="color:rgb(0, 101, 204)" target="_blank">http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz</a></div><div><br></div><div>Thanks.</div>
</span><div><div></div><div class="h5"><br><div class="gmail_quote">On Thu, Nov 11, 2010 at 12:43 PM, thrillerbee <span dir="ltr"><<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Bogdan,<div><br></div><div>I had already increased the PKG_MEM_POOL_SIZE by *10. I'm making the changes suggested in the link you provided to try to narrow down the memory issue. I'll send over as soon as I have another crash with mem info.</div>
<div><br></div><div>Thanks.<div><div></div><div><br><br><div class="gmail_quote">On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This last crash had the same bt as the one from previous email ? the prev fix I made reports this:<div><br>
ERROR:db_flatstore:flat_db_insert: uninitialized connection<br></div>
and does not crash, so this time the crash may be in a different place.<br>
<br>
Now, it seams after all that the root of your problem is the mem exhaustion. To see what causes this (if a mem leak or simply not enough mem), there is a doc - <a href="http://www.opensips.org/Resources/DocsTsMem" target="_blank">http://www.opensips.org/Resources/DocsTsMem</a> (it is for shm, but the same for pkg).<br>
<br>
Maybe, first you should simply try to increase the pkg mem ( in config.h you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile and reinstall after that.<br>
<br>
If more pkg mem does not solve the problem, I can help you with using the memory debugger to see what is going on there.<br>
<br>
Regards,<br>
Bogdan<br>
<br>
<br>
thrillerbee wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
My other proxy crashed as well with these ERRORs in the syslog:<br>
<br>
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure<br>
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name<br>
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection<br>
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection<br>
...<br>
Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection<br>
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection<br>
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left<br>
Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000]<br>
<br></div><div>
On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee <<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a> <mailto:<a href="mailto:thrillerbee@gmail.com" target="_blank">thrillerbee@gmail.com</a>>> wrote:<br>
<br>
Bogdan,<br>
<br>
Well, I spoke too soon - it's not just an issue with the<br>
opensipsctl fifo calls - looks more like a memory leak. It<br>
crashed again today, but I did get some errors in the syslog this<br>
time right before the crash:<br>
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:<br>
ERROR:db_flatstore:new_flat_id: no pkg memory left<br>
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:<br>
segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in<br>
db_flatstore.so[7fa7ff749000+5000]<br>
To be thorough, I've attached the backtrace & output from print<br>
commands (although they're the same as before).<br>
<br>
To answer your question, yes - I do use the flat_rotate MI command.<br>
<br>
Thanks again.<br>
<br>
On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu<br></div><div><div></div><div>
<<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>>> wrote:<br>
<br>
Hi,<br>
<br>
opensipsctl takes care that each command takes a separate fifo<br>
reply, so here it should be no problem. But the problem may be<br>
when comes with sending multiple commands (via FIFO) in the<br>
same time - this translates into parallel writes to the same<br>
file and depends on the atomicity of the write op.<br>
<br>
But in the worst case, a mixture at the FIFO level may lead to<br>
bogus command and not in any kind of crash....Do you use the<br>
"flat_rotate" MI command ?<br>
<br>
Regards,<br>
Bogdan<br>
<br>
thrillerbee wrote:<br>
<br>
Bogdan,<br>
<br>
It seems the issue is with 'opensipsctl fifo' - it's very<br>
sensitive to simultaneous calls. Basically, I've combined<br>
all my scripts to prevent 'opensipsctl fifo' from being<br>
called too frequently and that seems (so far) to have<br>
mitigated the issue. Is there anything one should know<br>
about how (not) to use /opensipsctl/?<br>
<br>
Thanks.<br>
<br>
On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu<br>
<<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a><br>
<mailto:<a href="mailto:bogdan@voice-system.ro" target="_blank">bogdan@voice-system.ro</a>>>> wrote:<br>
<br>
Hi,<br>
<br>
strange if you do not have any errors :(....<br>
<br>
I just made a fix on both trunk and 1.6 to extend some<br>
checks in<br>
flatstore and prevent crashing (even if the DB op will<br>
not be<br>
executed).<br>
<br>
Could you update from SVN and see if stops crashing ?<br>
<br>
Regards,<br>
Bogdan<br>
<br>
<br></div></div><div>
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
</div></blockquote><div><div></div><div>
<br>
<br>
-- <br>
Bogdan-Andrei Iancu<br>
OpenSIPS Bootcamp<br>
15 - 19 November 2010, Edison, New Jersey, USA<br>
<a href="http://www.voice-system.ro" target="_blank">www.voice-system.ro</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>