Bogdan,<div>I do periodic cache stores also in the background. But that's a rather new thing I do now. Even before I was doing those I got these hangs.. I'll try to get another bt that shows it hanging but without the cache_store. However, I think the same underlying problem is affecting both.</div>
<div><br><br><div class="gmail_quote">On Tue, Sep 22, 2009 at 7:43 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Brett,<br>
<br>
hmm...that is rather strange as the backtrace you show here definitely<br>
is from the fifo process and the run command is clear to be the<br>
"cache_store".<br>
<br>
Maybe there is some other app that (in the same time with your tests) do<br>
run some caching commands??<br>
<br>
BTW, can you simply run via fifo the the cache_store command via<br>
memcached backend? Just to see if that is blocking or not.<br>
<div class="im"><br>
Regards,<br>
Bogdan<br>
<br>
Brett Nemeroff wrote:<br>
</div><div><div></div><div class="h5">> Bogdan,<br>
> Here's my bt full. Towards the top there is a cache_store line. The<br>
> fifo operation I'm waiting on is a profile_get_values.. Interesting<br>
> that I don't call cache_store in any of these hung opensipsctl calls.<br>
><br>
> # ps ax|grep opensipsctl<br>
> 17929 ? S 0:00 /bin/bash /usr/local/sbin/opensipsctl fifo<br>
> profile_get_values trunk<br>
> 17974 ? S 0:00 /bin/bash /usr/local/sbin/opensipsctl fifo<br>
> profile_get_values trunk<br>
> 17976 ? S 0:00 /bin/bash /usr/local/sbin/opensipsctl fifo<br>
> profile_get_values trunk<br>
> 27643 ? S 0:00 /bin/bash /usr/local/sbin/opensipsctl fifo<br>
> profile_get_size inbound<br>
> 27687 ? S 0:00 /bin/bash /usr/local/sbin/opensipsctl fifo<br>
> profile_get_size inbound<br>
> 27688 ? S 0:00 /bin/bash /usr/local/sbin/opensipsctl fifo<br>
> profile_get_size inbound<br>
><br>
><br>
> I took the mi_fifo ps for the gdb..<br>
><br>
> 0x00000034c1ac54f0 in __read_nocancel () from /lib64/libc.so.6<br>
> (gdb) bt full<br>
> #0 0x00000034c1ac54f0 in __read_nocancel () from /lib64/libc.so.6<br>
> No symbol table info available.<br>
> #1 0x00000034c1a6cbb7 in _IO_new_file_underflow () from /lib64/libc.so.6<br>
> No symbol table info available.<br>
> #2 0x00000034c1a6d57e in _IO_default_uflow_internal () from<br>
> /lib64/libc.so.6<br>
> No symbol table info available.<br>
> #3 0x00000034c1a62664 in _IO_getline_info_internal () from<br>
> /lib64/libc.so.6<br>
> No symbol table info available.<br>
> #4 0x00000034c1a61509 in fgets () from /lib64/libc.so.6<br>
> No symbol table info available.<br>
> #5 0x00002b68c3c2e619 in mi_read_line (b=0x79cd50 ":cache_store",<br>
> max=1024, stream=0x1e81b110, read=0x7fffe834bad4) at fifo_fnc.c:255<br>
> retry_cnt = 0<br>
> len = <value optimized out><br>
> __FUNCTION__ = "mi_read_line"<br>
> #6 0x00002b68c3c2f52f in mi_fifo_server (fifo_stream=0x1e81b110) at<br>
> fifo_fnc.c:400<br>
> mi_cmd = (struct mi_root *) 0x7a1568<br>
> mi_rpl = (struct mi_root *) 0x7a17c0<br>
> hdl = (struct mi_handler *) 0x0<br>
> line_len = 35<br>
> file_sep = 0x79cd5c ""<br>
> command = 0x79cd51 "cache_store"<br>
> file = 0x79d160 "/tmp/opensips_receiver_1372"<br>
> f = (struct mi_cmd *) 0x7995e8<br>
> reply_stream = (FILE *) 0x1e82e7e0<br>
> __FUNCTION__ = "mi_fifo_server"<br>
> #7 0x00002b68c3c31001 in fifo_process (rank=<value optimized out>) at<br>
> mi_fifo.c:212<br>
> fifo_stream = (FILE *) 0x1e81b110<br>
> __FUNCTION__ = "fifo_process"<br>
> #8 0x000000000046b4bc in start_module_procs () at sr_module.c:585<br>
> m = (struct sr_module *) 0x76ca88<br>
> n = 0<br>
> l = 0<br>
> x = <value optimized out><br>
> __FUNCTION__ = "start_module_procs"<br>
> #9 0x000000000042745d in main (argc=5, argv=0x7fffe834bcf8) at main.c:823<br>
> cfg_log_stderr = <value optimized out><br>
> cfg_stream = (FILE *) 0x1e80c010<br>
> c = <value optimized out><br>
> r = 8<br>
> tmp = 0x7fffe834dbe4 ""<br>
> tmp_len = <value optimized out><br>
> port = <value optimized out><br>
> proto = <value optimized out><br>
> ret = <value optimized out><br>
> seed = 2557210813<br>
> rfd = <value optimized out><br>
> __FUNCTION__ = "main"<br>
><br>
><br>
> On Thu, Sep 17, 2009 at 11:10 AM, Bogdan-Andrei Iancu<br>
</div></div><div class="im">> <<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a> <mailto:<a href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>>> wrote:<br>
><br>
> Hi Brett,<br>
><br>
> if you get a hang, do try to attach with gdb to the fifo proc (you can<br>
> do a "fifo ps" in the beginning to see the PID of the fifo process).<br>
><br>
> A backtrace of the fifo proc will help in understanding the issue.<br>
><br>
> BTW - what about cpu load ?<br>
><br>
> Regards,<br>
> Bogdan<br>
><br>
> Brett Nemeroff wrote:<br>
> > Hello list,<br>
> > I periodically call opensipsctl fifo profile_get_values <profile<br>
> > name>. Manually, it always works great.. but called from cron every<br>
> > minute, it frequently hangs indefinitely until I kill it. Often<br>
> I run<br>
> > ps ax and see about 5 of them in there.<br>
> ><br>
> > Running 1.5.3. Any ideas?<br>
> > -Brett<br>
> ><br>
> ><br>
> ------------------------------------------------------------------------<br>
> ><br>
> > _______________________________________________<br>
> > Users mailing list<br>
</div>> > <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>><br>
<div class="im">> > <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
> ><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
</div>> <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a> <mailto:<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>><br>
<div><div></div><div class="h5">> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
><br>
><br>
> ------------------------------------------------------------------------<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>