<div dir="ltr">Hi Bogdan-Andrei,<div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>This does look to be frozen in this state too. I guess GDB is the next step.</div><div><br></div><div>Andrew</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 6 Oct 2021 at 17:12, Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<font face="monospace">Hi Andrew,<br>
<br>
What you can do is, after getting the FIFO blocked, to do a trap
"opensips-cli trap" to try to see what the FIFO process is doing
(or trying to do). If the trap does not work, let me know, I will
give you the instructions on how to directly attache with GDB to
the process.<br>
<br>
Best regards,<br>
</font>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
OpenSIPS eBootcamp 2021
<a href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/" target="_blank">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a></pre>
<div>On 10/6/21 12:24 AM, Andrew Yager
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Also restarting opensips
produces this:</div>
<div><br>
</div>
<div>Restarting opensips (via
systemctl):
opensips.serviceERROR:
communication exception for
'which' returned: cannot
access fifo file
/tmp/opensips_fifo: [Errno
32] Broken pipe!<br>
</div>
<div><br>
</div>
<div>Andrew</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 6 Oct 2021 at 08:23,
Andrew Yager <<a href="mailto:andrew@rwts.com.au" target="_blank">andrew@rwts.com.au</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Just replicated this on
the 3.2.2 nightly build.
Debug logs are enabled,
and this is all I can see
on the last request that
stalled:</div>
<div><br>
</div>
<div>Oct 6 04:49:32 hvprxy
osips[1186130]:
DBG:mi_fifo:mi_fifo_callback:
running command
[{"jsonrpc": "2.0", "id":
"5319", "method": "which",
"params": []}]<br>
Oct 6 04:49:32 hvprxy
osips[1186130]:
DBG:mi_fifo:mi_fifo_callback:
got mi response =
[0x558865cc9560]<br>
Oct 6 04:49:32 hvprxy
osips[1186130]:
DBG:mi_fifo:mi_fifo_callback:
running command
[{"jsonrpc": "2.0", "id":
"8377", "method":
"get_statistics",
"params": [["all"]]}]<br>
Oct 6 04:49:32 hvprxy
osips[1186133]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186134]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186135]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186136]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186137]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186138]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186139]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186140]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186141]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186142]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186143]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186144]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186145]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186146]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186147]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186148]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186149]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186150]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186151]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186152]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186153]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186154]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186155]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186156]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186157]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186158]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186159]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186160]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186161]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186162]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186163]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186164]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186165]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186166]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186167]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186168]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186169]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186170]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186171]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186172]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186173]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
Oct 6 04:49:32 hvprxy
osips[1186174]:
DBG:core:ipc_handle_job:
received job type 0[RPC]
from process 1<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div>Andrew</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 4 Oct 2021 at
22:37, Andrew Yager <<a href="mailto:andrew@rwts.com.au" target="_blank">andrew@rwts.com.au</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Just further to
this, I think it's
the ul_dump command
that seems to cause
the issue first.</div>
<div><br>
</div>
<div>Andrew</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, 2 Oct 2021 at
13:05, Andrew Yager <<a href="mailto:andrew@rwts.com.au" target="_blank">andrew@rwts.com.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>Not entirely sure
where to start
digging on this one.
On 3.1.3 we've had
an issue appear
"suddenly" whereby
our mi command
output seems to
"block" and not
return any useful
data. Restarting
opensips processes
restores comms.</div>
<div><br>
</div>
<div>We end up with a
huge number of
opensips_fifo_reply_\*
files in the /tmp
directory, but
effectively no data
is ever written and
opensips-cli
freezes.</div>
<div><br>
</div>
<div>We've restarted
twice to "resolve"
now, but the issue
appears to reoccur.
We're using the mi
get_statistics
command to monitor
stats/health and
uldump to pull some
detail about usrloc
data every 5
minutes.</div>
<div><br>
</div>
<div>I don't think I
can cause a debug
level increase
because the MI
process doesn't seem
to be communicating
(at least it doesn't
seem to work).</div>
<div><br>
</div>
<div>opensips
3.1.3-1</div>
<div>opensips-cli
0.1~20210707~572d2db-</div>
<div>5.4.0-88-generic
#99-Ubuntu SMP Thu
Sep 23 17:29:00 UTC
2021 x86_64 x86_64
x86_64 GNU/Linux</div>
<div><br>
</div>
<div>I can restart
again, but if this
is going to keep
happening, I want to
make sure I have
something useful in
the logs to be able
to trace down.</div>
<div><br>
</div>
<div>Andrew <br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
</blockquote></div>