<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<font face="monospace">Hi Andrew,<br>
<br>
OK, let's try these steps:<br>
<br>
1) right after starting opensips, do an "opensips-cli -x mi ps"
and identify the PID of the FIFO process <br>
<br>
2) wait for the blocking to happen<br>
<br>
3) from a console with root privileges , do "gdb opensips
_PID_OF_FIFO_" and run "bt full"<br>
<br>
Best regards,<br>
</font>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
OpenSIPS eBootcamp 2021
<a class="moz-txt-link-freetext" href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a></pre>
<div class="moz-cite-prefix">On 10/7/21 2:19 AM, Andrew Yager wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEX38CND+cDWN2rzaSxGnpF3FwXbf=HP4GdVj=cib+QqgcvmrA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hi Bogdan-Andrei,
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>This does look to be
frozen in this state too.
I guess GDB is the next
step.</div>
<div><br>
</div>
<div>Andrew</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 6 Oct 2021 at 17:12,
Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true">bogdan@opensips.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> <font face="monospace">Hi Andrew,<br>
<br>
What you can do is, after getting the FIFO blocked, to do
a trap "opensips-cli trap" to try to see what the FIFO
process is doing (or trying to do). If the trap does not
work, let me know, I will give you the instructions on how
to directly attache with GDB to the process.<br>
<br>
Best regards,<br>
</font>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="https://www.opensips-solutions.com" target="_blank" moz-do-not-send="true">https://www.opensips-solutions.com</a>
OpenSIPS eBootcamp 2021
<a href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/" target="_blank" moz-do-not-send="true">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a></pre>
<div>On 10/6/21 12:24 AM, Andrew Yager wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Also restarting
opensips produces
this:</div>
<div><br>
</div>
<div>Restarting
opensips (via
systemctl):
opensips.serviceERROR:
communication
exception for
'which' returned:
cannot access fifo
file
/tmp/opensips_fifo:
[Errno 32] Broken
pipe!<br>
</div>
<div><br>
</div>
<div>Andrew</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 6 Oct 2021 at
08:23, Andrew Yager <<a
href="mailto:andrew@rwts.com.au" target="_blank"
moz-do-not-send="true">andrew@rwts.com.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Just replicated
this on the 3.2.2
nightly build.
Debug logs are
enabled, and this
is all I can see
on the last
request that
stalled:</div>
<div><br>
</div>
<div>Oct 6 04:49:32
hvprxy
osips[1186130]:
DBG:mi_fifo:mi_fifo_callback:
running command
[{"jsonrpc":
"2.0", "id":
"5319", "method":
"which", "params":
[]}]<br>
Oct 6 04:49:32
hvprxy
osips[1186130]:
DBG:mi_fifo:mi_fifo_callback:
got mi response =
[0x558865cc9560]<br>
Oct 6 04:49:32
hvprxy
osips[1186130]:
DBG:mi_fifo:mi_fifo_callback:
running command
[{"jsonrpc":
"2.0", "id":
"8377", "method":
"get_statistics",
"params":
[["all"]]}]<br>
Oct 6 04:49:32
hvprxy
osips[1186133]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186134]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186135]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186136]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186137]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186138]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186139]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186140]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186141]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186142]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186143]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186144]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186145]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186146]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186147]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186148]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186149]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186150]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186151]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186152]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186153]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186154]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186155]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186156]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186157]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186158]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186159]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186160]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186161]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186162]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186163]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186164]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186165]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186166]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186167]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186168]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186169]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186170]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186171]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186172]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186173]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
Oct 6 04:49:32
hvprxy
osips[1186174]:
DBG:core:ipc_handle_job:
received job type
0[RPC] from
process 1<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<div>Andrew</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 4 Oct 2021
at 22:37, Andrew Yager <<a
href="mailto:andrew@rwts.com.au" target="_blank"
moz-do-not-send="true">andrew@rwts.com.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Just
further to
this, I think
it's the
ul_dump
command that
seems to cause
the issue
first.</div>
<div><br>
</div>
<div>Andrew</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, 2 Oct
2021 at 13:05, Andrew Yager <<a
href="mailto:andrew@rwts.com.au"
target="_blank" moz-do-not-send="true">andrew@rwts.com.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>Not
entirely sure
where to start
digging on
this one. On
3.1.3 we've
had an issue
appear
"suddenly"
whereby our mi
command output
seems to
"block" and
not return any
useful data.
Restarting
opensips
processes
restores
comms.</div>
<div><br>
</div>
<div>We end up
with a huge
number of
opensips_fifo_reply_\*
files in the
/tmp
directory, but
effectively no
data is ever
written and
opensips-cli
freezes.</div>
<div><br>
</div>
<div>We've
restarted
twice to
"resolve" now,
but the issue
appears to
reoccur. We're
using the mi
get_statistics
command to
monitor
stats/health
and uldump to
pull some
detail about
usrloc data
every 5
minutes.</div>
<div><br>
</div>
<div>I don't
think I can
cause a debug
level increase
because the MI
process
doesn't seem
to be
communicating
(at least it
doesn't seem
to work).</div>
<div><br>
</div>
<div>opensips
3.1.3-1</div>
<div>opensips-cli
0.1~20210707~572d2db-</div>
<div>5.4.0-88-generic
#99-Ubuntu SMP
Thu Sep 23
17:29:00 UTC
2021 x86_64
x86_64 x86_64
GNU/Linux</div>
<div><br>
</div>
<div>I can
restart again,
but if this is
going to keep
happening, I
want to make
sure I have
something
useful in the
logs to be
able to trace
down.</div>
<div><br>
</div>
<div>Andrew <br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank" moz-do-not-send="true">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>