<div dir="ltr">Hi Bogdan-Andrei,<div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>This does look to be frozen in this state too. I guess GDB is the next step.</div><div><br></div><div>Andrew</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 6 Oct 2021 at 17:12, Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div>
    <font face="monospace">Hi Andrew,<br>
      <br>
      What you can do is, after getting the FIFO blocked, to do a trap
      "opensips-cli trap" to try to see what the FIFO process is doing
      (or trying to do). If the trap does not work, let me know, I will
      give you the instructions on how to directly attache with GDB to
      the process.<br>
      <br>
      Best regards,<br>
    </font>
    <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
OpenSIPS eBootcamp 2021 
  <a href="https://opensips.org/training/OpenSIPS_eBootcamp_2021/" target="_blank">https://opensips.org/training/OpenSIPS_eBootcamp_2021/</a></pre>
    <div>On 10/6/21 12:24 AM, Andrew Yager
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div>
          <div dir="ltr">
            <div dir="ltr">
              <div>
                <div dir="ltr">
                  <div dir="ltr">
                    <div dir="ltr">
                      <div dir="ltr">
                        <div dir="ltr">
                          <div dir="ltr">
                            <div dir="ltr">
                              <div dir="ltr">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div dir="ltr">
                                          <div>Also restarting opensips
                                            produces this:</div>
                                          <div><br>
                                          </div>
                                          <div>Restarting opensips (via
                                            systemctl):
                                            opensips.serviceERROR:
                                            communication exception for
                                            'which' returned: cannot
                                            access fifo file
                                            /tmp/opensips_fifo: [Errno
                                            32] Broken pipe!<br>
                                          </div>
                                          <div><br>
                                          </div>
                                          <div>Andrew</div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
        <br>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Wed, 6 Oct 2021 at 08:23,
          Andrew Yager <<a href="mailto:andrew@rwts.com.au" target="_blank">andrew@rwts.com.au</a>> wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">Hi,
            <div><br>
            </div>
            <div>
              <div>
                <div dir="ltr">
                  <div dir="ltr">
                    <div dir="ltr">
                      <div dir="ltr">
                        <div dir="ltr">
                          <div dir="ltr">
                            <div dir="ltr">
                              <div dir="ltr">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div dir="ltr">
                                          <div dir="ltr">
                                            <div>Just replicated this on
                                              the 3.2.2 nightly build.
                                              Debug logs are enabled,
                                              and this is all I can see
                                              on the last request that
                                              stalled:</div>
                                            <div><br>
                                            </div>
                                            <div>Oct  6 04:49:32 hvprxy
                                              osips[1186130]:
                                              DBG:mi_fifo:mi_fifo_callback:
                                              running command
                                              [{"jsonrpc": "2.0", "id":
                                              "5319", "method": "which",
                                              "params": []}]<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186130]:
                                              DBG:mi_fifo:mi_fifo_callback:
                                              got mi response =
                                              [0x558865cc9560]<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186130]:
                                              DBG:mi_fifo:mi_fifo_callback:
                                              running command
                                              [{"jsonrpc": "2.0", "id":
                                              "8377", "method":
                                              "get_statistics",
                                              "params": [["all"]]}]<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186133]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186134]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186135]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186136]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186137]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186138]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186139]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186140]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186141]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186142]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186143]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186144]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186145]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186146]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186147]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186148]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186149]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186150]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186151]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186152]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186153]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186154]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186155]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186156]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186157]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186158]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186159]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186160]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186161]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186162]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186163]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186164]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186165]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186166]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186167]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186168]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186169]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186170]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186171]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186172]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186173]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                              Oct  6 04:49:32 hvprxy
                                              osips[1186174]:
                                              DBG:core:ipc_handle_job:
                                              received job type 0[RPC]
                                              from process 1<br>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
              <br>
            </div>
            <div>Andrew</div>
          </div>
          <br>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Mon, 4 Oct 2021 at
              22:37, Andrew Yager <<a href="mailto:andrew@rwts.com.au" target="_blank">andrew@rwts.com.au</a>> wrote:<br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <div dir="ltr">
                      <div>
                        <div dir="ltr">
                          <div dir="ltr">
                            <div dir="ltr">
                              <div dir="ltr">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div dir="ltr">
                                          <div dir="ltr">
                                            <div dir="ltr">
                                              <div dir="ltr">
                                                <div dir="ltr">
                                                  <div>Just further to
                                                    this, I think it's
                                                    the ul_dump command
                                                    that seems to cause
                                                    the issue first.</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Andrew</div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
                <br>
              </div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr" class="gmail_attr">On Sat, 2 Oct 2021 at
                  13:05, Andrew Yager <<a href="mailto:andrew@rwts.com.au" target="_blank">andrew@rwts.com.au</a>>
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                  <div dir="ltr">
                    <div dir="ltr">
                      <div dir="ltr">
                        <div dir="ltr">
                          <div dir="ltr">
                            <div dir="ltr">
                              <div dir="ltr">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div dir="ltr">
                                          <div dir="ltr">
                                            <div dir="ltr">
                                              <div dir="ltr">
                                                <div dir="ltr">
                                                  <div>Hi,</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Not entirely sure
                                                    where to start
                                                    digging on this one.
                                                    On 3.1.3 we've had
                                                    an issue appear
                                                    "suddenly" whereby
                                                    our mi command
                                                    output seems to
                                                    "block" and not
                                                    return any useful
                                                    data. Restarting
                                                    opensips processes
                                                    restores comms.</div>
                                                  <div><br>
                                                  </div>
                                                  <div>We end up with a
                                                    huge number of
                                                    opensips_fifo_reply_\*
                                                    files in the /tmp
                                                    directory, but
                                                    effectively no data
                                                    is ever written and
                                                    opensips-cli
                                                    freezes.</div>
                                                  <div><br>
                                                  </div>
                                                  <div>We've restarted
                                                    twice to "resolve"
                                                    now, but the issue
                                                    appears to reoccur.
                                                    We're using the mi
                                                    get_statistics
                                                    command to monitor
                                                    stats/health and
                                                    uldump to pull some
                                                    detail about usrloc
                                                    data every 5
                                                    minutes.</div>
                                                  <div><br>
                                                  </div>
                                                  <div>I don't think I
                                                    can cause a debug
                                                    level increase
                                                    because the MI
                                                    process doesn't seem
                                                    to be communicating
                                                    (at least it doesn't
                                                    seem to work).</div>
                                                  <div><br>
                                                  </div>
                                                  <div>opensips        
                                                                       
                                                    3.1.3-1</div>
                                                  <div>opensips-cli    
                                                                       
0.1~20210707~572d2db-</div>
                                                  <div>5.4.0-88-generic
                                                    #99-Ubuntu SMP Thu
                                                    Sep 23 17:29:00 UTC
                                                    2021 x86_64 x86_64
                                                    x86_64 GNU/Linux</div>
                                                  <div><br>
                                                  </div>
                                                  <div>I can restart
                                                    again, but if this
                                                    is going to keep
                                                    happening, I want to
                                                    make sure I have
                                                    something useful in
                                                    the logs to be able
                                                    to trace down.</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Andrew <br>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </blockquote>
              </div>
            </blockquote>
          </div>
        </blockquote>
      </div>
      <br>
      <fieldset></fieldset>
      <pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
    </blockquote>
    <br>
  </div>

</blockquote></div>