<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <font face="monospace">Hi Yury,<br>
      <br>
      For the crash -> is there any core file to check ?<br>
      <br>
      For mem usage -> you should try to get a memory dump for
      further investigation [1].<br>
      <br>
      [1] <a class="moz-txt-link-freetext" href="https://opensips.org/Documentation/TroubleShooting-OutOfMem">https://opensips.org/Documentation/TroubleShooting-OutOfMem</a><br>
      <br>
      Best regards,<br>
    </font>
    <pre class="moz-signature" cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2022Athens/">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
    <div class="moz-cite-prefix">On 9/14/22 10:13 PM, Yury Kirsanov
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAD1_seuO_D_8Rkwq7zcaPqV+_UnqQ-UQG3F1UPxEqGGfLYzidA@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">Hi Bogdan,
        <div>Thanks a lot for your help and support! The only question I
          know have is why OpenSIPS was going into a crash if all TCP
          processes were blocked waiting for connection? It was starting
          to consume more and more memory and then it was crashing with
          a segfault upon reaching then -m memory parameter. I do
          understand that TCP listeners were in a blocking mode and were
          not able to do any work until the session could be fully
          established, not being able to forward any SIP packets, but
          isn't that a bug that OpenSIPS was starting to eat memory and
          then crash? Do I need to open a bug report on this? Thanks!</div>
        <div><br>
        </div>
        <div>Best regards,</div>
        <div>Yury.</div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Wed, Sep 14, 2022 at 10:58
          PM Bogdan-Andrei Iancu <<a
            href="mailto:bogdan@opensips.org" moz-do-not-send="true">bogdan@opensips.org</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div> <font face="monospace">Hi Yury,<br>
              <br>
              You need to check the TCP setting and to be sure your
              OpenSIPS will (1) not try to perform TCP connect against
              destination known not to be able to accept (like TCP/WS
              end points behind NAT) - see the tcp_no_new_conn_bflag [1]
              - or (2) not block for long time while attempting a
              connect - see the tcp_connect_timeout [2] or consider
              enabling async [3].<br>
              <br>
              [1]
              <a
href="https://www.opensips.org/Documentation/Script-CoreParameters-3-2#tcp_no_new_conn_bflag"
                target="_blank" moz-do-not-send="true">https://www.opensips.org/Documentation/Script-CoreParameters-3-2#tcp_no_new_conn_bflag</a><br>
              [2]
              <a
href="https://www.opensips.org/Documentation/Script-CoreParameters-3-2#tcp_connect_timeout"
                target="_blank" moz-do-not-send="true">https://www.opensips.org/Documentation/Script-CoreParameters-3-2#tcp_connect_timeout</a><br>
              [3] <a
href="https://opensips.org/html/docs/modules/3.2.x/proto_tcp.html#idp168992"
                target="_blank" moz-do-not-send="true">https://opensips.org/html/docs/modules/3.2.x/proto_tcp.html#idp168992</a><br>
              <br>
              Regards,<br>
            </font>
            <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" target="_blank" moz-do-not-send="true">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" target="_blank" moz-do-not-send="true">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
            <div>On 9/13/22 12:01 PM, Yury Kirsanov wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="auto">Hi Bogdan,
                <div dir="auto">Thanks for this update, but it looks
                  like I can't check autoscaler because of this first
                  issue with blocking TCP connect. Is there a way to
                  resolve it? Am I doing something wrong? Or is that
                  something to do with OpenSIPS code? As yes, you're
                  right, as soon as I restart OpenSIPS having a lot of
                  SIP devices trying to connect to it - it goes crazy,
                  starts to consume memory and stops to forward packets
                  sitting there at 100% load until it runs out of memory
                  and segfaults. Sometimes I can't even restart it to
                  come to normal state to make it work, it just loops
                  into same crash whatever I try to do.</div>
                <div dir="auto"><br>
                </div>
                <div dir="auto">I've compiled OpenSIPS 3.3.1 with your
                  patch and was able to start it but not sure, maybe I
                  was just lucky this time.</div>
                <div dir="auto"><br>
                </div>
                <div dir="auto">What should I do? Thanks!</div>
                <div dir="auto"><br>
                </div>
                <div dir="auto">Best regards,</div>
                <div dir="auto">Yury.</div>
              </div>
              <br>
              <div class="gmail_quote">
                <div dir="ltr" class="gmail_attr">On Tue, 13 Sept 2022,
                  18:56 Bogdan-Andrei Iancu, <<a
                    href="mailto:bogdan@opensips.org" target="_blank"
                    moz-do-not-send="true">bogdan@opensips.org</a>>
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px
                  0px 0.8ex;border-left:1px solid
                  rgb(204,204,204);padding-left:1ex">
                  <div> Hi Yury,<br>
                    <br>
                    it looks like you some multiple issues, overlapping
                    here. The traps you sent here have nothing to do
                    with the auto-scaling, but with a blocking TCP
                    connect for SIP - most of the procs get blocked into
                    a sync TCP connect.<br>
                    <br>
                    Regards,<br>
                    <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
                    <div>On 9/12/22 4:39 PM, Yury Kirsanov wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">Hi Bogdan,
                        <div>I've applied the patch (had to find where
                          to apply it manually for 3.2.8 downloaded from
                          Web page, line 1568 instead of 1652) and
                          restarted the server with only about 300-350
                          SIP devices and immediately got into same
                          issue. I'm attaching two GDB dumps made within
                          several minutes from each other. Autoscale was
                          now OFF, please see my previous message as
                          currently for some reason I'm experiencing
                          lockups even when it's off :(</div>
                      </div>
                    </blockquote>
                    <br>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div>Best regards,</div>
                        <div>Yury.</div>
                      </div>
                      <br>
                      <div class="gmail_quote">
                        <div dir="ltr" class="gmail_attr">On Mon, Sep
                          12, 2022 at 7:48 PM Bogdan-Andrei Iancu <<a
                            href="mailto:bogdan@opensips.org"
                            rel="noreferrer" target="_blank"
                            moz-do-not-send="true">bogdan@opensips.org</a>>
                          wrote:<br>
                        </div>
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
                          0.8ex;border-left:1px solid
                          rgb(204,204,204);padding-left:1ex">
                          <div> <font face="monospace">Hi Yuri,<br>
                              <br>
                              Could you give this patch a try? it should
                              fix the blocking you experience (it should
                              apply on 3.2 too).<br>
                              <br>
                              Best regards,<br>
                            </font>
                            <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
                            <div>On 9/7/22 2:54 PM, Bogdan-Andrei Iancu
                              wrote:<br>
                            </div>
                            <blockquote type="cite"> <font
                                face="monospace">Hi Yury,<br>
                                <br>
                                Thanks for the details info here - let
                                me do a review of some code and run some
                                tests, as at this point I have a good
                                idea on the direction to dig into.<br>
                                <br>
                                I will update here.<br>
                                <br>
                                Best regards,<br>
                              </font>
                              <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
                              <div>On 9/6/22 11:24 AM, Yury Kirsanov
                                wrote:<br>
                              </div>
                              <blockquote type="cite">
                                <div dir="auto">Hi Bogdan,
                                  <div dir="auto">Yes, I'm listening on
                                    all types of sockets including UDP,
                                    TCP and TLS on the outside public
                                    interface and then forward traffic
                                    into internal LAN via UDP only.</div>
                                  <div dir="auto"><br>
                                  </div>
                                  <div dir="auto">Previously it was
                                    getting stuck quite easily, now I
                                    had to wait for a while before this
                                    actually happened. I've routed part
                                    of my customers to this server to
                                    obtain this result so I will have to
                                    do that again.</div>
                                  <div dir="auto"><br>
                                  </div>
                                  <div dir="auto">As soon as I see one
                                    of the processes stuck I'll dot the
                                    trap command and send you all the
                                    details including processes load, ps
                                    output and so on.</div>
                                  <div dir="auto"><br>
                                  </div>
                                  <div dir="auto">For now I had to
                                    switch autoscaling off and just
                                    create many listeners. Do I
                                    understand correctly that I need to
                                    restart OpenSIPS in order to apply
                                    autoscaling profiles and
                                    reload-routes is not sufficient?</div>
                                  <div dir="auto"><br>
                                  </div>
                                  <div dir="auto">Also, do I need
                                    separate UDP profiles for public and
                                    private interfaces? And do I need to
                                    apply autoscaling profile just to a
                                    socket or I need to specify udp or
                                    tcp_workers with autoscaler too?</div>
                                  <div dir="auto"><br>
                                  </div>
                                  <div dir="auto">Thanks and best
                                    regards,</div>
                                  <div dir="auto">Yury.</div>
                                </div>
                                <br>
                                <div class="gmail_quote">
                                  <div dir="ltr" class="gmail_attr">On
                                    Tue, 6 Sept 2022, 18:18
                                    Bogdan-Andrei Iancu, <<a
                                      href="mailto:bogdan@opensips.org"
                                      rel="noreferrer" target="_blank"
                                      moz-do-not-send="true">bogdan@opensips.org</a>>
                                    wrote:<br>
                                  </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0px 0px 0px
                                    0.8ex;border-left:1px solid
                                    rgb(204,204,204);padding-left:1ex">
                                    <div> <font face="monospace">Hi
                                        Yury,<br>
                                        <br>
                                        Thanks for the info. I see that
                                        the stuck process (24) is an
                                        auto-scalled one (based on its
                                        id). Do you have SIP traffic
                                        from UDP to TCP or doing some
                                        HEP capturing for SIP ? I saw a
                                        recent similar report where a
                                        UDP auto-scalled worked got
                                        stuck when trying to do some
                                        communication with the TCP
                                        main/manager process (in order
                                        to handle a TCP operation).<br>
                                        <br>
                                        BTW, any chance to do a
                                        "opensips-cli -x trap" when you
                                        have that stuck process, just to
                                        see where is it stuck? and is it
                                        hard to reproduce? as I may ask
                                        you to extract some information
                                        from the running process....<br>
                                        <br>
                                        Regards,<br>
                                      </font>
                                      <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" rel="noreferrer noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" rel="noreferrer noreferrer" target="_blank" moz-do-not-send="true">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
                                      <div>On 9/3/22 6:54 PM, Yury
                                        Kirsanov wrote:<br>
                                      </div>
                                    </div>
                                  </blockquote>
                                </div>
                              </blockquote>
                              <br>
                              <br>
                              <fieldset></fieldset>
                              <pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" rel="noreferrer" target="_blank" moz-do-not-send="true">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
                            </blockquote>
                            <br>
                          </div>
                        </blockquote>
                      </div>
                    </blockquote>
                    <br>
                  </div>
                </blockquote>
              </div>
            </blockquote>
            <br>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <br>
  </body>
</html>