<div dir="auto">Hi Bogdan,<div dir="auto">Thanks for this update, but it looks like I can't check autoscaler because of this first issue with blocking TCP connect. Is there a way to resolve it? Am I doing something wrong? Or is that something to do with OpenSIPS code? As yes, you're right, as soon as I restart OpenSIPS having a lot of SIP devices trying to connect to it - it goes crazy, starts to consume memory and stops to forward packets sitting there at 100% load until it runs out of memory and segfaults. Sometimes I can't even restart it to come to normal state to make it work, it just loops into same crash whatever I try to do.</div><div dir="auto"><br></div><div dir="auto">I've compiled OpenSIPS 3.3.1 with your patch and was able to start it but not sure, maybe I was just lucky this time.</div><div dir="auto"><br></div><div dir="auto">What should I do? Thanks!</div><div dir="auto"><br></div><div dir="auto">Best regards,</div><div dir="auto">Yury.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 13 Sept 2022, 18:56 Bogdan-Andrei Iancu, <<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div>
    Hi Yury,<br>
    <br>
    it looks like you some multiple issues, overlapping here. The traps
    you sent here have nothing to do with the auto-scaling, but with a
    blocking TCP connect for SIP - most of the procs get blocked into a
    sync TCP connect.<br>
    <br>
    Regards,<br>
    <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" target="_blank" rel="noreferrer">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" target="_blank" rel="noreferrer">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
    <div>On 9/12/22 4:39 PM, Yury Kirsanov
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">Hi Bogdan,
        <div>I've applied the patch (had to find where to apply it
          manually for 3.2.8 downloaded from Web page, line 1568 instead
          of 1652) and restarted the server with only about 300-350 SIP
          devices and immediately got into same issue. I'm attaching two
          GDB dumps made within several minutes from each other.
          Autoscale was now OFF, please see my previous message as
          currently for some reason I'm experiencing lockups even when
          it's off :(</div>
      </div>
    </blockquote>
    <br>
    <blockquote type="cite">
      <div dir="ltr">
        <div>Best regards,</div>
        <div>Yury.</div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Mon, Sep 12, 2022 at 7:48
          PM Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org" target="_blank" rel="noreferrer">bogdan@opensips.org</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div> <font face="monospace">Hi Yuri,<br>
              <br>
              Could you give this patch a try? it should fix the
              blocking you experience (it should apply on 3.2 too).<br>
              <br>
              Best regards,<br>
            </font>
            <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" target="_blank" rel="noreferrer">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" target="_blank" rel="noreferrer">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
            <div>On 9/7/22 2:54 PM, Bogdan-Andrei Iancu wrote:<br>
            </div>
            <blockquote type="cite"> <font face="monospace">Hi Yury,<br>
                <br>
                Thanks for the details info here - let me do a review of
                some code and run some tests, as at this point I have a
                good idea on the direction to dig into.<br>
                <br>
                I will update here.<br>
                <br>
                Best regards,<br>
              </font>
              <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" target="_blank" rel="noreferrer">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" target="_blank" rel="noreferrer">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
              <div>On 9/6/22 11:24 AM, Yury Kirsanov wrote:<br>
              </div>
              <blockquote type="cite">
                <div dir="auto">Hi Bogdan,
                  <div dir="auto">Yes, I'm listening on all types of
                    sockets including UDP, TCP and TLS on the outside
                    public interface and then forward traffic into
                    internal LAN via UDP only.</div>
                  <div dir="auto"><br>
                  </div>
                  <div dir="auto">Previously it was getting stuck quite
                    easily, now I had to wait for a while before this
                    actually happened. I've routed part of my customers
                    to this server to obtain this result so I will have
                    to do that again.</div>
                  <div dir="auto"><br>
                  </div>
                  <div dir="auto">As soon as I see one of the processes
                    stuck I'll dot the trap command and send you all the
                    details including processes load, ps output and so
                    on.</div>
                  <div dir="auto"><br>
                  </div>
                  <div dir="auto">For now I had to switch autoscaling
                    off and just create many listeners. Do I understand
                    correctly that I need to restart OpenSIPS in order
                    to apply autoscaling profiles and reload-routes is
                    not sufficient?</div>
                  <div dir="auto"><br>
                  </div>
                  <div dir="auto">Also, do I need separate UDP profiles
                    for public and private interfaces? And do I need to
                    apply autoscaling profile just to a socket or I need
                    to specify udp or tcp_workers with autoscaler too?</div>
                  <div dir="auto"><br>
                  </div>
                  <div dir="auto">Thanks and best regards,</div>
                  <div dir="auto">Yury.</div>
                </div>
                <br>
                <div class="gmail_quote">
                  <div dir="ltr" class="gmail_attr">On Tue, 6 Sept 2022,
                    18:18 Bogdan-Andrei Iancu, <<a href="mailto:bogdan@opensips.org" target="_blank" rel="noreferrer">bogdan@opensips.org</a>>
                    wrote:<br>
                  </div>
                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div> <font face="monospace">Hi Yury,<br>
                        <br>
                        Thanks for the info. I see that the stuck
                        process (24) is an auto-scalled one (based on
                        its id). Do you have SIP traffic from UDP to TCP
                        or doing some HEP capturing for SIP ? I saw a
                        recent similar report where a UDP auto-scalled
                        worked got stuck when trying to do some
                        communication with the TCP main/manager process
                        (in order to handle a TCP operation).<br>
                        <br>
                        BTW, any chance to do a "opensips-cli -x trap"
                        when you have that stuck process, just to see
                        where is it stuck? and is it hard to reproduce?
                        as I may ask you to extract some information
                        from the running process....<br>
                        <br>
                        Regards,<br>
                      </font>
                      <pre cols="72">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a href="https://www.opensips-solutions.com" rel="noreferrer noreferrer" target="_blank">https://www.opensips-solutions.com</a>
OpenSIPS Summit 27-30 Sept 2022, Athens
  <a href="https://www.opensips.org/events/Summit-2022Athens/" rel="noreferrer noreferrer" target="_blank">https://www.opensips.org/events/Summit-2022Athens/</a></pre>
                      <div>On 9/3/22 6:54 PM, Yury Kirsanov wrote:<br>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </blockquote>
              <br>
              <br>
              <fieldset></fieldset>
              <pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank" rel="noreferrer">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank" rel="noreferrer">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
            </blockquote>
            <br>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <br>
  </div>

</blockquote></div>