<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: courier new,courier,monaco,monospace,sans-serif; font-size: 12pt; color: #000000'><font face="courier new, courier, monaco, monospace, sans-serif">Hello Bogdan,</font><div style="font-family: "courier new", courier, monaco, monospace, sans-serif;"><span style="font-family: monospace;">Upscaling error is didn't show up any more in logs at least I didn't so it. Might need trigger bigger load.</span></div><div><font face="monospace"><br></font></div><div><span style="font-family: monospace;"><b>do you do auto-scaling only for TCP ?</b></span></div><div><font face="monospace"><br></font></div><div><font face="monospace">Our  setup in most  multi homed, as example public side TCP private side UDP</font></div><div><font face="monospace"><br></font></div><div><font face="monospace"><br></font></div><div><span style="font-family: monospace;"><b>Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ?</b></span></div><div><font face="courier new, courier, monaco, monospace, sans-serif"><br>Yes, that how is look like end result that opensips start blocking TCP connections. If specific process  handle sip registration via tcp and opensips trying close to down scale end point need send re register in order do not loose registration. Other wise it cause stale connections. Just close properly connection not enough I think.</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif"><br></font></div><div><font face="courier new, courier, monaco, monospace, sans-serif"><br></font></div><div><font face="courier new, courier, monaco, monospace, sans-serif">volga629</font></div><div><br></div><div><br></div><div><br></div><div><br><hr id="zwchr" style="font-family: "courier new", courier, monaco, monospace, sans-serif;"><div style="font-family: Helvetica, Arial, sans-serif; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-size: 12pt;"><b>From: </b>"Bogdan-Andrei Iancu" <bogdan@opensips.org><br><b>To: </b>"OpenSIPS users mailling list" <users@lists.opensips.org>, "Slava Bendersky" <volga629@networklab.ca><br><b>Sent: </b>Thursday, March 14, 2019 1:45:20 PM<br><b>Subject: </b>Re: [OpenSIPS-Users] auto scaling<br><br>
  
    
  
  
    <tt>Slava,<br>
      <br>
      What about the upscaling error you originally reported ? is there
      any way to reproduce it ? do you do auto-scaling only for TCP (no
      UDP,timer) ?<br>
      <br>
      Now, about the TCP downscaling - I guess you see as terminated by
      OpenSIPS the TCP conns anchored in the process that is terminated
      ? Is is a proper close of the connection ? or ?<br>
      <br>
      Regards,<br>
    </tt>
    <pre class="moz-signature">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
OpenSIPS Summit 2019
  <a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2019Amsterdam/" target="_blank">https://www.opensips.org/events/Summit-2019Amsterdam/</a>
</pre>
    <div class="moz-cite-prefix">On 03/14/2019 02:14 PM, Slava Bendersky
      wrote:<br>
    </div>
    <blockquote cite="mid:902388553.152.1552565672232.JavaMail.bendersky@nlws01.networklab.lan">
      <style>p { margin: 0; }</style>
      <div style="font-family: courier
        new,courier,monaco,monospace,sans-serif; font-size: 12pt; color:
        #000000">Hello Bogdan,
        <div>After we doing more testing with auto scaling we observed
          issue with TCP, where opensips try scale down the processes it
          tear down them and break TCP connections with end points and
          that cause blocked connection state where opensips stops
          process TCP connections.</div>
        <div><br>
        </div>
        <div>volga629 <br>
          <br><hr id="zwchr">
          <div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From:
            </b>"Slava Bendersky" <a class="moz-txt-link-rfc2396E" href="mailto:volga629@networklab.ca" target="_blank"><volga629@networklab.ca></a><br>
            <b>To: </b>"Bogdan-Andrei Iancu"
            <a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org" target="_blank"><bogdan@opensips.org></a><br>
            <b>Cc: </b>"OpenSIPS users mailling list"
            <a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org" target="_blank"><users@lists.opensips.org></a><br>
            <b>Sent: </b>Wednesday, March 13, 2019 12:21:14 AM<br>
            <b>Subject: </b>Re: [OpenSIPS-Users] auto scaling<br>
            <br>
            <style>p { margin: 0; }</style>
            <div style="font-family: courier
              new,courier,monaco,monospace,sans-serif; font-size: 12pt;
              color: #000000">I will do pull today.
              <div><br>
              </div>
              <div>volga629<br>
                <hr id="zwchr">
                <div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From:
                  </b>"Bogdan-Andrei Iancu" <a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org" target="_blank"><bogdan@opensips.org></a><br>
                  <b>To: </b>"Slava Bendersky"
                  <a class="moz-txt-link-rfc2396E" href="mailto:volga629@networklab.ca" target="_blank"><volga629@networklab.ca></a><br>
                  <b>Cc: </b>"OpenSIPS users mailling list"
                  <a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org" target="_blank"><users@lists.opensips.org></a><br>
                  <b>Sent: </b>Tuesday, March 12, 2019 4:52:06 AM<br>
                  <b>Subject: </b>Re: [OpenSIPS-Users] auto scaling<br>
                  <br>
                  <tt>Slava,<br>
                    <br>
                    Have you pulled my change on the improved log (just
                    update from master) ? and post here the new output
                    you get for the error.<br>
                    <br>
                    Thanks,<br>
                  </tt>
                  <pre class="moz-signature">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
OpenSIPS Summit 2019
  <a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2019Amsterdam/" target="_blank">https://www.opensips.org/events/Summit-2019Amsterdam/</a>
</pre>
                  <div class="moz-cite-prefix">On 03/12/2019 05:51 AM,
                    Slava Bendersky wrote:<br>
                  </div>
                  <blockquote cite="mid:423067196.274.1552362703793.JavaMail.bendersky@nlws01.networklab.lan">
                    <style>p { margin: 0; }</style>
                    <div style="font-family: courier
                      new,courier,monaco,monospace,sans-serif;
                      font-size: 12pt; color: #000000">Hello Bogdan,
                      <div>My question how to calculate correctly </div>
                      <div><br>
                      </div>
                      <div><b>tcp_workers</b></div>
                      <div><b>udp_workers</b></div>
                      <div><b><br>
                        </b></div>
                      <div>to properly assign with in  auto scaling
                        profile. Which parameters need use as base ?</div>
                      <div><br>
                      </div>
                      <div>As example if vm 4GB memory up what will be
                        max number for scale up processes and what
                        should be value for tcp and udp workers.</div>
                      <div>It message is appear in log on first attempt
                        to fork in scale up.</div>
                      <div><br>
                      </div>
                      <div>volga629 </div>
                      <div>   </div>
                      <div>
                        <hr id="zwchr">
                        <div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From:
                          </b>"Bogdan-Andrei Iancu" <a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org" target="_blank"><bogdan@opensips.org></a><br>
                          <b>To: </b>"OpenSIPS users mailling list" <a class="moz-txt-link-rfc2396E" href="mailto:users@lists.opensips.org" target="_blank"><users@lists.opensips.org></a>,
                          "Slava Bendersky" <a class="moz-txt-link-rfc2396E" href="mailto:volga629@networklab.ca" target="_blank"><volga629@networklab.ca></a><br>
                          <b>Sent: </b>Monday, March 11, 2019 4:44:57
                          AM<br>
                          <b>Subject: </b>Re: [OpenSIPS-Users] auto
                          scaling<br>
                          <br>
                          <tt>Hi Slava,<br>
                            <br>
                            Based on the OpenSIPS logs, did you get that
                            error on the first forking attempt or
                            sometime later after some number of new tcp
                            processes forking ? More or less, my
                            question is : what was the process
                            creation/termination history before getting
                            that bug log ?<br>
                            <br>
                            Also I just pushed a small commit to improve
                            the date provided by the bug log.<br>
                            <br>
                            In regards to the "not clear how to
                            calculate"...could you be specific ?<br>
                            <br>
                            Best regards,<br>
                          </tt>
                          <pre class="moz-signature">Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  <a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com" target="_blank">https://www.opensips-solutions.com</a>
OpenSIPS Summit 2019
  <a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2019Amsterdam/" target="_blank">https://www.opensips.org/events/Summit-2019Amsterdam/</a>
</pre>
                          <div class="moz-cite-prefix">On 03/10/2019
                            05:12 PM, Slava Bendersky wrote:<br>
                          </div>
                          <blockquote cite="mid:1387008888.101.1552230768174.JavaMail.bendersky@nlws01.networklab.lan">
                            <style>p { margin: 0; }</style>
                            <div style="font-family: courier
                              new,courier,monaco,monospace,sans-serif;
                              font-size: 12pt; color: #000000"><font face="courier new, courier, monaco,
                                monospace, sans-serif"><span style="font-size: 12pt;">Hello
                                  Everyone,</span></font>
                              <div style="color: rgb(0, 0, 0);
                                font-family: "courier new",
                                courier, monaco, monospace, sans-serif;
                                font-size: 12pt;">Trying test auto
                                scaling feature with following config</div>
                              <div style="color: rgb(0, 0, 0);
                                font-family: "courier new",
                                courier, monaco, monospace, sans-serif;
                                font-size: 12pt;"><br>
                              </div>
                              <div>
                                <div><font face="courier new, courier,
                                    monaco, monospace, sans-serif">auto_scaling_profile
                                    = PRF_SIP</font></div>
                                <div><font face="courier new, courier,
                                    monaco, monospace, sans-serif">   
                                     scale up to 24 on 70% for 4 cycles
                                    within 5</font></div>
                                <div><font face="courier new, courier,
                                    monaco, monospace, sans-serif">   
                                     scale down to 4 on 20% for 8 cycles</font></div>
                              </div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif">On box
                                  4096 Mb memory and 4 virtual CPU</font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif">Getting
                                  in log </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif">CRITICAL:core:fork_dynamic_tcp_process:
                                  #012>>> trying to fork one
                                  more TCP worker but no free slots in
                                  the TCP table#012#012It seems you have
                                  hit a programming bug.#012Please help
                                  us make OpenSIPS better by reporting
                                  it at <a class="moz-txt-link-freetext" href="https://github.com/OpenSIPS/opensips/issues" target="_blank">https://github.com/OpenSIPS/opensips/issues</a></font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><b>Not
                                    clear how to calculate workers for
                                    tcp or udp</b></font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><b>tcp_workers=10</b>
                                  use_auto_scaling_profile PRF_SIP    </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><b>listen=udp:lan_ip:5060</b>
                                  use_auto_scaling_profile PRF_SIP</font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif"><br>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif">
                                  <div>[root@vprx00 ~]# opensips -V</div>
                                  <div>version: opensips 3.0.0-dev
                                    (x86_64/linux)</div>
                                  <div>flags: STATS: On,
                                    SHM_EXTRA_STATS, DISABLE_NAGLE,
                                    USE_MCAST, SHM_MMAP, PKG_MALLOC,
                                    QM_MALLOC, DBG_MALLOC,
                                    FAST_LOCK-ADAPTIVE_WAIT</div>
                                  <div>ADAPTIVE_WAIT_LOOPS=1024,
                                    MAX_RECV_BUFFER_SIZE 262144,
                                    MAX_LISTEN 16, MAX_URI_SIZE 1024,
                                    BUF_SIZE 65535</div>
                                  <div>poll method support: poll, epoll,
                                    sigio_rt, select.</div>
                                  <div>main.c compiled on 10:06:47 Mar
                                     7 2019 with gcc 8</div>
                                  <div><br>
                                  </div>
                                  <div><br>
                                  </div>
                                </font></div>
                              <div><font face="courier new, courier,
                                  monaco, monospace, sans-serif">volga629</font></div>
                            </div>
                            <br>
                            <fieldset class="mimeAttachmentHeader"></fieldset>
                            <br>
                            <pre>_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
                          </blockquote>
                          <br>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
            <br>
            _______________________________________________<br>
            Users mailing list<br>
            <a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
            <a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
          </div>
          <br>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre>_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
    </blockquote>
    <br>
  

</div><br></div></div></body></html>