<div dir="ltr">Thanks @liviuchircu - sadly GitHub isn't letting me comment at the moment (bug) so coming back here.<br><br>Checking my active masters I am indeed seeing: `INFO:usrloc:replicate_ucontact_update: All destinations in cluster: 25 are down or probing`<br><br>Browsing the repo I can see that patch 40b7422 is not included in any tagged releases at this time so I will amend the `ping_timeout` as suggested.<br><br>Thanks for the invaluable insight.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 19 Feb 2020 at 14:29, Liviu Chircu <<a href="mailto:liviu@opensips.org">liviu@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div><tt>Hi, Callum!</tt><br>
</div>
<div><br>
</div>
<div>On 18.02.2020 15:56, Callum Guy wrote:<br>
</div>
<blockquote type="cite">
<div><br>
</div>
<div><font face="monospace"><a>INFO:usrloc:receive_ucontact_insert</a>:
failed to fetch local urecord - creating new one (ci: '<a href="mailto:0_751733367@10.0.0.13" target="_blank">0_751733367@10.0.0.13</a>')
<br>
<a>INFO:usrloc:receive_ucontact_update</a>: failed to fetch local
urecord - create new record and contact (ci: '<a href="mailto:0_3576574827@10.0.0.26" target="_blank">0_3576574827@10.0.0.26</a>')<br>
<a>INFO:usrloc:receive_ucontact_update</a>: failed to fetch local
urecord - create new record and contact (ci: '<a href="mailto:0_4169036315@10.0.0.12" target="_blank">0_4169036315@10.0.0.12</a>')<br>
</font></div>
<div><br>
</div>
<div>Is this a simple matter of sessions timing out on the backup
and being removed before the next registration on the primary
instance?</div>
</blockquote>
<tt>Exactly. The UA re-REGISTERs too close to the expiration point,
creating a race</tt><tt> condition<br>
between the two nodes. By the time the backup processes the
contact</tt><tt> refresh packet, its<br>
contact is long gone, so it has nothing to match it against.
Hence this harmless INFO message.</tt><br>
<blockquote type="cite">
<div><br>
</div>
<div>Secondly I am graphing registration counts based on the
following CLI request:</div>
<div><br>
</div>
<div><font face="monospace">opensips-cli -x mi ul_dump brief=1 |
grep AOR | wc -l</font><br>
</div>
<div><br>
</div>
<div>Several times a day I see a small drop in the backup
registrations (lighter red line) before a resync operation as
shown here:</div>
<div><br>
</div>
<div><br>
</div>
<div>Can anyone help to explain what's going on here? There aren't
any log messages at this time however I do see the backup node
reporting a primary node ping loss ~1 hour before the dip:</div>
<div><br>
</div>
<div><font face="monospace">2020-02-18T11:38:37.786021+00:00
opensips[91789]: <a>INFO:clusterer:do_action_trans_2</a>: Ping reply
not received, node [13] is down<br>
2020-02-18T11:38:38.795838+00:00 opensips[91804]:
<a>INFO:clusterer:handle_internal_msg</a>: Node [13] is UP<br>
</font></div>
</blockquote>
<font face="monospace">This is 100% identical to what Alexey
reported in #1976 [1]. Please try to incorporate at least one of<br>
my proposed solutions -- this should make it much more harder for
the link between the nodes to do down.<br>
<br>
</font>
<p><font face="monospace">Best regards,</font></p>
<p><font face="monospace">[1]: </font><font face="monospace"><a href="https://github.com/OpenSIPS/opensips/issues/1976" target="_blank">https://github.com/OpenSIPS/opensips/issues/1976</a></font></p>
<pre cols="72">--
Liviu Chircu
<a href="http://www.twitter.com/liviuchircu" target="_blank">www.twitter.com/liviuchircu</a> | <a href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a>
OpenSIPS Summit, Amsterdam, May 2020
<a href="http://www.opensips.org/events" target="_blank">www.opensips.org/events</a>
OpenSIPS Bootcamp, Miami, March 2020
<a href="http://www.opensips.org/training" target="_blank">www.opensips.org/training</a></pre>
</div>
</blockquote></div>
<br>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-align:justify"><font size="3" face="Verdana"><span style="font-size:8px;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"></span></font></p><div><img src="https://www.x-on.co.uk/email/footer/General-Practice-Awards-winners.jpg"></div><div><br></div><div><div><div><font size="4"><b><sup><font face="Verdana">0333 332 0000 | <a href="http://www.x-on.co.uk" target="_blank">www.x-on.co.uk</a> | <sub> </sub></font></sup></b></font><font size="4"><b><sub><sup><font face="Verdana"><a href="https://www.linkedin.com/company/x-on" target="_blank"><img src="http://www.x-on.co.uk//images/icon/linkedin.png" width="24" height="24"></a> <a href="https://www.facebook.com/XonTel" target="_blank"><img src="http://www.x-on.co.uk//images/icon/facebook.png" width="24" height="24"></a> <a href="https://twitter.com/xonuk" target="_blank"><img src="http://www.x-on.co.uk//images/icon/twitter.png" width="24" height="24"></a></font></sup></sub> </b></font><br><p><span style="font-size:6.0pt;font-family:Verdana;color:black">X-on
is a trading name of Storacall Technology Ltd a limited company registered in
England and Wales.<br>
Registered Office : Avaland House, 110 London Road, Apsley, Hemel Hempstead,
Herts, HP3 9SD. Company Registration No. 2578478.<br>
The information in this e-mail is confidential and for use by the addressee(s)
only. If you are not the intended recipient, please notify X-on immediately on <span>+44(0)333 332 0000</span> and delete the<br>message from your computer. If you are not a named addressee you must not use,
disclose, disseminate, distribute, copy, print or reply to this email. </span><span style="font-size:6.0pt;font-family:Verdana;color:black">Views
or opinions expressed by an individual<br>within this email may not necessarily
reflect the views of X-on or its associated companies. Although X-on routinely
screens for viruses, addressees should scan this email and any attachments<br>for
viruses. X-on makes no representation or warranty as to the absence of viruses
in this email or any attachments.</span></p>
<p><span style="font-size:6.0pt;font-family:Verdana;color:black"></span><font size="2"><span style="font-size:6.0pt;font-family:Verdana;color:black"></span></font></p></div></div></div>