<div dir="ltr"><div>Hello Liviu!</div><div>And first of all thank you for your detailed explanation.</div><div><br></div><div>Now I completely understand the approach was taken when developing this feature of Clusterer.</div><div>And this looks logical to me. It gives less chances to make a human error.</div><div><br></div><div>What I did for now, is a clustering super-structure (that works apart OpenSIPS),</div><div>and here what happens when both nodes see each other again (when recovering from networking issues):</div><div><br></div><div>Shared IP remains working on the Master side, and one-shot systemd service activates <br></div><div>"clusterer_shtag_set_active" on the Master right away.<br>Thus suppressing a Stand-by node to apply backup state.</div><div><br></div><div>For now this schema works perfectly.</div><div>Might be I will come up with more robust solution later.. In case this happens, I will share my experience.</div><div><br></div><div>Have a nice day!<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Apr 11, 2020 at 2:13 PM Liviu Chircu <<a href="mailto:liviu@opensips.org">liviu@opensips.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 09.04.2020 15:03, Donat Zenichev wrote:<br>
> I have question and it's almost theoretic.<br>
> The question relates to Clusterer module and its behavior.<br>
><br>
> Will Clusterer module solve this contradiction on its own?<br>
> And if so, to which side the precedence is given?<br>
><br>
> The other way around could be to manually re-activate all services,<br>
> when all the cluster resumes into normal working process (all nodes <br>
> are present).<br>
> Thus this gives us a warranty that shared tag will only be activated <br>
> on one of the sides.<br>
<br>
Hi, Donat!<br>
<br>
A very good question and one that we had to answer ourselves when we <br>
came up with the current design. To begin with, in your scenario, for <br>
all OpenSIPS 2.4+ clusterer versions, after the link between the nodes <br>
comes back online, you will have the following:<br>
<br>
* node A: ACTIVE state (holds the VIP), sharing tag state: ACTIVE (1)<br>
* node B: BACKUP state, sharing tag state: ACTIVE (1)<br>
<br>
The main reason behind this inconsistent state is that we did not <br>
provide an MI command to force a sharing tag to BACKUP (0), which could <br>
be triggered on node B's transition from ACTIVE -> BACKUP once the link <br>
is restored, so recovering from this state will not work automatically - <br>
you have to provide handling for this scenario as well (see last paragraph).<br>
<br>
Reasoning behind this design<br>
----------------------------<br>
<br>
Ultimately, our priority was not to get into solving consensus problems, <br>
Paxos algorithms, etc. What we wanted is a robust active/backup <br>
solution which you could flip back and forth with ease, thus achieving <br>
both High-Availability and easy maintenance. By not providing a <br>
"set_sharing_tag vip 0" command, we _avoid_ the situation where, due to <br>
a developer error, both tags end up being BACKUP (0)!! In such a <br>
scenario: there will be no more CDRs and you will be able to run <br>
infinite CPS/CC through that instance, since all call profile counters <br>
are equal to 0. None of the instances take responsibility for any call <br>
running through them, so a lot of data will be lost.<br>
<br>
On the flip side, in a scenario where both tags are bugged in the ACTIVE <br>
(1) state, you would have: duplicated CDRs (along with maybe some DB <br>
error logs due to conflicting unique keys) and possibly extra-counted <br>
calls, leading to a reduction of the maximally supported CC/CPS. Assume <br>
that the platform wasn't even at 50% of the max limits to begin with, <br>
and the latter has 0 impact on the live system. Thinking about this, <br>
this didn't sound that bad at all to us: no data loss, at the expense of <br>
a few error logs and possibly some added call limits.<br>
<br>
So you can see that we went for a design which minimizes any errors that <br>
the developers can make, and protects the overall system. The platform <br>
will work decently, regardless of network conditions or how the <br>
tag-managing MI commands are sent or abused.<br>
<br>
How to automatically recover from the ACTIVE/ACTIVE sharing tag state<br>
---------------------------------------------------------------------<br>
<br>
Given that the "clusterer_shtag_set_active" [1] MI command issued to a <br>
node will force all other nodes to transition from ACTIVE -> BACKUP, you <br>
could enhance your system with a logic that sends this command to the <br>
opposite node any time a node's VIP performs the ACTIVE -> BACKUP <br>
transition. This should fix the original problem, where both tags end <br>
up in the ACTIVE state, due to the link between nodes being temporarily <br>
down, without any of the OS'es necessarily being down.<br>
<br>
PS: we haven't implemented the above ^ ourselves yet, but it should work <br>
in theory :) let me know if it works for you if you do decide to plug <br>
this rare issue for your setup!<br>
<br>
Best regards,<br>
<br>
[1]: <br>
<a href="https://opensips.org/docs/modules/3.1.x/clusterer#mi_clusterer_shtag_set_active" rel="noreferrer" target="_blank">https://opensips.org/docs/modules/3.1.x/clusterer#mi_clusterer_shtag_set_active</a><br>
<br>
-- <br>
Liviu Chircu<br>
<a href="http://www.twitter.com/liviuchircu" rel="noreferrer" target="_blank">www.twitter.com/liviuchircu</a> | <a href="http://www.opensips-solutions.com" rel="noreferrer" target="_blank">www.opensips-solutions.com</a><br>
<br>
OpenSIPS Summit, Amsterdam, May 2020<br>
<a href="http://www.opensips.org/events" rel="noreferrer" target="_blank">www.opensips.org/events</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr"><font style="background-color:rgb(255,255,255)" color="#0b5394">Best regards,<br></font></div><div dir="ltr"><font style="background-color:rgb(255,255,255)" color="#0b5394">Donat Zenichev<br><br></font></div></div></div></div></div></div></div></div>