<div dir="ltr">Hi,<div><br></div><div>Thanks all for your help.</div><div><br></div><div>I have removed the offending line and we are not seeing that behaviour.</div><div><br></div><div>Unfortunately, the two dispatcher endpoints are not managed by us. It would certainly be nice to have an indication that the errors response it from the far end. Likewise, how could the same be achieved in opensips when we respond back to our platform?</div><div><br></div><div>Thank you.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 8 Jun 2020 at 13:46, Ben Newlin <<a href="mailto:Ben.Newlin@genesys.com">Ben.Newlin@genesys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-1582059780582198670WordSection1">
<p class="MsoNormal">Solarmon,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Yes, that code is your issue. While it makes sense to mark the remote destination unreachable on some error responses possibly, or certainly on lack of response, this code is marking the destination unreachable on receipt of any 5xx or
6xx response (or timeout), without regard to whether the response came directly from that node or from further upstream. This does not seem desirable.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The code is also performing failover using the dispatcher for any of these codes, which also does not account for the possibility that the error came from further upstream and trying another dispatcher node may not be warranted or helpful.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">If you have control over the nodes that are the dispatcher targets, what we have done is to add a customer header to any error replies sent directly from that node. This allows the receiving node to know that the issue was local to the
remote node and not relayed from further upstream and that ds_mark_dst and ds_next_dst are appropriate. We also still check for timeout too, of course. But there are other ways to do this as well.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="color:black">Ben Newlin </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Users <<a href="mailto:users-bounces@lists.opensips.org" target="_blank">users-bounces@lists.opensips.org</a>> on behalf of solarmon <<a href="mailto:solarmon@one-n.co.uk" target="_blank">solarmon@one-n.co.uk</a>><br>
<b>Reply-To: </b>OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a>><br>
<b>Date: </b>Monday, June 8, 2020 at 6:57 AM<br>
<b>To: </b>OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a>><br>
<b>Subject: </b>Re: [OpenSIPS-Users] 502 Bad Gateway events leads to calls being rejected with 480 Temporarily Unavailable<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">Hi Diptesh ,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for your reply.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Apologies, I'm using the term 'blacklist' to generally mean that the endpoints are not available.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Also, the 502 Bad Gateway is response to an INVITE, not SIP OPTIONS, returned by the far end and the ITSP is just passing that back to us, because the call has failed. For such call failures, I'm not expecting for the dispatcher endpoints
to be marked as unavailable for routing.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I am not using, or have not set, the '<b>ds_define_blacklist (str)</b>' option in my dispatcher module config.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">My probing mode is:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">modparam("dispatcher", "ds_probing_mode", 1)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm not seeing anything in the logs regarding the dispatcher nodes going into Probing mode - should there be logs for that, or can it be enabled to be logged?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">When I check the endpoints with ' opensipsctl dispatcher dump' they always seem to be '<b>Active</b>' - so it is either they are like that, or they may have only been in '<b>Probing</b>' mode very briefly. Again, I was hoping to see mode/state
change in the historical logs. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In my <b>opensips.cfg</b> (which was created for me) I can see the following code, which looks like this is where it is introducing this behaviour in question:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<blockquote style="margin-left:30pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">failure_route[call_failover]</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">{</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> xlog("[$ci] call failed to established with $T_reply_code code\n");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> rtpproxy_unforce("$avp(rtpp_set)");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> if (t_was_cancelled()) {</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> t_reply("487","Request cancelled");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> exit;</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> # any failure indication ?</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> if ( t_check_status("[56][0-9][0-9]")</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> || (t_check_status("408") && t_local_replied("all"))</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> ) {</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> xlog("[$ci] destination $rd failed with $T_reply_code -> retry\n ");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> <b><span style="color:red;background:yellow"> ds_mark_dst("p");</span></b></span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> if ( ds_next_domain() ) {</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> xlog("[$ci] using new destination <$rd>\n ");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> # send it out again</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> t_on_failure("call_failover");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> t_relay();</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> exit;</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> } else {</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> xlog("[$ci] no other destination to retry\n ");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> t_reply("503","Service not available");</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> exit;</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> # if call failure, allow the reply to propagate to caller</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> exit;</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">}</span><u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thank you for the tip about the 'modparam("dispatcher", "options_reply_codes", "502")' option. I will try that if it is not recommend to change the above code.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thank you.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Mon, 8 Jun 2020 at 10:47, Diptesh Patel <<a href="mailto:diptesh.patel@ecosmob.com" target="_blank">diptesh.patel@ecosmob.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hello Solarmon,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Need some clarification on term blacklisting, Are you using the blacklist for Probing Mode of destination? or Are you using the
<b>'ds_define_blacklist (str)' </b>parameter. If you are not using the blacklist parameter then below information help you. It is great if you share your script snippet and output of
<b>'opensipsctl dispatcher dump'</b> which shows you the current status of your destinations.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">If you are getting success(200 OK) response on OPTIONS then it is not possible that you got a negative response from a destination and it will not be blacklisted(probing mode) in dispatcher until you blacklist(probing mode) from the script
using 'ds_mark_dst()' exported function. I doubt that you are also getting '502 Bad Gateway' on OPTIONS which is sending to the destination to check the availability. If It is right and you want to add the 502 response as a good response for OPTIONS. you
can add the 502 as <b>'modparam("dispatcher", "options_reply_codes", "502")'.</b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Thanks & Regards<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><b>Diptesh Patel</b><u></u><u></u></p>
<div>
<p class="MsoNormal">Software Developer <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Ecosmob Technologies Ltd, <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Ahmedabad<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mo:<b>+919898962659</b><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Mon, Jun 8, 2020 at 2:24 PM solarmon <<a href="mailto:solarmon@one-n.co.uk" target="_blank">solarmon@one-n.co.uk</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm trying to understand whether this is the correct or expected behaviour.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We have two destinations configured in Dispatcher.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">What I am noticing is that when we receive 502 Bad Gateway messages (logged as <span style="font-size:10.5pt;font-family:Roboto;color:rgb(32,33,36);background:rgb(248,249,250)">("call failed to established with 502 code")
</span>from both endpoints. After both endpoints have returned 502 Bad Gateway, opensips pass back 503 Service Unavailable back to the originating endpoint of the call. However, subsequent calls are being immediately rejected with 480 Temporarily Unavailable
(logged as <span style="font-size:10.5pt;font-family:Roboto;color:rgb(32,33,36);background:rgb(248,249,250)">"failed to find an available destination, rejecting") for a period of time.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It seems that opensips is blacklisting the Dispatcher endpoints because of receiving the 502 Bad Gateway messages. Is this the correct/expected behaviour? I would have thought the blacklisting should be based on the SIP OPTIONS sent to
the Dispatcher endpoints.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I do not currently see any issues with SIP OPTIONS to these endpoints so I'm confused as to why they are seemingly blacklisted.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If this is the correct/expected behaviour, can it be changed to only blacklist based on the SIP OPTIONs pings?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thank you.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:gray;background:white">Disclaimer</span></b><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;font-family:Arial,sans-serif;color:gray;background:white">In addition to generic Disclaimer which you have agreed on our website, any views or opinions presented in this email are solely those of the originator
and do not necessarily represent those of the Company or its sister concerns. Any liability (in negligence, contract or otherwise) arising from any third party taking any action, or refraining from taking any action on the basis of any of the information contained
in this email is hereby excluded.</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:gray;background:white">Confidentiality</span></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Arial,sans-serif;color:gray;background:white">This communication (including any attachment/s) is intended only for the use of the addressee(s) and contains information that is PRIVILEGED AND CONFIDENTIAL.
Unauthorized reading, dissemination, distribution, or copying of this communication is prohibited. Please inform originator if you have received it in error.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;font-family:Arial,sans-serif;color:gray;background:white">Caution for viruses, malware etc.</span></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Arial,sans-serif;color:gray;background:white">This communication, including any attachments, may not be free of viruses, trojans, similar or new contaminants/malware, interceptions or interference,
and may not be compatible with your systems. You shall carry out virus/malware scanning on your own before opening any attachment to this e-mail. The sender of this e-mail and Company including its sister concerns shall not be liable for any damage that may
incur to you as a result of viruses, incompleteness of this message, a delay in receipt of this message or any other computer problems. </span><u></u><u></u></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote></div>