<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Let<tt>'s see if the up<tt>date solves the prob<tt>lem (btw, it
is <tt>the same thread as <tt>the one related to #1736)<br>
<br>
<tt>Regards,</tt><br>
</tt></tt></tt></tt></tt></tt>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="https://www.opensips-solutions.com">https://www.opensips-solutions.com</a>
OpenSIPS Summit 2019
<a class="moz-txt-link-freetext" href="https://www.opensips.org/events/Summit-2019Amsterdam/">https://www.opensips.org/events/Summit-2019Amsterdam/</a>
</pre>
<div class="moz-cite-prefix">On 06/26/2019 06:56 PM, Ben Newlin
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:94A9B09C-B8F6-4493-BD32-87D0ED3164B6@genesys.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \,serif";
panose-1:2 7 3 9 2 2 5 2 4 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;}
tt
{mso-style-priority:99;
font-family:"Courier New",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle27
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am glad
the cause is known at least. :)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is there a
ticket I can follow for more information and to be notified
of resolution?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Ben
Newlin </span>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a class="moz-txt-link-rfc2396E" href="mailto:bogdan@opensips.org"><bogdan@opensips.org></a><br>
<b>Date: </b>Wednesday, June 26, 2019 at 7:37 AM<br>
<b>To: </b>Ben Newlin <a class="moz-txt-link-rfc2396E" href="mailto:Ben.Newlin@genesys.com"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a class="moz-txt-link-rfc2396E" href="mailto:devel@lists.opensips.org"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Thank
you Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier
New",serif"><br>
<br>
<tt>This backtrace confirms the initial suspicion - I'm
trying to find out a way to get it fixed in a more generic
way; what you experience is just one face of the problem
(with many faces :P) and there are other related reports.</tt><br>
<br>
<tt>Regards,</tt><br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit 2019<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 06/26/2019 02:34 AM, Ben Newlin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have now
been able to reproduce this crash reliably, I believe.
Please see the backtrace [1]. Is there some other
information you would like me to collect when this occurs?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[1] <a
href="https://pastebin.com/n0Ph8XH5"
moz-do-not-send="true">
https://pastebin.com/n0Ph8XH5</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Friday, June 7, 2019 at 9:15 AM<br>
<b>To: </b>Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a>,
OpenSIPS devel mailling list
<a href="mailto:devel@lists.opensips.org"
moz-do-not-send="true"><devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
</span><tt><span style="font-size:10.0pt">How often/easy is
to reproduce this crash (if possible) ? Brainstorming
with Razvan, we suspect a race (on the msg save in shmem
in transaction) between the process doing the cleanup
after the async resume and the process running the
failure route (due th 503).</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
</span><tt><span style="font-size:10.0pt">But this is just a
supposition, eventually you can validate it or not by
removing the async ??</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
</span><tt><span style="font-size:10.0pt">And on the double
ACK - I'm not 100% it is a actually a double one, as the
second has a smaller MF value (69, versus the 70 on the
first ACK).
</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit 2019<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 06/07/2019 03:52 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Bogdan,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Sorry, I
should have thought to actually look at the trace and
examine this call.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">1) Yes</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2) The
Called Party is 10.32.20.60, which is another OpenSIPS
instance. The crashed instance received the "503 Service
Unavailable" approximately 8-10 ms after sending the
INVITE.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">There is
a SIP trace of the exchange here:
<a href="https://pastebin.com/6bttsSVD"
moz-do-not-send="true">
https://pastebin.com/6bttsSVD</a>.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">One
oddity I saw is that the crashed process appears to send
(or at least siptrace) the ACK twice.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Bogdan-Andrei Iancu
<a href="mailto:bogdan@opensips.org"
moz-do-not-send="true"><bogdan@opensips.org></a><br>
<b>Date: </b>Thursday, June 6, 2019 at 11:42 AM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a>, Ben Newlin <a
href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true">
<Ben.Newlin@genesys.com></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Hi
Ben,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
</span><tt><span style="font-size:10.0pt">Thanks for
"another" report :).</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
</span><tt><span style="font-size:10.0pt">Questions:</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
</span><tt><span style="font-size:10.0pt">1) do you do any
async for the INVITE in this crash ?</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
</span><tt><span style="font-size:10.0pt">2) if it is an
YES to (1), is the caller party generating the "503
Service Unavailable" (which triggers the crash) -
10.32.20.60 ?? - a really close (from net delay
perspective) and fast to answer party ?</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
</span><tt><span style="font-size:10.0pt">Regards,</span></tt><span
style="font-size:10.0pt;font-family:"Courier New
,serif",serif"><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>Bogdan-Andrei Iancu<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>OpenSIPS Founder and Developer<o:p></o:p></pre>
<pre> <a href="https://www.opensips-solutions.com" moz-do-not-send="true">https://www.opensips-solutions.com</a><o:p></o:p></pre>
<pre>OpenSIPS Summit 2019<o:p></o:p></pre>
<pre> <a href="https://www.opensips.org/events/Summit-2019Amsterdam/" moz-do-not-send="true">https://www.opensips.org/events/Summit-2019Amsterdam/</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 06/05/2019 10:02 PM, Ben Newlin
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">We
have had another crash today.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Backtrace
is here: <a href="https://pastebin.com/q4RQC7kS"
moz-do-not-send="true">
https://pastebin.com/q4RQC7kS</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I
found this in the log at the time of the crash:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Jun 5
17:54:10 [4978] CRITICAL:core:sig_usr: segfault in
process pid: 4978, id: 8</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Please
let me know if any further information can be useful.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Devel
<a href="mailto:devel-bounces@lists.opensips.org"
moz-do-not-send="true"><devel-bounces@lists.opensips.org></a>
on behalf of Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Reply-To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Date: </b>Friday, May 10, 2019 at 6:31 PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>Re: [OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">I
found this in the log at the time of the crash:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">kernel:
opensips[5003]: segfault at 30 ip 00007fbd4c8f59d0 sp
00007ffcaa850c80 error 6 in tm.so[7fbd4c887000+8e000]</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:
</span></b><span style="color:black">Devel
<a href="mailto:devel-bounces@lists.opensips.org"
moz-do-not-send="true"><devel-bounces@lists.opensips.org></a>
on behalf of Ben Newlin
<a href="mailto:Ben.Newlin@genesys.com"
moz-do-not-send="true"><Ben.Newlin@genesys.com></a><br>
<b>Reply-To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Date: </b>Friday, May 10, 2019 at 5:44 PM<br>
<b>To: </b>OpenSIPS devel mailling list <a
href="mailto:devel@lists.opensips.org"
moz-do-not-send="true">
<devel@lists.opensips.org></a><br>
<b>Subject: </b>[OpenSIPS-Devel] OpenSIPS Crash</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We had
a crash today of our OpenSIPS instance.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Backtrace
is here: </span><a
href="https://pastebin.com/QbRJimwx"
moz-do-not-send="true">https://pastebin.com/QbRJimwx</a><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#
opensips -V</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">version:
opensips 2.4.5 (x86_64/linux)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">flags:
STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">poll
method support: poll, epoll, sigio_rt, select.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">git
revision: d025b4f61</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">main.c
compiled on 20:58:31 May 9 2019 with gcc 7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;color:black">Ben Newlin </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Devel mailing list<o:p></o:p></pre>
<pre><a href="mailto:Devel@lists.opensips.org" moz-do-not-send="true">Devel@lists.opensips.org</a><o:p></o:p></pre>
<pre><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel" moz-do-not-send="true">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>