<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
span.EmailStyle17
        {mso-style-type:personal;
        font-family:"Calibri","sans-serif";
        color:windowtext;}
span.EmailStyle18
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Bogdan, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Performance testing is really getting hit because of this issue. I will appreciate any feedback on this issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Rahul<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Gupta, Rahul
<br>
<b>Sent:</b> Friday, July 17, 2015 2:36 PM<br>
<b>To:</b> users@lists.opensips.org<br>
<b>Subject:</b> opensips taking 100% cpu<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am using 1.11.3-tls version and opensips processes are taking 100% CPU.
<o:p></o:p></p>
<p class="MsoNormal">PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<o:p></o:p></p>
<p class="MsoNormal">12635 root 20 0 335m 1876 1068 R 100.0 0.0 335:02.96 opensips
<o:p></o:p></p>
<p class="MsoNormal">12636 root 20 0 335m 1944 1140 R 100.0 0.0 335:03.71 opensips
<o:p></o:p></p>
<p class="MsoNormal">12641 root 20 0 335m 88m 87m R 100.0 0.1 333:26.96 opensips
<o:p></o:p></p>
<p class="MsoNormal">12645 root 20 0 335m 63m 62m R 100.0 0.1 333:23.76 opensips <o:p></o:p></p>
<p class="MsoNormal">12632 root 20 0 335m 1196 408 R 99.8 0.0 38:22.68 opensips <o:p></o:p></p>
<p class="MsoNormal">12647 root 20 0 335m 19m 18m R 99.8 0.0 561:16.07 opensips <o:p></o:p></p>
<p class="MsoNormal">12634 root 20 0 335m 1876 1068 R 99.5 0.0 333:22.10 opensips
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">None of the commands from opensipsctl are working, so I can’t use opensipsctl trap to generate gbd info. I ran the gdb on couple of the PID and got the following back trace. Seems like opensips is stuck in some lock.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@sa-z2-ccm1 ~]# gdb /usr/sbin/opensips 12633<o:p></o:p></p>
<p class="MsoNormal">GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)<o:p></o:p></p>
<p class="MsoNormal">Copyright (C) 2010 Free Software Foundation, Inc.<o:p></o:p></p>
<p class="MsoNormal">License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html">http://gnu.org/licenses/gpl.html</a>><o:p></o:p></p>
<p class="MsoNormal">This is free software: you are free to change and redistribute it.<o:p></o:p></p>
<p class="MsoNormal">There is NO WARRANTY, to the extent permitted by law. Type "show copying"<o:p></o:p></p>
<p class="MsoNormal">and "show warranty" for details.<o:p></o:p></p>
<p class="MsoNormal">This GDB was configured as "x86_64-redhat-linux-gnu".<o:p></o:p></p>
<p class="MsoNormal">For bug reporting instructions, please see:<o:p></o:p></p>
<p class="MsoNormal"><<a href="http://www.gnu.org/software/gdb/bugs/">http://www.gnu.org/software/gdb/bugs/</a>>...<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/sbin/opensips...done.<o:p></o:p></p>
<p class="MsoNormal">Attaching to program: /usr/sbin/opensips, process 12633<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/snoopy.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/snoopy.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libdl.so.2<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libresolv.so.2<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/libssl.so.10<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/libcrypto.so.10<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libc.so.6<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/ld-linux-x86-64.so.2<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libgssapi_krb5.so.2<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libkrb5.so.3<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libcom_err.so.2<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libk5crypto.so.3<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libz.so.1<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libkrb5support.so.0<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libkeyutils.so.1<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">[Thread debugging using libthread_db enabled]<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libpthread.so.0<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libselinux.so.1<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/signaling.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/signaling.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/sl.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/sl.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/tm.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/tm.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/rr.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/rr.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/maxfwd.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/maxfwd.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/sipmsgops.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/sipmsgops.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/mi_fifo.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/mi_fifo.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/uri.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/uri.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/usrloc.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/usrloc.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/registrar.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/registrar.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /usr/lib64/opensips/modules/acc.so...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /usr/lib64/opensips/modules/acc.so<o:p></o:p></p>
<p class="MsoNormal">Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.<o:p></o:p></p>
<p class="MsoNormal">Loaded symbols for /lib64/libnss_files.so.2<o:p></o:p></p>
<p class="MsoNormal">0x00007fbea5ead527 in sched_yield () from /lib64/libc.so.6<o:p></o:p></p>
<p class="MsoNormal">Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-18.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 openssl-1.0.1e-30.el6.x86_64
zlib-1.2.3-29.el6.x86_64<o:p></o:p></p>
<p class="MsoNormal">(gdb) bt<o:p></o:p></p>
<p class="MsoNormal">#0 0x00007fbea5ead527 in sched_yield () from /lib64/libc.so.6<o:p></o:p></p>
<p class="MsoNormal">#1 0x00000000004b29cc in get_lock (send_sock=0x7fbea3a25398, type=2,
<o:p></o:p></p>
<p class="MsoNormal"> buf=0x7fbea3a34ae8 "INFO <a href="sip:msml@10.204.178.9:5069;transport=TCP">
sip:msml@10.204.178.9:5069;transport=TCP</a> SIP/2.0\r\nContent-Type: application/msml+xml\r\nTo: <<a href="sip:402649@10.204.178.9">sip:402649@10.204.178.9</a>>;tag=10.204.178.95069+1+155c0002+eae86f8a\r\nVia: SIP/2.0/TCP 10.241.242.4:5060;br"..., len=864,
to=0x7fffce949820, id=0) at fastlock.h:182<o:p></o:p></p>
<p class="MsoNormal">#2 tcpconn_get (send_sock=0x7fbea3a25398, type=2, <o:p></o:p></p>
<p class="MsoNormal"> buf=0x7fbea3a34ae8 "INFO <a href="sip:msml@10.204.178.9:5069;transport=TCP">
sip:msml@10.204.178.9:5069;transport=TCP</a> SIP/2.0\r\nContent-Type: application/msml+xml\r\nTo: <<a href="sip:402649@10.204.178.9">sip:402649@10.204.178.9</a>>;tag=10.204.178.95069+1+155c0002+eae86f8a\r\nVia: SIP/2.0/TCP 10.241.242.4:5060;br"..., len=864,
to=0x7fffce949820, id=0) at tcp_main.c:900<o:p></o:p></p>
<p class="MsoNormal">#3 tcp_send (send_sock=0x7fbea3a25398, type=2, <o:p></o:p></p>
<p class="MsoNormal"> buf=0x7fbea3a34ae8 "INFO <a href="sip:msml@10.204.178.9:5069;transport=TCP">
sip:msml@10.204.178.9:5069;transport=TCP</a> SIP/2.0\r\nContent-Type: application/msml+xml\r\nTo: <<a href="sip:402649@10.204.178.9">sip:402649@10.204.178.9</a>>;tag=10.204.178.95069+1+155c0002+eae86f8a\r\nVia: SIP/2.0/TCP 10.241.242.4:5060;br"..., len=864,
to=0x7fffce949820, id=0) at tcp_main.c:1088<o:p></o:p></p>
<p class="MsoNormal">#4 0x000000000042e469 in msg_send (send_sock=<value optimized out>, proto=<value optimized out>, to=0x7fffce949820, id=<value optimized out>,
<o:p></o:p></p>
<p class="MsoNormal"> buf=0x7fbea3a34ae8 "INFO <a href="sip:msml@10.204.178.9:5069;transport=TCP">
sip:msml@10.204.178.9:5069;transport=TCP</a> SIP/2.0\r\nContent-Type: application/msml+xml\r\nTo: <<a href="sip:402649@10.204.178.9">sip:402649@10.204.178.9</a>>;tag=10.204.178.95069+1+155c0002+eae86f8a\r\nVia: SIP/2.0/TCP 10.241.242.4:5060;br"..., len=864)
at forward.h:115<o:p></o:p></p>
<p class="MsoNormal">#5 0x000000000042ecfd in forward_request (msg=0x7fbea3a37018, p=0x7fbea3a37bd0) at forward.c:444<o:p></o:p></p>
<p class="MsoNormal">#6 0x0000000000415156 in do_action (a=0x7fbea3a310f8, msg=0x7fbea3a37018) at action.c:498<o:p></o:p></p>
<p class="MsoNormal">#7 0x00000000004137dc in run_action_list (a=<value optimized out>, msg=0x7fbea3a37018) at action.c:171<o:p></o:p></p>
<p class="MsoNormal">#8 0x000000000048de9f in eval_elem (e=0x7fbea3a311d8, msg=0x7fbea3a37018, val=0x0) at route.c:1618<o:p></o:p></p>
<p class="MsoNormal">#9 0x000000000048f7f3 in eval_expr (e=0x7fbea3a311d8, msg=0x7fbea3a37018, val=0x0) at route.c:1963<o:p></o:p></p>
<p class="MsoNormal">#10 0x000000000048f78a in eval_expr (e=0x7fbea3a31228, msg=0x7fbea3a37018, val=0x0) at route.c:1979<o:p></o:p></p>
<p class="MsoNormal">#11 0x000000000048f7b9 in eval_expr (e=0x7fbea3a31278, msg=0x7fbea3a37018, val=0x0) at route.c:1984<o:p></o:p></p>
<p class="MsoNormal">#12 0x0000000000416864 in do_action (a=0x7fbea3a31570, msg=0x7fbea3a37018) at action.c:1087<o:p></o:p></p>
<p class="MsoNormal">#13 0x00000000004137dc in run_action_list (a=<value optimized out>, msg=0x7fbea3a37018) at action.c:171<o:p></o:p></p>
<p class="MsoNormal">#14 0x0000000000413ae9 in run_actions (a=<value optimized out>, msg=<value optimized out>) at action.c:136<o:p></o:p></p>
<p class="MsoNormal">#15 0x00000000004190bc in do_action (a=0x7fbea3a2af60, msg=0x7fbea3a37018) at action.c:743<o:p></o:p></p>
<p class="MsoNormal">#16 0x00000000004137dc in run_action_list (a=<value optimized out>, msg=0x7fbea3a37018) at action.c:171<o:p></o:p></p>
<p class="MsoNormal">#17 0x0000000000419c93 in do_action (a=0x7fbea3a2bb28, msg=0x7fbea3a37018) at action.c:1105<o:p></o:p></p>
<p class="MsoNormal">#18 0x00000000004137dc in run_action_list (a=<value optimized out>, msg=0x7fbea3a37018) at action.c:171<o:p></o:p></p>
<p class="MsoNormal">#19 0x0000000000419c93 in do_action (a=0x7fbea3a2bce8, msg=0x7fbea3a37018) at action.c:1105<o:p></o:p></p>
<p class="MsoNormal">#20 0x00000000004137dc in run_action_list (a=<value optimized out>, msg=0x7fbea3a37018) at action.c:171<o:p></o:p></p>
<p class="MsoNormal">#21 0x000000000041c508 in run_actions (a=0x7fbea3a287d8, msg=0x7fbea3a37018) at action.c:136<o:p></o:p></p>
<p class="MsoNormal">#22 run_top_route (a=0x7fbea3a287d8, msg=0x7fbea3a37018) at action.c:211<o:p></o:p></p>
<p class="MsoNormal">#23 0x0000000000479c87 in receive_msg (buf=<value optimized out>, len=<value optimized out>, rcv_info=<value optimized out>) at receive.c:172<o:p></o:p></p>
<p class="MsoNormal">#24 0x00000000004dd771 in udp_rcv_loop () at udp_server.c:424<o:p></o:p></p>
<p class="MsoNormal">#25 0x0000000000437cd3 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:942<o:p></o:p></p>
<p class="MsoNormal">#26 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1634<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any Ideas why is this happening ? I am running opensips with debug=2 mode and my logs are filled with the following messages<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2015-07-17T08:54:02.000-04:00 [local2] [crit] sa-z2-ccm1.ipc.com /usr/sbin/opensips[12647]: CRITICAL:core:io_watch_del: invalid fd 5735, not in [0, 2189)<o:p></o:p></p>
<p class="MsoNormal">2015-07-17T08:54:02.000-04:00 [local2] [crit] sa-z2-ccm1.ipc.com /usr/sbin/opensips[12647]: CRITICAL:core:io_watch_del: invalid fd 5736, not in [0, 2189)<o:p></o:p></p>
<p class="MsoNormal">2015-07-17T08:54:02.000-04:00 [local2] [crit] sa-z2-ccm1.ipc.com /usr/sbin/opensips[12647]: CRITICAL:core:io_watch_del: invalid fd 5737, not in [0, 2189)<o:p></o:p></p>
<p class="MsoNormal">2015-07-17T08:54:02.000-04:00 [local2] [crit] sa-z2-ccm1.ipc.com /usr/sbin/opensips[12647]: CRITICAL:core:handle_io: uknown fd type -1549462248<o:p></o:p></p>
<p class="MsoNormal">2015-07-17T08:54:02.000-04:00 [local2] [crit] sa-z2-ccm1.ipc.com /usr/sbin/opensips[12647]: CRITICAL:core:handle_io: uknown fd type -1549462224<o:p></o:p></p>
<p class="MsoNormal">2015-07-17T08:54:02.000-04:00 [local2] [crit] sa-z2-ccm1.ipc.com /usr/sbin/opensips[12647]: CRITICAL:core:handle_io: uknown fd type -1549462176<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is this a known issue ? Any workarounds or fixes available for this ?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Rahul Gupta<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p><font color="#808080" size="2" face="Arial"><font color="#808080" size="2" face="Arial">------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</font></font></p>
<p><font color="#808080" size="2" face="Arial">DISCLAIMER: This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it
by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unintended recipients are prohibited from taking action on the basis of information in this e-mail.E-mail messages may contain computer viruses or
other defects, may not be accurately replicated on other systems, or may be intercepted, deleted or interfered with without the knowledge of the sender or the intended recipient. If you are not comfortable with the risks associated with e-mail messages, you
may decide not to use e-mail to communicate with IPC. IPC reserves the right, to the extent and under circumstances permitted by applicable law, to retain, monitor and intercept e-mail messages to and from its systems.</font><br>
</p>
</div>
</body>
</html>