<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman,new york,times,serif;font-size:10pt">Hey Bogdan,<br><br>Thanks for responding.<br><br>We have resolved our issue. It was a network\carrier issue.<br><br>We noticed that all of the corrupt sip messages were coming from 2 specific SBCs. It took a while, but we were finally able to narrow it down. When we dynamically removed those SBCs from the carrierroute.conf map, the parser via issue disappeared.<br><div> </div>Thanks,<br>Joel<div><br></div><div style="font-family: times new roman,new york,times,serif; font-size: 10pt;"><br><div style="font-family: arial,helvetica,sans-serif; font-size: 13px;"><font size="2" face="Tahoma"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Bogdan-Andrei Iancu <bogdan@voice-system.ro><br><b><span style="font-weight: bold;">To:</span></b> joel garcia
<joelphilipgarcia@yahoo.com><br><b><span style="font-weight: bold;">Cc:</span></b> Serge JF <serge@elasticall.com>; users@lists.opensips.org<br><b><span style="font-weight: bold;">Sent:</span></b> Tuesday, March 3, 2009 8:44:38 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [OpenSIPS-Users] OpenSIPS 1.4.2 memory corruption issue under extreme load?<br></font><br>
Hi Joel,<br><br>whatever you do from the script, it is impossible to change the original buffer (which is parsed by the via_parser).<br><br>Can you post (or privately sent) the log (debug=6) for such a request processing (if possible to capture).<br><br>Thanks and regards,<br>Bogdan<br><br>joel garcia wrote:<br>> Hey Bogdan,<br>> <br>> I'm working with Serge on this. Thank you for the helper facts.<br>> <br>> The TRACE log is before the error msg in the opensips.log. I've attached the log to the bottom.<br>> <br>> I'm hoping that our changes to the opensips.cfg are the cause of these errors. We're currently manipulating $avp(i:25) and $avp(i:35) variables to handle different carriers. Do you see anything wrong in this logic that could be corrupting the header or msg? Specifically, what I notice is that the Carrier-Name is not set in the header for this invite. We recently added some logic to default to a particular carrier if no
carrier is specified.<br>> <br>> Under volume, we are seeing a large number of these type of parser errors in both the header and the msg. I have other examples.<br>> I've also added a portion of our route() implementation.<br>> <br>> Thanks for your assistance,<br>> Joel<br>> <br>> <br>> route{<br>> <br>> xlog("TRACE:ROUTE: time($Ts) src($si:$sp) dst($Ri:$Rp) msg($mb)\n");<br>> ...<br>> if (is_method("INVITE")) {<br>> <br>> # set carrier name<br>> if (is_present_hf("Carrier-Name")) {<br>> $avp(i:25) = $(hdr(Carrier-Name){s.tolower});<br>> # valid carrier name
passed?<br>> switch ($avp(i:25)) {<br>> case "carrierA" :<br>> $avp(i:25) = "doA";<br>> break;<br>> case "carrierB" :<br>> if ($rU =~ "^\+?1?8[0|6|7|8]{2}[0-9]{5,8}") {<br>> # doA needs to be used for 1-800, 1-866, 1-877 or 1-888 calls<br>> $avp(i:25) =
"doA";<br>> } else {<br>> # doB is required for non 1-800 calls<br>> $avp(i:25) = "doB";<br>> }<br>> default :<br>> ...<br>> }<br>> } else {<br>> xlog("L_WARN", "WARN:ROUTE:CARRIER: no carrier name
specified\n");<br>> # dialing to the US or Canada?<br>> # We didn't find the Carrier-Name in the header. DoweI need to<br>> # add the Carrier-Name tag to the header or will just setting $avp(i:25) work?<br>> if ($rU =~ "^1" || $rU =~ "^\+1") {<br>> # doA is the default carrier for US and Canadian calls<br>> $avp(i:25) = "doA";<br>>
}<br>> ... handle international<br>> }<br>> }<br>> }<br>> <br>> We also tweak $avp(i:35)<br>> <br>> route[1] {<br>> if ($(hdr(Reason)) != null) {<br>> $avp(i:35) = $(hdr(Reason));<br>> };<br>> if (!t_relay()) {<br>> sl_reply_error();<br>> };<br>> exit;<br>> }<br>> <br>> parse_via error msg logs<br>> <br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: TRACE:ROUTE: time(1235826197) src(63.xx.xx.xx:5060) dst(8.xx.xx.xx:5060) msg(INVITE sip:+<a ymailto="mailto:18566654221@4.xx.xx.xx" href="mailto:18566654221@4.xx.xx.xx">18566654221@4.xx.xx.xx</a>:5060
SIP/2.0<br>> From: <sip:+<a ymailto="mailto:18668496441@63.xx.xx.xx" href="mailto:18668496441@63.xx.xx.xx">18668496441@63.xx.xx.xx</a>:5060>;tag=telstage-67aa-49a935d9<br>> To: sip:+<a ymailto="mailto:18566654221@8.xx.xx.xx" href="mailto:18566654221@8.xx.xx.xx">18566654221@8.xx.xx.xx</a>;tag=gK02b2b6e2<br>> Contact: <sip:63.xx.xx.xx:5060;transport=udp><br>> Call-ID: <a ymailto="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx" href="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx">49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx</a><br>> CSeq: 32043 INVITE<br>> Content-Length: 177<br>> Content-Type: application/sdp<br>> Content-Disposition: session; handling=required<br>> Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage%2D67aa%2D49a935d9><br>> Session-Expires: 1800;refresher=uac<br>> Supported: timer<br>>
Max-Forwards: 70<br>> Via: SIP/2.0/UDP 63.xx.xx.xx:5060;branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> v=0<br>> o=Sonus_UAC 8758 21805 IN IP4 4.xx.xx.xx<br>> s=SIP Media Capabilities<br>> c=IN IP4 4.xx.xx.xx<br>> t=0 0<br>> m=audio 11750 RTP/AVP 0<br>> a=rtpmap:0 PCMU/8000<br>> a=sendrecv<br>> a=maxptime:20<br>> )<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: invalid port number <5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: <SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> From: <sip:+<a ymailto="mailto:18668496441@63.xx.xx.xx"
href="mailto:18668496441@63.xx.xx.xx">18668496441@63.xx.xx.xx</a>:5060>;tag=telstage-67aa-49a935d9<br>> To: sip:+<a ymailto="mailto:18566654221@8.xx.xx.xx" href="mailto:18566654221@8.xx.xx.xx">18566654221@8.xx.xx.xx</a>;tag=gK02b2b6e2<br>> Call-ID: <a ymailto="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx" href="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx">49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx</a><br>> CSeq: 32043 INVITE<br>> Record-Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage-67aa-49a935d9><br>> Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed<br>> Contact: <sip:+<a ymailto="mailto:18566654221@4.xx.xx.xx" href="mailto:18566654221@4.xx.xx.xx">18566654221@4.xx.xx.xx</a>:5060><br>> Allow: INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br>> Content-Length: 177<br>>
Content-Disposition: session; handling=required<br>> Content-Type: application/sdp<br>> v=0<br>> o=Sonus_UAC 11722 5484 IN IP4 4.xx.xx.xx<br>> s=SIP Media Capabilities<br>> c=IN IP4 4.xx.xx.xx<br>> t=0 0<br>> m=audio 22942 RTP/AVP 0<br>> a=rtpmap:0 PCMU/8000<br>> a=sendrecv<br>> a=maxptime:20<br>> ><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> ><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:get_hdr_field: bad via<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: INFO:core:parse_headers: bad header field<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:tm:t_check: reply cannot be parsed<br>> Feb 28 08:03:17 sip101
/usr/local/sbin/opensips[11614]: ERROR:core:parse_via: invalid port number <5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: <SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> From: <sip:+<a ymailto="mailto:18668496441@63.xx.xx.xx" href="mailto:18668496441@63.xx.xx.xx">18668496441@63.xx.xx.xx</a>:5060>;tag=telstage-67aa-49a935d9<br>> To: sip:+<a ymailto="mailto:18566654221@8.xx.xx.xx" href="mailto:18566654221@8.xx.xx.xx">18566654221@8.xx.xx.xx</a>;tag=gK02b2b6e2<br>> Call-ID: <a ymailto="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx" href="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx">49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx</a><br>> CSeq: 32043 INVITE<br>> Record-Route:
<sip:8.xx.xx.xx:5060;lr;ftag=telstage-67aa-49a935d9><br>> Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed<br>> Contact: <sip:+<a ymailto="mailto:18566654221@4.xx.xx.xx" href="mailto:18566654221@4.xx.xx.xx">18566654221@4.xx.xx.xx</a>:5060><br>> Allow: INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br>> Content-Length: 177<br>> Content-Disposition: session; handling=required<br>> Content-Type: application/sdp<br>> v=0<br>> o=Sonus_UAC 11722 5484 IN IP4 4.xx.xx.xx<br>> s=SIP Media Capabilities<br>> c=IN IP4 4.xx.xx.xx<br>> t=0 0<br>> m=audio 22942 RTP/AVP 0<br>> a=rtpmap:0 PCMU/8000<br>> a=sendrecv<br>> a=maxptime:20<br>> ><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP
63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> ><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:get_hdr_field: bad via<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: INFO:core:parse_headers: bad header field<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:forward_reply: no 2nd via found in reply<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11609]: TRACE:ONREPLY_ROUTE: time(1235826197) src(4.xx.xx.xx:5060) dst(8.xx.xx.xx:5060) msg(SIP/2.0 200 OK<br>> Via: SIP/2.0/UDP 8.xx.xx.xx;branch=z9hG4bKff9d.4d7cb833.0<br>> Via: SIP/2.0/UDP 63.xx.xx.xx:5060;branch=z9hG4bK49a935fc-039b-00672428-07d84f2d-3f76ff5a<br>> From: <sip:+<a ymailto="mailto:18666287097@63.xx.xx.xx" href="mailto:18666287097@63.xx.xx.xx">18666287097@63.xx.xx.xx</a>:5060>;tag=telstage-31e5-49a935fc<br>> To: sip:+<a
ymailto="mailto:19197639388@8.xx.xx.xx" href="mailto:19197639388@8.xx.xx.xx">19197639388@8.xx.xx.xx</a>;tag=gK0ad718b1<br>> Call-ID: <a ymailto="mailto:49a935fc-039b-00672427-07d84f2d-3f76ff5a@63.xx.xx.xx" href="mailto:49a935fc-039b-00672427-07d84f2d-3f76ff5a@63.xx.xx.xx">49a935fc-039b-00672427-07d84f2d-3f76ff5a@63.xx.xx.xx</a><br>> CSeq: 6600 INVITE<br>> Record-Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage-31e5-49a935fc><br>> Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed<br>> Contact: <sip:+<a ymailto="mailto:19197639388@4.xx.xx.xx" href="mailto:19197639388@4.xx.xx.xx">19197639388@4.xx.xx.xx</a>:5060><br>> Allow: INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br>> Content-Length: 175<br>> Content-Disposition: session; handling=required<br>> Content-Type: application/sdp<br>> v=0<br>>
o=Sonus_UAC 27689 2463 IN IP4 4.xx.xx.xx<br>> s=SIP Media Capabilities<br>> c=IN IP4 4.55.20.66<br>> t=0 0<br>> m=audio 26588 RTP/AVP 0<br>> a=rtpmap:0 PCMU/8000<br>> a=<br>> <br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> ><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:get_hdr_field: bad via<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: INFO:core:parse_headers: bad header field<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:tm:t_check: reply cannot be parsed<br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: invalid port number <5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf><br>> Feb 28 08:03:17 sip101
/usr/local/sbin/opensips[11614]: ERROR:core:parse_via: <SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> From: <sip:+<a ymailto="mailto:18668496441@63.xx.xx.xx" href="mailto:18668496441@63.xx.xx.xx">18668496441@63.xx.xx.xx</a>:5060>;tag=telstage-67aa-49a935d9<br>> To: sip:+<a ymailto="mailto:18566654221@8.xx.xx.xx" href="mailto:18566654221@8.xx.xx.xx">18566654221@8.xx.xx.xx</a>;tag=gK02b2b6e2<br>> Call-ID: <a ymailto="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx" href="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx">49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx</a><br>> CSeq: 32043 INVITE<br>> Record-Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage-67aa-49a935d9><br>> Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed<br>> Contact: <sip:+<a
ymailto="mailto:18566654221@4.xx.xx.xx" href="mailto:18566654221@4.xx.xx.xx">18566654221@4.xx.xx.xx</a>:5060><br>> Allow: INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br>> Content-Length: 177<br>> Content-Disposition: session; handling=required<br>> Content-Type: application/sdp<br>> v=0<br>> o=Sonus_UAC 11722 5484 IN IP4 4.xx.xx.xx<br>> s=SIP Media Capabilities<br>> c=IN IP4 4.xx.xx.xx<br>> t=0 0<br>> m=audio 22942 RTP/AVP 0<br>> a=rtpmap:0 PCMU/8000<br>> a=sendrecv<br>> a=maxptime:20<br>> ><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> <br>> <br>> ------------------------------------------------------------------------<br>> *From:* Bogdan-Andrei Iancu <<a
ymailto="mailto:bogdan@voice-system.ro" href="mailto:bogdan@voice-system.ro">bogdan@voice-system.ro</a>><br>> *To:* Serge JF <<a ymailto="mailto:serge@elasticall.com" href="mailto:serge@elasticall.com">serge@elasticall.com</a>><br>> *Cc:* <a ymailto="mailto:users@lists.opensips.org" href="mailto:users@lists.opensips.org">users@lists.opensips.org</a><br>> *Sent:* Monday, March 2, 2009 1:18:41 AM<br>> *Subject:* Re: [OpenSIPS-Users] OpenSIPS 1.4.2 memory corruption issue under extreme load?<br>> <br>> Hi Sergio,<br>> <br>> First, some helper facts:<br>> <br>> 1) the message buffer is kept in private memory, so it cannot be written<br>> by other processes<br>> <br>> 2) parsing of the first via is done before starting the script<br>> execution, so, none of the modules can interfere with the buffer.<br>> <br>> So, how do you get the TRACE log ? do you use the SIP TRACE module?<br>> <br>> Is the
TRACE log after the via error?<br>> <br>> Regards,<br>> Bogdan<br>> <br>> Serge JF wrote:<br>> > Hello,<br>> ><br>> > We run a very high volume OpenSIPS 1.4.2 deployment with over 6 million<br>> > calls processed daily on a single server running CentOS 5. After 3 days at<br>> > maximum load we started seeing errors such as:<br>> ><br>> > Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]:<br>> > ERROR:core:parse_via: invalid port number<br>> > <5060?branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf><br>> ><br>> > You'll notice the question mark ? after the port number. The OpenSIPS parser<br>> > does not like this and fails in the parsing - which had to be expected. The<br>> > issue is that the message according to the XLOG statement we got at the very<br>> > beginning of our route[] was received with a semicolon as expected:<br>>
><br>> > Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: TRACE:ROUTE:<br>> > time(1235826197) src(63.xx.xx.108:5060) dst(8.xx.xx.14:5060) msg(INVITE<br>> > sip:+<a ymailto="mailto:18566654221@4.xx.xx" href="mailto:18566654221@4.xx.xx">18566654221@4.xx.xx</a> <mailto:<a ymailto="mailto:18566654221@4.xx.xx" href="mailto:18566654221@4.xx.xx">18566654221@4.xx.xx</a>>.227:5060 SIP/2.0<br>> > From: <sip:+<a ymailto="mailto:18668496441@63.xxx.xx" href="mailto:18668496441@63.xxx.xx">18668496441@63.xxx.xx</a> <mailto:<a ymailto="mailto:18668496441@63.xxx.xx" href="mailto:18668496441@63.xxx.xx">18668496441@63.xxx.xx</a>>.108:5060>;tag=telstage-67aa-49a935d9<br>> > To: sip:+<a ymailto="mailto:18566654221@8.xx.xx" href="mailto:18566654221@8.xx.xx">18566654221@8.xx.xx</a> <mailto:<a ymailto="mailto:18566654221@8.xx.xx"
href="mailto:18566654221@8.xx.xx">18566654221@8.xx.xx</a>>.19;tag=gK02b2b6e2<br>> > Contact: <sip:63.xxx.xx.108:5060;transport=udp><br>> > Call-ID: <a ymailto="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx" href="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx">49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx</a> <mailto:<a ymailto="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx" href="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx">49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx</a>>.108<br>> > CSeq: 32043 INVITE<br>> > Content-Length: 177<br>> > Content-Type: application/sdp<br>> > Content-Disposition: session; handling=required<br>> > Route: <sip:8.xx.xx.14:5060;lr;ftag=telstage%2D67aa%2D49a935d9><br>> > Session-Expires: 1800;refresher=uac<br>> > Supported: timer<br>>
> Max-Forwards: 70<br>> > Via: SIP/2.0/UDP<br>> > 63.xxx.xx.108:5060;branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>> ><br>> > Could this be due to some overwriting of string buffers in the OpenSIPS CORE<br>> > or TM module? How should we go about debugging this issue? It only seems to<br>> > happen after a few days under load. For the time being we have introduced a<br>> > nightly restart of the OpenSIPS to clear up the memory.<br>> ><br>> > Any pointer (sic) would be most welcome!<br>> ><br>> > Best Regards,<br>> ><br>> > Serge<br>> > <br>> <br>> _______________________________________________<br>> Users mailing list<br>> <a ymailto="mailto:Users@lists.opensips.org" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a> <mailto:<a ymailto="mailto:Users@lists.opensips.org"
href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>><br>> <a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br><br></div></div></div></body></html>