<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman,new york,times,serif;font-size:10pt">Hey Bogdan,<br><br>I'm working with Serge on this. Thank you for the helper facts.<br><br>The TRACE log is before the error msg in the opensips.log. I've attached the log to the bottom.<br><br>I'm hoping that our changes to the opensips.cfg are the cause of these errors. We're currently manipulating $avp(i:25) and $avp(i:35) variables to handle different carriers. Do you see anything wrong in this logic that could be corrupting the header or msg? Specifically, what I notice is that the Carrier-Name is not set in the header for this invite. We recently added some logic to default to a particular carrier if no carrier is specified.<br><br>Under volume, we are seeing a large number of these type of parser errors in both the header and the msg. I have other examples. <br> <br>I've also added a
portion of our route() implementation.<br><br>Thanks for your assistance,<br>Joel<br><br><br>route{<br><br> xlog("TRACE:ROUTE: time($Ts) src($si:$sp) dst($Ri:$Rp) msg($mb)\n");<br> ...<br> if (is_method("INVITE")) {<br><br> # set carrier name<br> if (is_present_hf("Carrier-Name")) {<br> $avp(i:25) = $(hdr(Carrier-Name){s.tolower});<br> # valid carrier name
passed?<br> switch ($avp(i:25)) {<br> case "carrierA" :<br> $avp(i:25) = "doA";<br> break;<br> case "carrierB" :<br> if
($rU =~ "^\+?1?8[0|6|7|8]{2}[0-9]{5,8}") {<br> # doA needs to be used for 1-800, 1-866, 1-877 or 1-888 calls<br> $avp(i:25) = "doA";<br> } else {<br> # doB is required for non 1-800 calls<br> $avp(i:25) = "doB";<br> }<br> <br>
default :<br> ...<br> }<br> } else {<br> xlog("L_WARN", "WARN:ROUTE:CARRIER: no carrier name specified\n");<br> # dialing to the US or
Canada?<br> # We didn't find the Carrier-Name in the header. DoweI need to <br> # add the Carrier-Name tag to the header or will just setting $avp(i:25) work?<br> if ($rU =~ "^1" || $rU =~ "^\+1") {<br> # doA is the default carrier for US and Canadian
calls<br> $avp(i:25) = "doA";<br> }<br> ... handle international<br> }<br> }<br>}<br><br>We also tweak $avp(i:35) <br><br>route[1] {<br> if ($(hdr(Reason)) != null) {<br> $avp(i:35) = $(hdr(Reason));<br> };<br> <br> if (!t_relay()) {<br>
sl_reply_error();<br> };<br> exit;<br>}<br><br>parse_via error msg logs<br><br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: TRACE:ROUTE: time(1235826197) src(63.xx.xx.xx:5060) dst(8.xx.xx.xx:5060) msg(INVITE sip:+18566654221@4.xx.xx.xx:5060 SIP/2.0<br> From: <sip:+18668496441@63.xx.xx.xx:5060>;tag=telstage-67aa-49a935d9<br> To: sip:+18566654221@8.xx.xx.xx;tag=gK02b2b6e2<br> Contact: <sip:63.xx.xx.xx:5060;transport=udp><br> Call-ID: 49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx<br> CSeq: 32043 INVITE<br> Content-Length: 177<br> Content-Type: application/sdp<br> Content-Disposition: session; handling=required<br> Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage%2D67aa%2D49a935d9><br> Session-Expires: 1800;refresher=uac<br> Supported: timer<br> Max-Forwards: 70<br> Via: SIP/2.0/UDP
63.xx.xx.xx:5060;branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br> <br> v=0<br> o=Sonus_UAC 8758 21805 IN IP4 4.xx.xx.xx<br> s=SIP Media Capabilities<br> c=IN IP4 4.xx.xx.xx<br> t=0 0<br> m=audio 11750 RTP/AVP 0<br> a=rtpmap:0 PCMU/8000<br> a=sendrecv<br> a=maxptime:20<br> ) <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: invalid port number <5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf> <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: <SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br> From: <sip:+18668496441@63.xx.xx.xx:5060>;tag=telstage-67aa-49a935d9<br> To: sip:+18566654221@8.xx.xx.xx;tag=gK02b2b6e2<br> Call-ID: 49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx<br> CSeq: 32043 INVITE<br> Record-Route:
<sip:8.xx.xx.xx:5060;lr;ftag=telstage-67aa-49a935d9><br> Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed<br> Contact: <sip:+18566654221@4.xx.xx.xx:5060><br> Allow: INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br> Content-Length: 177<br> Content-Disposition: session; handling=required<br> Content-Type: application/sdp<br> <br> v=0<br> o=Sonus_UAC 11722 5484 IN IP4 4.xx.xx.xx<br> s=SIP Media Capabilities<br> c=IN IP4 4.xx.xx.xx<br> t=0 0<br> m=audio 22942 RTP/AVP 0<br> a=rtpmap:0 PCMU/8000<br> a=sendrecv<br> a=maxptime:20<br> > <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br> > <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]:
ERROR:core:get_hdr_field: bad via <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: INFO:core:parse_headers: bad header field <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:tm:t_check: reply cannot be parsed <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: invalid port number <5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf> <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: <SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br> From: <sip:+18668496441@63.xx.xx.xx:5060>;tag=telstage-67aa-49a935d9<br> To: sip:+18566654221@8.xx.xx.xx;tag=gK02b2b6e2<br> Call-ID: 49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx<br> CSeq: 32043 INVITE<br> Record-Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage-67aa-49a935d9><br> Accept: application/sdp, application/isup,
application/dtmf, application/dtmf-relay, multipart/mixed<br> Contact: <sip:+18566654221@4.xx.xx.xx:5060><br> Allow: INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br> Content-Length: 177<br> Content-Disposition: session; handling=required<br> Content-Type: application/sdp<br> <br> v=0<br> o=Sonus_UAC 11722 5484 IN IP4 4.xx.xx.xx<br> s=SIP Media Capabilities<br> c=IN IP4 4.xx.xx.xx<br> t=0 0<br> m=audio 22942 RTP/AVP 0<br> a=rtpmap:0 PCMU/8000<br> a=sendrecv<br> a=maxptime:20<br> > <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br> > <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:get_hdr_field: bad via <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: INFO:core:parse_headers: bad header
field <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:forward_reply: no 2nd via found in reply <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11609]: TRACE:ONREPLY_ROUTE: time(1235826197) src(4.xx.xx.xx:5060) dst(8.xx.xx.xx:5060) msg(SIP/2.0 200 OK<br> Via: SIP/2.0/UDP 8.xx.xx.xx;branch=z9hG4bKff9d.4d7cb833.0<br> Via: SIP/2.0/UDP 63.xx.xx.xx:5060;branch=z9hG4bK49a935fc-039b-00672428-07d84f2d-3f76ff5a<br> From: <sip:+18666287097@63.xx.xx.xx:5060>;tag=telstage-31e5-49a935fc<br> To: sip:+19197639388@8.xx.xx.xx;tag=gK0ad718b1<br> Call-ID: 49a935fc-039b-00672427-07d84f2d-3f76ff5a@63.xx.xx.xx<br> CSeq: 6600 INVITE<br> Record-Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage-31e5-49a935fc><br> Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed<br> Contact: <sip:+19197639388@4.xx.xx.xx:5060><br> Allow:
INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br> Content-Length: 175<br> Content-Disposition: session; handling=required<br> Content-Type: application/sdp<br> <br> v=0<br> o=Sonus_UAC 27689 2463 IN IP4 4.xx.xx.xx<br> s=SIP Media Capabilities<br> c=IN IP4 4.55.20.66<br> t=0 0<br> m=audio 26588 RTP/AVP 0<br> a=rtpmap:0 PCMU/8000<br> a=<br><br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br> > <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:get_hdr_field: bad via <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: INFO:core:parse_headers: bad header field <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:tm:t_check: reply cannot be parsed <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]:
ERROR:core:parse_via: invalid port number <5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf> <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: <SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br> From: <sip:+18668496441@63.xx.xx.xx:5060>;tag=telstage-67aa-49a935d9<br> To: sip:+18566654221@8.xx.xx.xx;tag=gK02b2b6e2<br> Call-ID: 49a935d9-029f-0065abfa-8162901a-4330cddf@63.xx.xx.xx<br> CSeq: 32043 INVITE<br> Record-Route: <sip:8.xx.xx.xx:5060;lr;ftag=telstage-67aa-49a935d9><br> Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed<br> Contact: <sip:+18566654221@4.xx.xx.xx:5060><br> Allow: INVITE,ACK,CANCEL,BYE,PRACK,UPDATE<br> Content-Length: 177<br> Content-Disposition: session; handling=required<br> Content-Type:
application/sdp<br> <br> v=0<br> o=Sonus_UAC 11722 5484 IN IP4 4.xx.xx.xx<br> s=SIP Media Capabilities<br> c=IN IP4 4.xx.xx.xx<br> t=0 0<br> m=audio 22942 RTP/AVP 0<br> a=rtpmap:0 PCMU/8000<br> a=sendrecv<br> a=maxptime:20<br> > <br>Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 63.xx.xx.xx:5060branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br><br><div style="font-family: times new roman,new york,times,serif; font-size: 10pt;"><br><div style="font-family: arial,helvetica,sans-serif; font-size: 13px;"><font size="2" face="Tahoma"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Bogdan-Andrei Iancu <bogdan@voice-system.ro><br><b><span style="font-weight: bold;">To:</span></b> Serge JF <serge@elasticall.com><br><b><span style="font-weight: bold;">Cc:</span></b> users@lists.opensips.org<br><b><span
style="font-weight: bold;">Sent:</span></b> Monday, March 2, 2009 1:18:41 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [OpenSIPS-Users] OpenSIPS 1.4.2 memory corruption issue under extreme load?<br></font><br>
Hi Sergio,<br><br>First, some helper facts:<br><br>1) the message buffer is kept in private memory, so it cannot be written <br>by other processes<br><br>2) parsing of the first via is done before starting the script <br>execution, so, none of the modules can interfere with the buffer.<br><br>So, how do you get the TRACE log ? do you use the SIP TRACE module?<br><br>Is the TRACE log after the via error?<br><br>Regards,<br>Bogdan<br><br>Serge JF wrote:<br>> Hello, <br>><br>> We run a very high volume OpenSIPS 1.4.2 deployment with over 6 million<br>> calls processed daily on a single server running CentOS 5. After 3 days at<br>> maximum load we started seeing errors such as:<br>><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]:<br>> ERROR:core:parse_via: invalid port number<br>> <5060?branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf> <br>><br>> You'll notice the question mark ? after the port
number. The OpenSIPS parser<br>> does not like this and fails in the parsing - which had to be expected. The<br>> issue is that the message according to the XLOG statement we got at the very<br>> beginning of our route[] was received with a semicolon as expected:<br>><br>> Feb 28 08:03:17 sip101 /usr/local/sbin/opensips[11614]: TRACE:ROUTE:<br>> time(1235826197) src(63.xx.xx.108:5060) dst(8.xx.xx.14:5060) msg(INVITE<br>> sip:+<a ymailto="mailto:18566654221@4.xx.xx" href="mailto:18566654221@4.xx.xx">18566654221@4.xx.xx</a>.227:5060 SIP/2.0<br>> From: <sip:+<a ymailto="mailto:18668496441@63.xxx.xx" href="mailto:18668496441@63.xxx.xx">18668496441@63.xxx.xx</a>.108:5060>;tag=telstage-67aa-49a935d9<br>> To: sip:+<a ymailto="mailto:18566654221@8.xx.xx" href="mailto:18566654221@8.xx.xx">18566654221@8.xx.xx</a>.19;tag=gK02b2b6e2<br>> Contact: <sip:63.xxx.xx.108:5060;transport=udp><br>>
Call-ID: <a ymailto="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx" href="mailto:49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx">49a935d9-029f-0065abfa-8162901a-4330cddf@63.xxx.xx</a>.108<br>> CSeq: 32043 INVITE<br>> Content-Length: 177<br>> Content-Type: application/sdp<br>> Content-Disposition: session; handling=required<br>> Route: <sip:8.xx.xx.14:5060;lr;ftag=telstage%2D67aa%2D49a935d9><br>> Session-Expires: 1800;refresher=uac<br>> Supported: timer<br>> Max-Forwards: 70<br>> Via: SIP/2.0/UDP<br>> 63.xxx.xx.108:5060;branch=z9hG4bK49a93615-0234-0065adf8-8162901a-4330cddf<br>><br>> Could this be due to some overwriting of string buffers in the OpenSIPS CORE<br>> or TM module? How should we go about debugging this issue? It only seems to<br>> happen after a few days under load. For the time being we have introduced a<br>> nightly
restart of the OpenSIPS to clear up the memory.<br>><br>> Any pointer (sic) would be most welcome!<br>><br>> Best Regards,<br>><br>> Serge<br>> <br><br><br>_______________________________________________<br>Users mailing list<br><a ymailto="mailto:Users@lists.opensips.org" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a><br><a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br></div></div></div></body></html>