<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<tt>Hi Bobby,<br>
<br>
I would say the leaking is related to RTPproxy stuff:<br>
<br>
32106 msg_callbacks.c: msg_callback_add(51)<br>
16053 rtpproxy_callbacks.c: rtpproxy_pre_fwd(51)<br>
</tt>
<div><tt> 16053 rtpproxy.c: force_rtp_proxy(2848)</tt></div>
<div><tt> 16053 rtpproxy.c: pkg_strdup(2301)</tt></div>
<tt><br>
Have you enabled the "rtpproxy_autobridge" option in rtpproxy
module ? and what rtpp functions are you using ?<br>
<br>
<br>
Regards,</tt><br>
<pre class="moz-signature" cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a></pre>
<br>
On 10/26/2012 09:57 PM, Bobby Smith wrote:
<blockquote
cite="mid:CAJQeEgu+COMmatFAfk6QLdpf51xq=VLmDU=mW=zKUrTZGLhHXQ@mail.gmail.com"
type="cite">Here's some information from the memory dump
summarized (total number of alloc's printed when doing a kill
-SIGUSR1 after running for a few hours, during off peak time)
<div><br>
</div>
<div>This is a count of remaining fragments:</div>
<div><br>
</div>
<div>
<div> 31 auth_mod.c: generate_random_secret(192)</div>
<div> 31 avpops.c: avpops_init(220)</div>
<div> 8866 cfg.lex: addstr(858)</div>
<div> 155 cfg.y: yyparse(1164)</div>
<div> 806 cfg.y: yyparse(1577)</div>
<div> 32 data_lump.c: anchor_lump(398)</div>
<div> 2 data_lump.c: del_lump(342)</div>
<div> 24 data_lump.c: insert_cond_lump_after(225)</div>
<div> 8 data_lump.c: insert_cond_lump_before(252)</div>
<div> 32 data_lump.c: insert_new_lump_after(120)</div>
<div> 46 data_lump.c: insert_new_lump_before(146)</div>
<div> 16 data_lump.c: insert_subst_lump_after(172)</div>
<div> 2 dbt_base.c: dbt_init(89)</div>
<div> 11 dlg_vals.c: fetch_dlg_value(156)</div>
<div> 1 fifo_fnc.c: mi_init_fifo_server(117)</div>
<div> 1 fifo_fnc.c: mi_init_fifo_server(118)</div>
<div> 155 ip_addr.c: mk_net(56)</div>
<div> 7 lb_data.c: do_load_balance(403)</div>
<div> 7 lb_data.c: do_load_balance(450)</div>
<div> 31 lb_parser.c: parse_resources_list(115)</div>
<div> 31 load_balancer.c: fixup_resources(163)</div>
<div> 31 load_balancer.c: fixup_resources(193)</div>
<div> 31 memcache.c: register_memcache(91)</div>
<div> 1 mi/fmt.c: mi_fmt_init(45)</div>
<div> 2 mi/mi_core.c: init_mi_uptime(62)</div>
<div> 31 mi/mi.c: register_mi_cmd(145)</div>
<div> 1 mi_parser.c: mi_parser_init(48)</div>
<div> 29 mi_writer.c: mi_writer_init(51)</div>
<div> 155 mod_fix.c: fixup_pvar(437)</div>
<div> 155 mod_fix.c: fixup_regexp(303)</div>
<div> 93 mod_fix.c: fixup_str(70)</div>
<div> 32106 msg_callbacks.c: msg_callback_add(51)</div>
<div> 93 name_alias.c: add_alias(59)</div>
<div> 93 name_alias.c: add_alias(61)</div>
<div> 5 parser/contact/contact.c: parse_contacts(192)</div>
<div> 5 parser/contact/parse_contact.c: parse_contact(81)</div>
<div> 16 parser/msg_parser.c: get_hdr_field(112)</div>
<div> 10 parser/msg_parser.c: get_hdr_field(134)</div>
<div> 10 parser/msg_parser.c: get_hdr_field(158)</div>
<div> 120 parser/msg_parser.c: parse_headers(314)</div>
<div> 3 parser/msg_parser.c: set_dst_uri(758)</div>
<div> 10 parser/parse_content.c: parse_content_type_hdr(432)</div>
<div> 8 parser/parse_from.c: parse_from_header(63)</div>
<div> 10 parser/parse_multipart.c: get_all_bodies(212)</div>
<div> 10 parser/parse_multipart.c: new_part(38)</div>
<div> 3 parser/parse_param.c: parse_params(365)</div>
<div> 5 parser/parse_rr.c: do_parse_rr_body(65)</div>
<div> 13 parser/parse_to.c: parse_to_param(280)</div>
<div> 24 parser/parse_via.c: parse_via(1774)</div>
<div> 96 proxy.c: hostent_cpy(119)</div>
<div> 96 proxy.c: hostent_cpy(130)</div>
<div> 96 proxy.c: hostent_cpy(153)</div>
<div> 96 proxy.c: hostent_cpy(163)</div>
<div> 96 proxy.c: mk_proxy(238)</div>
<div> 62 pvar.c: new_pv_context(4409)</div>
<div> 341 pvar.c: pv_add_extra(4334)</div>
<div> 31 pvar.c: pv_init_extra_list(4270)</div>
<div> 4309 pvar.c: pv_parse_format(3782)</div>
<div> 11 receive.c: receive_msg(92)</div>
<div> 16 record.c: build_rr(158)</div>
<div> 16 record.c: build_rr(159)</div>
<div> 16 record.c: build_rr(160)</div>
<div>
16 record.c: build_rr(161)</div>
<div> 14 record.c: insert_rr_param_lump(126)</div>
<div> 93 route.c: fix_actions(614)</div>
<div> 155 route.c: fix_expr(191)</div>
<div> 6386 route_struct.c: mk_action(106)</div>
<div> 1984 route_struct.c: mk_elem(72)</div>
<div> 2294 route_struct.c: mk_exp(56)</div>
<div> 31 rr_cb.c: register_rrcb(59)</div>
<div> 16053 rtpproxy_callbacks.c: rtpproxy_pre_fwd(51)</div>
<div> 30 rtpproxy.c: append_opts(2742)</div>
<div> 30 rtpproxy.c: connect_rtpproxies(1281)</div>
<div> 16053 rtpproxy.c: force_rtp_proxy(2848)</div>
<div> 16053 rtpproxy.c: pkg_strdup(2301)</div>
<div> 124 script_cb.c: add_callback(60)</div>
<div> 62 script_var.c: add_var(54)</div>
<div> 62 script_var.c: add_var(61)</div>
<div> 11 script_var.c: set_var_value(114)</div>
<div> 124 socket_info.c: fix_socket_list(569)</div>
<div> 124 socket_info.c: fix_socket_list(620)</div>
<div> 124 socket_info.c: fix_socket_list(699)</div>
<div> 1 socket_info.c: get_socket_list_from_proto(952)</div>
<div> 124 socket_info.c: new_sock_info(112)</div>
<div> 124 socket_info.c: new_sock_info(117)</div>
<div> 589 sr_module.c: register_module(140)</div>
<div> 279 textops.c: fixup_method(1315)</div>
<div> 62 textops.c: hname_fixup(1244)</div>
<div> 217 timer.c: new_sr_timer(150)</div>
<div> 31 timer.c: new_timer_process_list(69)</div>
<div> 186 transformations.c: parse_transformation(1626)</div>
<div> 2 t_reply.c: fake_req(541)</div>
<div> 27 udp_server.c: udp_rcv_loop(342)</div>
<div> 465 usr_avp.c: new_avp_alias(451)</div>
<div> 17 xlog.c: buf_init(50)</div>
<div> 1 xr_writer.c: xr_writer_init(43)</div>
<div><br>
</div>
<br>
<div class="gmail_quote">On Fri, Oct 26, 2012 at 11:06 AM, Bobby
Smith <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:bobby.smith@gmail.com" target="_blank">bobby.smith@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
Greetings list,
<div><br>
We're trying to track down a memory leak with 1.7 R9225+
(It's actually been happening since we've upgraded to 1.7
in general, but we've updated svn a few times and
recompiled to see if the problem goes away).</div>
<div><br>
</div>
<div>Here's a diagram of memory usage over a period of time:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="http://i45.tinypic.com/ngpqwp.png" target="_blank">http://i45.tinypic.com/ngpqwp.png</a></div>
<div><br>
</div>
<div>Basically we're seeing a package memory leak where
package memory is never freed under negligible load. The
memory used graph looks like a step graph, where flat
correlates to a period where we're off peak with traffic.
This will continue for 2-3 days before we see opensips
processes start to hang trying to free package memory. At
this point we force an HA restart and the memory dips.</div>
<div><br>
</div>
<div>##########################</div>
<div><br>
</div>
<div>
<div>2012-10-12T18:05:35.569400+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27800]:
ERROR:core:parse_headers: pkg memory allocation failed</div>
<div>2012-10-12T18:05:35.569439+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27800]:
ERROR:core:receive_msg: parse_msg failed</div>
<div>2012-10-12T18:05:35.569871+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27795]:
ERROR:core:receive_msg: no pkg mem left for sip_msg</div>
<div>2012-10-12T18:05:35.587197+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27805]: [REQUEST <a
moz-do-not-send="true"
href="mailto:d9c7633662d475f58263856ca5b91efe@10.2.1.173"
target="_blank">d9c7633662d475f58263856ca5b91efe@10.2.1.173</a>]
:: Beginning processing OPTIONS for <a
moz-do-not-send="true"
href="mailto:d9c7633662d475f58263856ca5b91efe@10.2.1.173"
target="_blank">d9c7633662d475f58263856ca5b91efe@10.2.1.173</a>
from 10.2.1.173</div>
<div>2012-10-12T18:05:35.590072+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27798]:
ERROR:core:receive_msg: no pkg mem left for sip_msg</div>
<div>2012-10-12T18:05:35.590094+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27798]:
ERROR:core:receive_msg: no pkg mem left for sip_msg</div>
<div>2012-10-12T18:05:35.590579+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:core:get_hdr_field: out of pkg memory</div>
<div>2012-10-12T18:05:35.590600+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.590618+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:core:pv_get_callid: cannot parse Call-Id header</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>2012-10-12T18:05:35.791241+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
ERROR:core:build_res_buf_from_sip_req: parse_headers
failed</div>
<div>2012-10-12T18:05:35.791258+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:maxfwd:is_maxfwd_present: parsing MAX_FORWARD
header failed!</div>
<div>2012-10-12T18:05:35.791280+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
ERROR:sl:sl_send_reply_helper: response building failed</div>
<div>2012-10-12T18:05:35.791295+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
ERROR:core:get_hdr_field: bad via</div>
<div>2012-10-12T18:05:35.791309+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:core:get_hdr_field: out of pkg memory</div>
<div>2012-10-12T18:05:35.791324+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.791338+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.791354+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:tm:t_newtran: parse_headers failed</div>
<div>2012-10-12T18:05:35.791366+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
ERROR:core:parse_via: no pkg memory left</div>
<div>2012-10-12T18:05:35.791376+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:core:get_hdr_field: out of pkg memory</div>
<div>2012-10-12T18:05:35.791384+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.791392+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
ERROR:core:get_hdr_field: bad via</div>
<div>2012-10-12T18:05:35.791401+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:core:get_hdr_field: out of pkg memory</div>
<div>2012-10-12T18:05:35.791409+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.791418+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.791426+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
ERROR:core:build_res_buf_from_sip_req: parse_headers
failed</div>
<div>2012-10-12T18:05:35.791436+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:core:build_res_buf_from_sip_req: parse_headers
failed</div>
<div>2012-10-12T18:05:35.791443+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27799]:
ERROR:sl:sl_send_reply_helper: response building failed</div>
<div>2012-10-12T18:05:35.791452+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27796]:
ERROR:sl:sl_send_reply_helper: response building failed</div>
<div>2012-10-12T18:05:35.791470+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27797]:
ERROR:core:receive_msg: no pkg mem left for sip_msg</div>
<div>2012-10-12T18:05:35.796399+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27808]: [REQUEST <a
moz-do-not-send="true"
href="mailto:a1db91ba-bfb8788d@192.168.1.66"
target="_blank">a1db91ba-bfb8788d@192.168.1.66</a>] ::
Beginning processing BYE for <a moz-do-not-send="true"
href="mailto:a1db91ba-bfb8788d@192.168.1.66"
target="_blank">a1db91ba-bfb8788d@192.168.1.66</a>
from 10.2.1.178</div>
<div>2012-10-12T18:05:35.796427+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27807]: [REQUEST <a
moz-do-not-send="true"
href="mailto:a5b2db42-50208983@192.168.1.12"
target="_blank">a5b2db42-50208983@192.168.1.12</a>] ::
Beginning processing BYE for <a moz-do-not-send="true"
href="mailto:a5b2db42-50208983@192.168.1.12"
target="_blank">a5b2db42-50208983@192.168.1.12</a>
from 10.2.1.175</div>
<div>2012-10-12T18:05:35.796550+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27804]: [REQUEST <a
moz-do-not-send="true"
href="mailto:149befeef9bb0c032d5ec2d34b0cf287@10.2.1.195"
target="_blank">149befeef9bb0c032d5ec2d34b0cf287@10.2.1.195</a>]
:: Beginning processing BYE for <a
moz-do-not-send="true"
href="mailto:149befeef9bb0c032d5ec2d34b0cf287@10.2.1.195"
target="_blank">149befeef9bb0c032d5ec2d34b0cf287@10.2.1.195</a>
from 10.2.1.195</div>
<div>2012-10-12T18:05:35.796749+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27803]:
CRITICAL:core:qm_free: freeing already freed pointer,
first free: dlg_hash.c: free_dlg_dlg(177) - aborting</div>
<div>2012-10-12T18:05:35.813460+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
ERROR:core:get_hdr_field: out of pkg memory</div>
<div>2012-10-12T18:05:35.813489+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.813508+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
ERROR:core:pv_get_callid: cannot parse Call-Id header</div>
<div>2012-10-12T18:05:35.813542+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
ERROR:core:get_hdr_field: out of pkg memory</div>
<div>2012-10-12T18:05:35.813578+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
INFO:core:parse_headers: bad header field</div>
<div>2012-10-12T18:05:35.813601+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]:
ERROR:core:pv_get_callid: cannot parse Call-Id header</div>
<div>2012-10-12T18:05:35.813638+00:00 atlsvpctrl09
/usr/local/opensips/sbin/opensips[27794]: [REQUEST
<null>] :: Beginning processing INVITE for
<null> from X.X.X.X</div>
</div>
<div><br>
</div>
<div>###############</div>
<div><br>
We've tried increasing the pkg mem from 32 to 64 MB per
process. However, we are also using a set of 1.6 proxies
with the same configuration, and these regularly approach
32 MB of memory but free memory during our "off" periods,
so we never encounter a leak there.</div>
<div><br>
</div>
<div>We have some memory logs with mem debugging support
compiled and enabled, is there a place we can upload these
and have someone take a look to try and figure out where
the memory leak might be coming from?<br>
<br>
Thanks,</div>
<div><br>
</div>
<div><br>
</div>
</blockquote>
</div>
<br>
</div>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</body>
</html>