Yes sir.<div><br></div><div>Thanks,</div><div><br>BobbyS<br><br><div class="gmail_quote">On Mon, Feb 28, 2011 at 4:02 PM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org">bogdan@opensips.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi Bobby,<br>
<br>
Is this the same case as the one you reported on the SF tracker?<br>
<br>
Regards,<br>
Bogdan<br>
<br>
Bobby Smith wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div class="h5">
This is on revision 7081.<br>
<br>
It seems to be fairly consistently happening around free_dlg_dlg in dlg_hash.c which is triggering it -- we have several cores showing this behavior.<br>
<br>
What are the next steps for debugging this? This is on a production machine and we haven't been able to reproduce it elsewhere, so we are somewhat sensitive to load issues while trying to troubleshoot this.<br>
<br>
Thanks.<br>
<br>
<br>
<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 0x00000000004a2bcf in fm_insert_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:155<br>
155 (*f)->prev = &(frag->u.nxt_free);<br>
(gdb) bt full<br>
#0 0x00000000004a2bcf in fm_insert_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:155<br>
f = 0x2b875eba4070<br>
hash = 3<br>
#1 fm_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:460<br>
f = 0x2b875f41be40<br>
n = <value optimized out><br>
__FUNCTION__ = "fm_free"<br>
#2 0x00002b875e14dcde in free_dlg_dlg () at dlg_hash.c:168<br>
i = 0<br>
#3 destroy_dlg_table () at dlg_hash.c:234<br>
dlg = 0x2b875f319a80<br>
i = 5<br>
#4 0x00002b875e13a651 in mod_destroy () at dialog.c:742<br>
No locals.<br>
#5 0x0000000000476ff4 in destroy_modules () at sr_module.c:370<br>
t = 0x785328<br>
foo = 0x785258<br>
#6 0x00000000004252e0 in cleanup (show_status=1) at main.c:336<br>
No locals.<br>
#7 0x00000000004261fb in handle_sigs () at main.c:533<br>
chld = 0<br>
chld_status = 139<br>
i = <value optimized out><br>
do_exit = 1<br>
__FUNCTION__ = "handle_sigs"<br>
#8 0x000000000042a5a9 in main_loop (argc=3, argv=0x4) at main.c:913<br>
i = 4<br>
pid = <value optimized out><br>
si = 0x0<br>
startup_done = 0x0<br>
chd_rank = 16<br>
__FUNCTION__ = "main_loop"<br>
#9 main (argc=3, argv=0x4) at main.c:1388<br>
cfg_log_stderr = <value optimized out><br>
cfg_stream = 0x831e010<br>
c = <value optimized out><br>
r = <value optimized out><br>
tmp = 0x4fc447 "H\215\005\262\333#"<br>
tmp_len = <value optimized out><br>
port = <value optimized out><br>
proto = <value optimized out><br>
ret = <value optimized out><br>
seed = 586369539<br>
rfd = <value optimized out><br>
__FUNCTION__ = "main"<br>
(gdb) <br>
<br>
(gdb) frame 0<br>
#0 0x00000000004a2bcf in fm_insert_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:155<br>
155 (*f)->prev = &(frag->u.nxt_free);<br>
(gdb) list<br>
150<br>
151 /*insert it here*/<br>
152 frag->prev = f;<br>
153 frag->u.nxt_free=*f;<br>
154 if( *f )<br>
155 (*f)->prev = &(frag->u.nxt_free);<br>
156<br>
157 *f=frag;<br>
158 qm->free_hash[hash].no++;<br>
159 free_plus(qm , frag->size);<br>
(gdb) info locals<br>
f = 0x2b875eba4070<br>
hash = 3<br>
(gdb) print *f<br>
$1 = (struct fm_frag *) 0x2b865f235290<br>
(gdb) print *(*f)<br>
Cannot access memory at address 0x2b865f235290<br>
(gdb) frame 1<br>
#1 fm_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:460<br>
460 fm_insert_free(qm, f);<br>
(gdb) print f<br>
$2 = (struct fm_frag *) 0x2b875f41be40<br>
(gdb) list<br>
455 goto join;<br>
456 }<br>
457<br>
458 no_join:<br>
459<br>
460 fm_insert_free(qm, f);<br>
461 }<br>
462<br>
463<br>
464 #ifdef DBG_F_MALLOC<br>
(gdb) <br>
<br>
And a second crash here a few hours after restarting from above:<br>
<br>
Core was generated by `/usr/local/opensips/sbin/opensips -u opensips'.<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 0x00000000004a2bcf in fm_insert_free (qm=0x2aca30c14000, p=<value optimized out>) at mem/f_malloc.c:155<br>
155 (*f)->prev = &(frag->u.nxt_free);<br>
(gdb) bt full<br>
#0 0x00000000004a2bcf in fm_insert_free (qm=0x2aca30c14000, p=<value optimized out>) at mem/f_malloc.c:155<br>
f = 0x2aca30c14070<br>
hash = 3<br>
#1 fm_free (qm=0x2aca30c14000, p=<value optimized out>) at mem/f_malloc.c:460<br>
f = 0x2aca315c1718<br>
n = <value optimized out><br>
__FUNCTION__ = "fm_free"<br>
#2 0x00002aca301bdcde in free_dlg_dlg () at dlg_hash.c:168<br>
i = 0<br>
#3 destroy_dlg_table () at dlg_hash.c:234<br>
dlg = 0x2aca31692c70<br>
i = 0<br>
#4 0x00002aca301aa651 in mod_destroy () at dialog.c:742<br>
No locals.<br>
#5 0x0000000000476ff4 in destroy_modules () at sr_module.c:370<br>
t = 0x785328<br>
foo = 0x785258<br>
#6 0x00000000004252e0 in cleanup (show_status=1) at main.c:336<br>
No locals.<br>
#7 0x00000000004261fb in handle_sigs () at main.c:533<br>
chld = 0<br>
chld_status = 139<br>
i = <value optimized out><br>
do_exit = 1<br>
__FUNCTION__ = "handle_sigs"<br>
#8 0x000000000042a5a9 in main_loop (argc=3, argv=0x4) at main.c:913<br>
i = 4<br>
pid = <value optimized out><br>
si = 0x0<br>
startup_done = 0x0<br>
chd_rank = 16<br>
__FUNCTION__ = "main_loop"<br>
#9 main (argc=3, argv=0x4) at main.c:1388<br>
cfg_log_stderr = <value optimized out><br>
cfg_stream = 0x14cfb010<br>
c = <value optimized out><br>
r = <value optimized out><br>
tmp = 0x4fc447 "H\215\005\262\333#"<br>
<br></div></div>
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
</blockquote>
<br>
<br>
-- <br>
Bogdan-Andrei Iancu<br>
OpenSIPS eBootcamp - 28th February 2011<br>
OpenSIPS solutions and "know-how"<br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote></div><br></div>