<div>This is on revision 7081.</div><div><br></div><div>It seems to be fairly consistently happening around free_dlg_dlg in dlg_hash.c which is triggering it -- we have several cores showing this behavior.</div><div><br></div>
<div>What are the next steps for debugging this? This is on a production machine and we haven't been able to reproduce it elsewhere, so we are somewhat sensitive to load issues while trying to troubleshoot this.</div>
<div><br>Thanks.</div><div><br></div><div><br></div><div><br></div><div>Program terminated with signal 11, Segmentation fault.</div><div>#0 0x00000000004a2bcf in fm_insert_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:155</div>
<div>155<span class="Apple-tab-span" style="white-space:pre"> </span>(*f)->prev = &(frag->u.nxt_free);</div><div>(gdb) bt full</div><div>#0 0x00000000004a2bcf in fm_insert_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:155</div>
<div> f = 0x2b875eba4070</div><div> hash = 3</div><div>#1 fm_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:460</div><div> f = 0x2b875f41be40</div><div> n = <value optimized out></div>
<div> __FUNCTION__ = "fm_free"</div><div>#2 0x00002b875e14dcde in free_dlg_dlg () at dlg_hash.c:168</div><div> i = 0</div><div>#3 destroy_dlg_table () at dlg_hash.c:234</div><div> dlg = 0x2b875f319a80</div>
<div> i = 5</div><div>#4 0x00002b875e13a651 in mod_destroy () at dialog.c:742</div><div>No locals.</div><div>#5 0x0000000000476ff4 in destroy_modules () at sr_module.c:370</div><div> t = 0x785328</div><div>
foo = 0x785258</div><div>#6 0x00000000004252e0 in cleanup (show_status=1) at main.c:336</div><div>No locals.</div><div>#7 0x00000000004261fb in handle_sigs () at main.c:533</div><div> chld = 0</div><div> chld_status = 139</div>
<div> i = <value optimized out></div><div> do_exit = 1</div><div> __FUNCTION__ = "handle_sigs"</div><div>#8 0x000000000042a5a9 in main_loop (argc=3, argv=0x4) at main.c:913</div><div>
i = 4</div><div> pid = <value optimized out></div><div> si = 0x0</div><div> startup_done = 0x0</div><div> chd_rank = 16</div><div> __FUNCTION__ = "main_loop"</div>
<div>#9 main (argc=3, argv=0x4) at main.c:1388</div><div> cfg_log_stderr = <value optimized out></div><div> cfg_stream = 0x831e010</div><div> c = <value optimized out></div><div> r = <value optimized out></div>
<div> tmp = 0x4fc447 "H\215\005\262\333#"</div><div> tmp_len = <value optimized out></div><div> port = <value optimized out></div><div> proto = <value optimized out></div>
<div> ret = <value optimized out></div><div> seed = 586369539</div><div> rfd = <value optimized out></div><div> __FUNCTION__ = "main"</div><div>(gdb) </div><div><br></div>
<div><br></div><div><div>(gdb) frame 0</div><div>#0 0x00000000004a2bcf in fm_insert_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:155</div><div>155<span class="Apple-tab-span" style="white-space:pre"> </span>(*f)->prev = &(frag->u.nxt_free);</div>
<div>(gdb) list</div><div>150<span class="Apple-tab-span" style="white-space:pre"> </span></div><div>151<span class="Apple-tab-span" style="white-space:pre"> </span>/*insert it here*/</div><div>152<span class="Apple-tab-span" style="white-space:pre"> </span>frag->prev = f;</div>
<div>153<span class="Apple-tab-span" style="white-space:pre"> </span>frag->u.nxt_free=*f;</div><div>154<span class="Apple-tab-span" style="white-space:pre"> </span>if( *f )</div><div>155<span class="Apple-tab-span" style="white-space:pre"> </span>(*f)->prev = &(frag->u.nxt_free);</div>
<div>156<span class="Apple-tab-span" style="white-space:pre"> </span></div><div>157<span class="Apple-tab-span" style="white-space:pre"> </span>*f=frag;</div><div>158<span class="Apple-tab-span" style="white-space:pre"> </span>qm->free_hash[hash].no++;</div>
<div>159<span class="Apple-tab-span" style="white-space:pre"> </span>free_plus(qm , frag->size);</div><div>(gdb) info locals</div><div>f = 0x2b875eba4070</div><div>hash = 3</div><div>(gdb) print *f</div><div>$1 = (struct fm_frag *) 0x2b865f235290</div>
<div>(gdb) print *(*f)</div><div>Cannot access memory at address 0x2b865f235290</div><div>(gdb) frame 1</div><div>#1 fm_free (qm=0x2b875eba4000, p=<value optimized out>) at mem/f_malloc.c:460</div><div>460<span class="Apple-tab-span" style="white-space:pre"> </span>fm_insert_free(qm, f);</div>
<div>(gdb) print f</div><div>$2 = (struct fm_frag *) 0x2b875f41be40</div><div>(gdb) list</div><div>455<span class="Apple-tab-span" style="white-space:pre"> </span>goto join;</div><div>456<span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
<div>457<span class="Apple-tab-span" style="white-space:pre"> </span></div><div>458<span class="Apple-tab-span" style="white-space:pre"> </span>no_join:</div><div>459<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div>460<span class="Apple-tab-span" style="white-space:pre"> </span>fm_insert_free(qm, f);</div><div>461<span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div>462<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div>463<span class="Apple-tab-span" style="white-space:pre"> </span></div><div>464<span class="Apple-tab-span" style="white-space:pre"> </span>#ifdef DBG_F_MALLOC</div><div>(gdb) </div></div><div><br></div><div><br></div>
<div>And a second crash here a few hours after restarting from above:</div><div><br></div><div><div>Core was generated by `/usr/local/opensips/sbin/opensips -u opensips'.</div><div>Program terminated with signal 11, Segmentation fault.</div>
<div>#0 0x00000000004a2bcf in fm_insert_free (qm=0x2aca30c14000, p=<value optimized out>) at mem/f_malloc.c:155</div><div>155<span class="Apple-tab-span" style="white-space:pre"> </span>(*f)->prev = &(frag->u.nxt_free);</div>
<div>(gdb) bt full</div><div>#0 0x00000000004a2bcf in fm_insert_free (qm=0x2aca30c14000, p=<value optimized out>) at mem/f_malloc.c:155</div><div> f = 0x2aca30c14070</div><div> hash = 3</div><div>#1 fm_free (qm=0x2aca30c14000, p=<value optimized out>) at mem/f_malloc.c:460</div>
<div> f = 0x2aca315c1718</div><div> n = <value optimized out></div><div> __FUNCTION__ = "fm_free"</div><div>#2 0x00002aca301bdcde in free_dlg_dlg () at dlg_hash.c:168</div><div> i = 0</div>
<div>#3 destroy_dlg_table () at dlg_hash.c:234</div><div> dlg = 0x2aca31692c70</div><div> i = 0</div><div>#4 0x00002aca301aa651 in mod_destroy () at dialog.c:742</div><div>No locals.</div><div>#5 0x0000000000476ff4 in destroy_modules () at sr_module.c:370</div>
<div> t = 0x785328</div><div> foo = 0x785258</div><div>#6 0x00000000004252e0 in cleanup (show_status=1) at main.c:336</div><div>No locals.</div><div>#7 0x00000000004261fb in handle_sigs () at main.c:533</div>
<div> chld = 0</div><div> chld_status = 139</div><div> i = <value optimized out></div><div> do_exit = 1</div><div> __FUNCTION__ = "handle_sigs"</div><div>#8 0x000000000042a5a9 in main_loop (argc=3, argv=0x4) at main.c:913</div>
<div> i = 4</div><div> pid = <value optimized out></div><div> si = 0x0</div><div> startup_done = 0x0</div><div> chd_rank = 16</div><div> __FUNCTION__ = "main_loop"</div>
<div>#9 main (argc=3, argv=0x4) at main.c:1388</div><div> cfg_log_stderr = <value optimized out></div><div> cfg_stream = 0x14cfb010</div><div> c = <value optimized out></div><div> r = <value optimized out></div>
<div> tmp = 0x4fc447 "H\215\005\262\333#"</div></div><div><br></div>