<div dir="ltr">Here is the <div><br></div><div>bt after installing all recommended debug infos</div><div> I am not sure why the debug info ssl version is asking for a previous one.</div><div><br></div><div><br></div><div><div><<a href="http://www.gnu.org/software/gdb/bugs/">http://www.gnu.org/software/gdb/bugs/</a>>...</div><div>Reading symbols from /usr/sbin/opensips...done.</div><div>[New LWP 984]</div><div>[Thread debugging using libthread_db enabled]</div><div>Using host libthread_db library "/lib64/libthread_db.so.1".</div><div>Core was generated by `/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -w /etc/o'.</div><div>Program terminated with signal 6, Aborted.</div><div>#0 0x00007f4c2e4b41d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56</div><div>56<span style="white-space:pre"> </span> return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);</div><div>(gdb) bt full</div><div>#0 0x00007f4c2e4b41d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56</div><div> resultvar = 0</div><div> pid = 984</div><div> selftid = 984</div><div>#1 0x00007f4c2e4b58c8 in __GI_abort () at abort.c:90</div><div> save_stage = 2</div><div> act = {__sigaction_handler = {sa_handler = 0x7f4c24d49d81 <__FUNCTION__.8131>, sa_sigaction = 0x7f4c24d49d81 <__FUNCTION__.8131>}, sa_mask = {__val = {3472120737606682963, 139965138952436, 104, </div><div> 140720825224720, 139965030355629, 140720825224768, 5, 9038976, 10292044412622288640, 140720825224896, 5234904, 5472328, 5444843, 139965169110616, 0, 140720825224752}}, </div><div> sa_flags = 746622792, sa_restorer = 0x89ec80 <mem_lock>}</div><div> sigs = {__val = {32, 0 <repeats 15 times>}}</div><div>#2 0x00007f4c24d3a63c in shm_free (_p=<optimized out>) at ../tm/../../mem/shm_mem.h:570</div><div>No locals.</div><div>#3 free_extra_array (tags=<optimized out>, array=<optimized out>, tags_len=<optimized out>) at acc_logic.c:198</div><div> i = <optimized out></div><div>#4 free_acc_ctx (ctx=0x7f4c2c7344b0) at acc_logic.c:220</div><div> i = <optimized out></div><div>#5 tm_free_acc_ctx (param=0x7f4c2c7344b0) at acc_logic.c:260</div><div> param = 0x7f4c2c7344b0</div><div> ctx = 0x7f4c2c7344b0</div><div>#6 0x00007f4c2b33f693 in empty_tmcb_list (head=head@entry=0x7f4c2c7239e0) at t_hooks.c:53</div><div> cbp = 0x0</div><div> cbp_tmp = 0x7f4c2c80a290</div><div>#7 0x00007f4c2b30d9bf in free_cell (dead_cell=0x7f4c2c723970) at h_table.c:127</div><div> b = <optimized out></div><div> i = <optimized out></div><div> rpl = <optimized out></div><div> tt = <optimized out></div><div> foo = <optimized out></div><div> p = <optimized out></div><div>#8 0x00007f4c2b310087 in free_hash_table () at h_table.c:383</div><div> p_cell = <optimized out></div><div> tmp_cell = 0x0</div><div> i = 13944</div><div>#9 0x00007f4c2b30cf75 in tm_shutdown () at t_funcs.c:89</div><div> __FUNCTION__ = "tm_shutdown"</div><div>#10 0x0000000000501b0d in destroy_modules () at sr_module.c:527</div><div> t = 0x7f4c2e289900</div><div> foo = 0x7f4c2e289820</div><div> __FUNCTION__ = "destroy_modules"</div><div>#11 0x00000000004fc203 in cleanup (show_status=show_status@entry=1) at main.c:319</div><div> __FUNCTION__ = "cleanup"</div><div>#12 0x00000000004fcb03 in handle_sigs () at main.c:456</div><div> chld = <optimized out></div><div> chld_status = 1</div><div> overall_status = 0</div><div> i = <optimized out></div><div> do_exit = <optimized out></div><div> __FUNCTION__ = "handle_sigs"</div><div>#13 0x000000000041c990 in main_loop () at main.c:722</div><div> startup_done = <optimized out></div><div> chd_rank = 25</div><div>#14 main (argc=<optimized out>, argv=<optimized out>) at main.c:1283</div><div> cfg_stream = <optimized out></div><div> c = <optimized out></div><div> r = <optimized out></div><div> tmp = 0x1 <Address 0x1 out of bounds></div><div> tmp_len = <optimized out></div><div> port = <optimized out></div><div> proto = <optimized out></div><div> protos_no = <optimized out></div><div> options = 0x6317c8 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"</div><div>---Type <return> to continue, or q <return> to quit---</div><div> ret = -1</div><div> seed = 1199514634</div><div> rfd = <optimized out></div><div> __FUNCTION__ = "main"</div><div>(gdb) </div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 9, 2017 at 1:16 AM, Tito Cumpen <span dir="ltr"><<a href="mailto:tito@xsvoce.com" target="_blank">tito@xsvoce.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">here is the backtrace<div><br></div><div><br></div><div><div><b>gdb /usr/sbin/opensips core.984 </b></div><div><b>GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7</b></div><div><b>Copyright (C) 2013 Free Software Foundation, Inc.</b></div><div><b>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.<wbr>html</a>></b></div><div><b>This is free software: you are free to change and redistribute it.</b></div><div><b>There is NO WARRANTY, to the extent permitted by law. Type "show copying"</b></div><div><b>and "show warranty" for details.</b></div><div><b>This GDB was configured as "x86_64-redhat-linux-gnu".</b></div><div><b>For bug reporting instructions, please see:</b></div><div><b><<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/<wbr>gdb/bugs/</a>>...</b></div><div><b>Reading symbols from /usr/sbin/opensips...done.</b></div><div><b>[New LWP 984]</b></div><div><b>[Thread debugging using libthread_db enabled]</b></div><div><b>Using host libthread_db library "/lib64/libthread_db.so.1".</b></div><div><b>Core was generated by `/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -w /etc/o'.</b></div><div><b>Program terminated with signal 6, Aborted.</b></div><div><b>#0 0x00007f4c2e4b41d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/<wbr>linux/raise.c:56</b></div><div><b>56<span style="white-space:pre-wrap"> </span> return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);</b></div><div><b>Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-20.el7_<wbr>2.x86_64 keyutils-libs-1.5.8-3.el7.x86_<wbr>64 krb5-libs-1.14.1-27.el7_3.x86_<wbr>64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.<wbr>x86_64 libgcc-4.8.5-11.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_<wbr>64 libstdc++-4.8.5-11.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64 nss-util-3.28.4-1.0.el7_3.x86_<wbr>64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.<wbr>1.x86_64 pcre-8.32-15.el7_2.1.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64</b></div><div><b>(gdb) bt full</b></div><div><b>#0 0x00007f4c2e4b41d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/<wbr>linux/raise.c:56</b></div><div><b> resultvar = 0</b></div><div><b> pid = 984</b></div><div><b> selftid = 984</b></div><div><b>#1 0x00007f4c2e4b58c8 in __GI_abort () at abort.c:90</b></div><div><b> save_stage = 2</b></div><div><b> act = {__sigaction_handler = {sa_handler = 0x7f4c24d49d81 <__FUNCTION__.8131>, sa_sigaction = 0x7f4c24d49d81 <__FUNCTION__.8131>}, sa_mask = {__val = {3472120737606682963, 139965138952436, 104, </b></div><div><b> 140720825224720, 139965030355629, 140720825224768, 5, 9038976, 10292044412622288640, 140720825224896, 5234904, 5472328, 5444843, 139965169110616, 0, 140720825224752}}, </b></div><div><b> sa_flags = 746622792, sa_restorer = 0x89ec80 <mem_lock>}</b></div><div><b> sigs = {__val = {32, 0 <repeats 15 times>}}</b></div><div><b>#2 0x00007f4c24d3a63c in shm_free (_p=<optimized out>) at ../tm/../../mem/shm_mem.h:570</b></div><div><b>No locals.</b></div><div><b>#3 free_extra_array (tags=<optimized out>, array=<optimized out>, tags_len=<optimized out>) at acc_logic.c:198</b></div><div><b> i = <optimized out></b></div><div><b>#4 free_acc_ctx (ctx=0x7f4c2c7344b0) at acc_logic.c:220</b></div><div><b> i = <optimized out></b></div><div><b>#5 tm_free_acc_ctx (param=0x7f4c2c7344b0) at acc_logic.c:260</b></div><div><b> param = 0x7f4c2c7344b0</b></div><div><b> ctx = 0x7f4c2c7344b0</b></div><div><b>#6 0x00007f4c2b33f693 in empty_tmcb_list (head=head@entry=<wbr>0x7f4c2c7239e0) at t_hooks.c:53</b></div><div><b> cbp = 0x0</b></div><div><b> cbp_tmp = 0x7f4c2c80a290</b></div><div><b>#7 0x00007f4c2b30d9bf in free_cell (dead_cell=0x7f4c2c723970) at h_table.c:127</b></div><div><b> b = <optimized out></b></div><div><b> i = <optimized out></b></div><div><b> rpl = <optimized out></b></div><div><b> tt = <optimized out></b></div><div><b> foo = <optimized out></b></div><div><b> p = <optimized out></b></div><div><b>#8 0x00007f4c2b310087 in free_hash_table () at h_table.c:383</b></div><div><b> p_cell = <optimized out></b></div><div><b> tmp_cell = 0x0</b></div><div><b> i = 13944</b></div><div><b>#9 0x00007f4c2b30cf75 in tm_shutdown () at t_funcs.c:89</b></div><div><b> __FUNCTION__ = "tm_shutdown"</b></div><div><b>#10 0x0000000000501b0d in destroy_modules () at sr_module.c:527</b></div><div><b> t = 0x7f4c2e289900</b></div><div><b> foo = 0x7f4c2e289820</b></div><div><b> __FUNCTION__ = "destroy_modules"</b></div><div><b>#11 0x00000000004fc203 in cleanup (show_status=show_status@entry<wbr>=1) at main.c:319</b></div><div><b> __FUNCTION__ = "cleanup"</b></div><div><b>#12 0x00000000004fcb03 in handle_sigs () at main.c:456</b></div><div><b> chld = <optimized out></b></div><div><b> chld_status = 1</b></div><div><b> overall_status = 0</b></div><div><b> i = <optimized out></b></div><div><b> do_exit = <optimized out></b></div><div><b> __FUNCTION__ = "handle_sigs"</b></div><div><b>#13 0x000000000041c990 in main_loop () at main.c:722</b></div><div><b> startup_done = <optimized out></b></div><div><b> chd_rank = 25</b></div><div><b>#14 main (argc=<optimized out>, argv=<optimized out>) at main.c:1283</b></div><div><b> cfg_stream = <optimized out></b></div><div><b> c = <optimized out></b></div><div><b> r = <optimized out></b></div><div><b> tmp = 0x1 <Address 0x1 out of bounds></b></div><div><b> tmp_len = <optimized out></b></div><div><b> port = <optimized out></b></div><div><b> proto = <optimized out></b></div><div><b> protos_no = <optimized out></b></div><div><b> options = 0x6317c8 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:<wbr>t:u:g:P:G:W:o:"</b></div><div><b>---Type <return> to continue, or q <return> to quit---</b></div><div><b> ret = -1</b></div><div><b> seed = 1199514634</b></div><div><b> rfd = <optimized out></b></div><div><b> __FUNCTION__ = "main"</b></div></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 9, 2017 at 12:30 AM, Tito Cumpen <span dir="ltr"><<a href="mailto:tito@xsvoce.com" target="_blank">tito@xsvoce.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Liviu,<div><br></div><div><br></div><div>I am also seeing this </div><div><br></div><div>: ERROR:proto_wss:tls_print_errs<wbr>tack: TLS errstack: error:0606E06F:digital envelope routines:EVP_MD_CTX_copy_ex:in<wbr>put not initialized<br></div><div><br></div><div>eventually it crashes. I think there may be an issue with tls. I am using OpenSSL 1.0.2l 25 May 2017</div><div><br></div><div><br></div><div>Any idea what this could be ?</div><div><br></div><div><br></div><div><br></div></div><div class="m_-2903467947597422233HOEnZb"><div class="m_-2903467947597422233h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 4, 2017 at 1:53 AM, Tito Cumpen <span dir="ltr"><<a href="mailto:tito@xsvoce.com" target="_blank">tito@xsvoce.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Liviu,<div><br></div><div><br></div><div>I am not sure if it's the same issue. I disabled all mongo related modules from loading and I am still getting some crashes with the following error:</div><div><br></div><div>Aug 4 08:00:56 cloud-server-09 /sbin/opensips[6630]: ERROR:proto_wss:tls_blocking_w<wbr>rite: TLS failed to send data</div><div>Aug 4 08:00:56 cloud-server-09 /sbin/opensips[6630]: ERROR:proto_wss:tls_write: TLS connection to someip:50803 write failed</div><div>Aug 4 08:00:56 cloud-server-09 /sbin/opensips[6630]: ERROR:proto_wss:tls_write: TLS write error:</div><div>Aug 4 08:00:56 cloud-server-09 /sbin/opensips[6630]: ERROR:proto_wss:tls_blocking_w<wbr>rite: TLS failed to send data</div><div>Aug<span class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301gmail-Apple-converted-space"> </span>4 08:01:56 cloud-server-09 /sbin/opensips[6630]: CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...<br></div><div><p class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301gmail-p1"><span class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301gmail-s1"><br></span></p><p class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301gmail-p1"><span class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301gmail-s1"><br></span></p></div></div><div class="m_-2903467947597422233m_3967838923311140655HOEnZb"><div class="m_-2903467947597422233m_3967838923311140655h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 3, 2017 at 1:09 AM, Liviu Chircu <span dir="ltr"><<a href="mailto:liviu@opensips.org" target="_blank">liviu@opensips.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p><tt>Hi, Tito!</tt></p>
<p><tt>It seems there are quite a few issues with the new
cachedb_mongodb, reported here [1]. On a happy note, most of
them have also been fixed in that pull request, and a merge is
pending soon.</tt></p>
<p><tt>Provided that you are not using tls_mgm, case in which you'd
likely need to recompile your libmongoc with
"--disable-automatic-init-and-<wbr>cleanup", the upcoming merge
should fix your crashes.</tt></p>
<p><tt>Best regards,<br>
</tt></p>
<p><tt>[1]: <a class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301m_-1884138629961186910moz-txt-link-freetext" href="https://github.com/OpenSIPS/opensips/pull/1163" target="_blank">https://github.com/OpenSIPS/op<wbr>ensips/pull/1163</a></tt><br>
</p>
<pre class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301m_-1884138629961186910moz-signature" cols="72">Liviu Chircu
OpenSIPS Developer
<a class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301m_-1884138629961186910moz-txt-link-freetext" href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.<wbr>com</a></pre><div><div class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301h5">
<div class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301m_-1884138629961186910moz-cite-prefix">On 03.08.2017 11:03, Tito Cumpen wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301h5">
<div dir="ltr">I am getting crashes on a daily basis with the
following message
<div><br>
</div>
<div>
<div>Aug 3 07:42:52 /sbin/opensips[7681]:
CRITICAL:core:receive_fd: EOF on 106</div>
<div>Aug 3 07:42:52 /sbin/opensips[7641]:
NOTICE:presence:destroy: destroy module ...</div>
<div>Aug 3 07:42:52 /sbin/opensips[7641]:
NOTICE:cachedb_mongodb:destroy<wbr>: destroy module
cachedb_mongodb ...</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>I am running opensips with the following memory flags -M
1024 -m 256</div>
<div><br>
</div>
<div>I am not seeing any core dumps being generated either.
Anyone have any ideas?</div>
</div>
<br>
<fieldset class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301m_-1884138629961186910mimeAttachmentHeader"></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
Users mailing list
<a class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301m_-1884138629961186910moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="m_-2903467947597422233m_3967838923311140655m_-8197048840973716301m_-1884138629961186910moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>