<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Sorry, I do have the core files, however, my /usr/sbin/opensips
is now 3.2.2 and the core dumps were taken against 3.2.3.</p>
<p>I should have thought to get the backtrace before downgrading.</p>
<p>This is a production server, however, I will try and schedule
some downtime and then restore 3.2.3 to generate the backtrace and
will try out the nightly release.</p>
<p>--Greg<br>
</p>
<div class="moz-cite-prefix">On 2021-11-12 13:09, Andrew Yager
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEX38CMHzJsiiA6ZWtR0gwUEUQzQ5gKZCbKAfSEk6Sp-Eu6kUA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Do you have a core dump with the backtrace when it
dies?
<div><br>
</div>
<div>There are a few fixes in the nightly releases around some
TLS things, and we've found these to be a better choice for
SSL performance.<br>
<br>
</div>
<div>Thanks,</div>
<div>Andrew</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 12 Nov 2021 at 02:30,
Gregory Massel <<a href="mailto:greg@switchtel.co.za"
moz-do-not-send="true" class="moz-txt-link-freetext">greg@switchtel.co.za</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Thanks. <br>
</p>
<p>This report lists it as a bug in WolfSSL, however, given
that I'm got similar errors (although not a segfault) when
changing to OpenSSL, it would appear more likely that it's
in proto_tls or tls_mgm.</p>
<p>--Greg<br>
</p>
<div>On 2021-11-11 17:06, Mark Farmer wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Confirmed here too, 3.2.2 is running fine.<br>
<div><br>
</div>
<div>There does seem to be a bug report about this:</div>
<div><br>
</div>
<div><a
href="https://github.com/OpenSIPS/opensips/issues/2667"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/OpenSIPS/opensips/issues/2667</a><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, 11 Nov 2021 at
14:33, Gregory Massel <<a
href="mailto:greg@switchtel.co.za" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">greg@switchtel.co.za</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>I've managed to downgrade to 3.2.2 and all is
working again. So this appears to be a bug
introduced in 3.2.3.</p>
<p>Environment is:</p>
<p>Linux msteams 5.4.0-90-generic
#101~18.04.1-Ubuntu SMP Fri Oct 22 09:25:04 UTC
2021 x86_64 x86_64 x86_64 GNU/Linux<br>
</p>
<p>The downgrade/fix was with:</p>
<p>apt-get install opensips=3.2.2-1
opensips-dialplan-module=3.2.2-1
opensips-http-modules=3.2.2-1
opensips-mysql-module=3.2.2-1
opensips-presence-modules=3.2.2-1
opensips-regex-module=3.2.2-1
opensips-tls-module=3.2.2-1
opensips-tls-openssl-module=3.2.2-1
opensips-tls-wolfssl-module=3.2.2-1
opensips-tlsmgm-module=3.2.2-1</p>
<p>--Greg<br>
</p>
<div>On 2021-11-11 15:56, Gregory Massel wrote:<br>
</div>
<blockquote type="cite">
<p>I'm running OpenSIPS 3.2.3 and it keeps bombing
out on SSL connections.<br>
</p>
<p>With WolfSSL it segfaults:</p>
<pre>Nov 11 11:52:04 msteams /usr/sbin/opensips[15322]: NOTICE:tls_wolfssl:verify_callback: depth = 1, verify success
Nov 11 11:52:04 msteams /usr/sbin/opensips[15322]: NOTICE:tls_wolfssl:verify_callback: depth = 0, verify success
Nov 11 11:52:04 msteams /usr/sbin/opensips[15322]: INFO:tls_wolfssl:_wolfssl_tls_async_connect: new TLS connection to <a href="http://52.114.75.24:5061" target="_blank" moz-do-not-send="true">52.114.75.24:5061</a> established
Nov 11 11:52:04 msteams /usr/sbin/opensips[15322]: NOTICE:tls_wolfssl:verify_callback: depth = 1, verify success
Nov 11 11:52:04 msteams /usr/sbin/opensips[15322]: NOTICE:tls_wolfssl:verify_callback: depth = 0, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15322]: INFO:tls_wolfssl:_wolfssl_tls_async_connect: new TLS connection to <a href="http://52.114.132.46:5061" target="_blank" moz-do-not-send="true">52.114.132.46:5061</a> established
Nov 11 11:52:05 msteams /usr/sbin/opensips[15328]: CRITICAL:core:sig_usr: segfault in process pid: 15328, id: 25
Nov 11 11:52:05 msteams kernel: [22403546.537543] opensips[15328]: segfault at 35 ip 00007ff7b4b3f790 sp 00007fff48dd0a30 error 4 in tls_wolfssl.so[7ff7b4a71000+1e3000]
Nov 11 11:52:05 msteams kernel: [22403546.537549] Code: ff ff e9 5b ff ff ff 0f 1f 00 53 48 8d 3d 08 40 3b 00 e8 53 09 00 00 85 c0 75 4f 48 8b 3d 50 40 3b 00 48 85 ff 74 14 0f 1f 00 <48> 8b 5f 08 e8 87 36 f7 ff 48 85 db 48 89 df 75
ef 5b 48 8d 3d d7
Nov 11 11:52:05 msteams /usr/sbin/opensips[15322]: NOTICE:tls_wolfssl:verify_callback: depth = 1, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15322]: NOTICE:tls_wolfssl:verify_callback: depth = 0, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15322]: INFO:tls_wolfssl:_wolfssl_tls_accept: New TLS connection from <a href="http://52.114.75.24:17152" target="_blank" moz-do-not-send="true">52.114.75.24:17152</a> accepted
Nov 11 11:52:05 msteams /usr/sbin/opensips[15322]: INFO:tls_wolfssl:tls_dump_cert_info: tls_accept: client TLS certificate subject: /CN=<a href="http://sip.pstnhub.microsoft.com" target="_blank" moz-do-not-send="true">sip.pstnhub.microsoft.com</a>, issuer: /C=US/O=Microsoft Corporation/CN=Microsoft RSA TLS CA 01
Nov 11 11:52:05 msteams /usr/sbin/opensips[15325]: NOTICE:tls_wolfssl:verify_callback: depth = 1, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15325]: NOTICE:tls_wolfssl:verify_callback: depth = 0, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15329]: CRITICAL:core:sig_usr: segfault in process pid: 15329, id: 26
Nov 11 11:52:05 msteams kernel: [22403546.732270] traps: opensips[15329] general protection fault ip:7ff7b4b0953d sp:7fff48dd0760 error:0 in tls_wolfssl.so[7ff7b4a71000+1e3000]
Nov 11 11:52:05 msteams /usr/sbin/opensips[15326]: NOTICE:tls_wolfssl:verify_callback: depth = 1, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15326]: NOTICE:tls_wolfssl:verify_callback: depth = 0, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15325]: INFO:tls_wolfssl:_wolfssl_tls_async_connect: new TLS connection to <a href="http://54.171.127.194:5061" target="_blank" moz-do-not-send="true">54.171.127.194:5061</a> established
Nov 11 11:52:05 msteams /usr/sbin/opensips[15330]: CRITICAL:core:sig_usr: segfault in process pid: 15330, id: 27
Nov 11 11:52:05 msteams kernel: [22403546.801626] traps: opensips[15330] general protection fault ip:7ff7b4b3f790 sp:7fff48dd0a30 error:0 in tls_wolfssl.so[7ff7b4a71000+1e3000]
Nov 11 11:52:05 msteams /usr/sbin/opensips[15326]: NOTICE:tls_wolfssl:verify_callback: depth = 1, verify success
Nov 11 11:52:05 msteams /usr/sbin/opensips[15326]: NOTICE:tls_wolfssl:verify_callback: depth = 0, verify success
Nov 11 11:52:06 msteams /usr/sbin/opensips[15325]: INFO:tls_wolfssl:_wolfssl_tls_async_connect: new TLS connection to <a href="http://52.114.14.70:5061" target="_blank" moz-do-not-send="true">52.114.14.70:5061</a> established
Nov 11 11:52:06 msteams /usr/sbin/opensips[15325]: INFO:tls_wolfssl:_wolfssl_tls_async_connect: new TLS connection to <a href="http://54.172.60.3:5061" target="_blank" moz-do-not-send="true">54.172.60.3:5061</a> established
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:handle_sigs: child process 15328 exited by a signal 11
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:handle_sigs: core was generated
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:handle_sigs: terminating due to SIGCHLD
Nov 11 11:52:06 msteams /usr/sbin/opensips[15296]: INFO:core:sig_usr: signal 15 received
Nov 11 11:52:06 msteams /usr/sbin/opensips[15297]: INFO:core:sig_usr: signal 15 received
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 3(15296) [time_keeper] terminated, still waiting for 28 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 4(15297) [timer] terminated, still waiting for 27 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 5(15298) [SIP receiver udp:<a href="http://127.0.0.1:5060" target="_blank" moz-do-not-send="true">127.0.0.1:5060</a>] terminated, still waiting for 26 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 6(15299) [SIP receiver udp:<a href="http://127.0.0.1:5060" target="_blank" moz-do-not-send="true">127.0.0.1:5060</a>] terminated, still waiting for 25 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 14(15308) [SIP receiver udp:[0:0:0:0:0:0:0:1]:5060] terminated, still waiting for 24 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 30(15333) [TCP main] terminated, still waiting for 23 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 9(15302) [SIP receiver udp:<a href="http://196.216.192.19:5060" target="_blank" moz-do-not-send="true">196.216.192.19:5060</a>] terminated, still waiting for 22 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 15(15310) [SIP receiver udp:[0:0:0:0:0:0:0:1]:5060] terminated, still waiting for 21 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 19(15319) [SIP receiver udp:[2001:43F8:BC0:1:0:0:0:1019]:5060] terminated, still waiting for 20 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 29(15332) [Timer handler] terminated, still waiting for 19 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 1(15294) [MI FIFO] terminated, still waiting for 18 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 21(15322) [TCP receiver] terminated, still waiting for 17 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 20(15320) [SIP receiver udp:[2001:43F8:BC0:1:0:0:0:1019]:5060] terminated, still waiting for 16 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 11(15304) [SIP receiver udp:<a href="http://196.216.192.19:5060" target="_blank" moz-do-not-send="true">196.216.192.19:5060</a>] terminated, still waiting for 15 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 8(15301) [SIP receiver udp:<a href="http://127.0.0.1:5060" target="_blank" moz-do-not-send="true">127.0.0.1:5060</a>] terminated, still waiting for 14 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 2(15295) [HTTPD INADDR_ANY:8888] terminated, still waiting for 13 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 10(15303) [SIP receiver udp:<a href="http://196.216.192.19:5060" target="_blank" moz-do-not-send="true">196.216.192.19:5060</a>] terminated, still waiting for 12 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 13(15307) [SIP receiver udp:[0:0:0:0:0:0:0:1]:5060] terminated, still waiting for 11 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 16(15311) [SIP receiver udp:[0:0:0:0:0:0:0:1]:5060] terminated, still waiting for 10 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 22(15323) [TCP receiver] terminated, still waiting for 9 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 12(15305) [SIP receiver udp:<a href="http://196.216.192.19:5060" target="_blank" moz-do-not-send="true">196.216.192.19:5060</a>] terminated, still waiting for 8 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 18(15317) [SIP receiver udp:[2001:43F8:BC0:1:0:0:0:1019]:5060] terminated, still waiting for 7 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 17(15315) [SIP receiver udp:[2001:43F8:BC0:1:0:0:0:1019]:5060] terminated, still waiting for 6 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 24(15326) [TCP receiver] terminated, still waiting for 5 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 7(15300) [SIP receiver udp:<a href="http://127.0.0.1:5060" target="_blank" moz-do-not-send="true">127.0.0.1:5060</a>] terminated, still waiting for 4 more
Nov 11 11:52:06 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 23(15325) [TCP receiver] terminated, still waiting for 3 more
Nov 11 11:52:09 msteams /usr/sbin/opensips[15293]: INFO:core:shutdown_opensips: process 26(15329) [TCP receiver] terminated, still waiting for 2 more
Nov 11 11:52:11 msteams /usr/sbin/opensips[15293]: INFO:core:cleanup: cleanup
Nov 11 11:52:12 msteams /usr/sbin/opensips[15293]: INFO:tls_wolfssl:mod_destroy: destroying tls_wolfssl module
Nov 11 11:52:12 msteams /usr/sbin/opensips[15293]: CRITICAL:core:fm_free: freeing already freed shm pointer (0x7ff7badd7878), first free: (null): (null)(0) - aborting!
Nov 11 11:52:18 msteams systemd[1]: opensips.service: Main process exited, code=dumped, status=6/ABRT
Nov 11 11:52:18 msteams systemd[1]: opensips.service: Failed with result 'core-dump'.
Nov 11 11:52:18 msteams systemd[1]: opensips.service: Service hold-off time over, scheduling restart.
Nov 11 11:52:18 msteams systemd[1]: opensips.service: Scheduled restart job, restart counter is at 1.
Nov 11 11:52:18 msteams systemd[1]: Stopped OpenSIPS is a very fast and flexible SIP (RFC3261) server.
</pre>
With OpenSSL it remains running, however, is still
extremely unhappy:<br>
<pre>Nov 11 15:46:35 msteams /usr/sbin/opensips[5044]: CRITICAL:core:io_watch_add: #012>>> fd_array idx 8 (fd=236) points to bogus map (fd=-1,type=0,flags=20000000,data=(nil))#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at <a href="https://github.com/OpenSIPS/opensips/issues" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/OpenSIPS/opensips/issues</a>
Nov 11 15:46:35 msteams /usr/sbin/opensips[5044]: CRITICAL:core:io_watch_add: #012>>> used fd map fd=227 is not present in fd_array (fd=227,type=19,flags=80000002,data=0x7f4b4e12eb80)#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at <a href="https://github.com/OpenSIPS/opensips/issues" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/OpenSIPS/opensips/issues</a>
Nov 11 15:46:35 msteams /usr/sbin/opensips[5044]: CRITICAL:core:io_watch_add: #012>>> unused fd_map fd=236 has bogus data (fd=-1,flags=20000000,data=(nil))#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at <a href="https://github.com/OpenSIPS/opensips/issues" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/OpenSIPS/opensips/issues</a>
Nov 11 15:46:35 msteams /usr/sbin/opensips[5044]: CRITICAL:core:io_watch_add: [TCP_main] check failed after successful fd add (fd=244,type=19,data=0x7f4b4e1c8240,flags=2) already=0
</pre>
<p>I didn't have these issues with OpenSIPS 3.1,
however, am not sure how to downgrade as, when I
do, it complains that the database table
structure is not suitable and there does not
appear to be a migration script to revert to an
older version.</p>
<p><br>
</p>
<div>-- <br>
<span
style="font-size:11pt;font-family:Assistant;color:black">Regards<br>
</span> <span
style="font-size:11pt;font-family:Assistant;color:rgb(68,84,106)">
<b>Gregory Massel</b><br>
</span> <span
style="font-size:11pt;font-family:Assistant;color:rgb(50,68,75)">
<b>T</b> +27 87 550 0000<br>
<b>F</b> +27 11 783 4877<br>
<b>W</b> <a
href="http://www.switchtel.co.za/"
target="_blank" moz-do-not-send="true">www.switchtel.co.za</a><br>
</span></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<div><span
style="font-size:11pt;font-family:Assistant;color:rgb(50,68,75)"><br>
</span></div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Users@lists.opensips.org</a><br>
<a
href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Mark Farmer<br>
<a href="mailto:farmorg@gmail.com" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">farmorg@gmail.com</a></div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<div>-- <br>
<span
style="font-size:11pt;font-family:Assistant;color:black">Regards<br>
</span> <span
style="font-size:11pt;font-family:Assistant;color:rgb(68,84,106)">
<b>Gregory Massel</b><br>
</span> <span
style="font-size:11pt;font-family:Assistant;color:rgb(50,68,75)">
<b>T</b> +27 87 550 0000<br>
<b>F</b> +27 11 783 4877<br>
<b>W</b> <a href="http://www.switchtel.co.za/"
target="_blank" moz-do-not-send="true">www.switchtel.co.za</a><br>
</span></div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">Users@lists.opensips.org</a><br>
<a
href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org">Users@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<div class="moz-signature"><span
style="font-size:11.0pt;font-family:Assistant;color:#32444B"><br>
</span></div>
</body>
</html>