<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><tt>Hi, Alec!</tt></p>
<p><tt>Unfortunately, the backtrace and logs are not very helpful,
as they only indicate a crash somewhere in libcurl.</tt></p>
<p><tt>Following from this, I tried to replicate the issue on a 2.2
(commit d250e1c), but with little success, even at a high CPS
rate. I did manage to spot an issue with the non-async call,
where libcurl clearly spawns a thread to do each blocking
transfer. This is not okay, as rest_client instructs libcurl to
use a series of faster, OpenSIPS-specific non thread safe
allocation functions. The good news is that we can revert this
with an easy patch.<br>
</tt></p>
<p><tt>So, as a next step, we have two directions:</tt></p>
<p><tt>- investigate your crash further / try to replicate. For this,
I would need some specific startup parameters. (OS, curl library
package/version, </tt><tt><tt>OpenSIPS startup command line, </tt>rest_client
module configuration)</tt></p>
<p><tt>- I notice you are running off the 2.2 release packages. If
it's possible for you to compile / install OpenSIPS 2.2 from
sources on your box, you can change dir to the root OpenSIPS
dir, and apply the patch below in order to fix the "thread
safety" bug. Maybe the original problem goes away with this as
well:</tt></p>
<p><tt></tt><tt>git apply <(base64 -d <<EOF</tt><tt><br>
</tt><tt>ZGlmZiAtLWdpdCBhL21vZHVsZXMvcmVzdF9jbGllbnQvcmVzdF9jbGllbnQuYyBiL21vZHVsZXMv</tt><tt><br>
</tt><tt>cmVzdF9jbGllbnQvcmVzdF9jbGllbnQuYwppbmRleCA5Y2E3N2VmLi5lZmRiOTliIDEwMDY0NAot</tt><tt><br>
</tt><tt>LS0gYS9tb2R1bGVzL3Jlc3RfY2xpZW50L3Jlc3RfY2xpZW50LmMKKysrIGIvbW9kdWxlcy9yZXN0</tt><tt><br>
</tt><tt>X2NsaWVudC9yZXN0X2NsaWVudC5jCkBAIC0xNTQsNyArMTU0LDcgQEAgc3RydWN0IG1vZHVsZV9l</tt><tt><br>
</tt><tt>eHBvcnRzIGV4cG9ydHMgPSB7CiAJbW9kX2Rlc3Ryb3ksCiAJY2hpbGRfaW5pdCwvKiBwZXItY2hp</tt><tt><br>
</tt><tt>bGQgaW5pdCBmdW5jdGlvbiAqLwogfTsKLQorI2lmIDAKIHN0YXRpYyB2b2lkICpvc2lwc19tYWxs</tt><tt><br>
</tt><tt>b2Moc2l6ZV90IHNpemUpCiB7CiAJdm9pZCAqcCA9IHBrZ19tYWxsb2Moc2l6ZSk7CkBAIC0xOTcs</tt><tt><br>
</tt><tt>NiArMTk3LDcgQEAgc3RhdGljIHZvaWQgb3NpcHNfZnJlZSh2b2lkICpwdHIpCiAJaWYgKHB0cikK</tt><tt><br>
</tt><tt>IAkJcGtnX2ZyZWUocHRyKTsKIH0KKyNlbmRpZgogCiBzdGF0aWMgaW50IG1vZF9pbml0KHZvaWQp</tt><tt><br>
</tt><tt>CiB7CkBAIC0yMTAsMTIgKzIxMSwxNCBAQCBzdGF0aWMgaW50IG1vZF9pbml0KHZvaWQpCiAJCWNv</tt><tt><br>
</tt><tt>bm5lY3RfcG9sbF9pbnRlcnZhbCA9IDIwOwogCX0KIAorI2lmIDAKIAljdXJsX2dsb2JhbF9pbml0</tt><tt><br>
</tt><tt>X21lbShDVVJMX0dMT0JBTF9BTEwsCiAJCQkJCQkgb3NpcHNfbWFsbG9jLAogCQkJCQkJIG9zaXBz</tt><tt><br>
</tt><tt>X2ZyZWUsCiAJCQkJCQkgb3NpcHNfcmVhbGxvYywKIAkJCQkJCSBvc2lwc19zdHJkdXAsCiAJCQkJ</tt><tt><br>
</tt><tt>CQkgb3NpcHNfY2FsbG9jKTsKKyNlbmRpZgogCiAJbXVsdGlfaGFuZGxlID0gY3VybF9tdWx0aV9p</tt><tt><br>
</tt><tt>bml0KCk7CiAK</tt><tt><br>
</tt><tt>EOF</tt><tt><br>
</tt><tt>)</tt><br>
</p>
<pre class="moz-signature" cols="72">Liviu Chircu
OpenSIPS Developer
<a class="moz-txt-link-freetext" href="http://www.opensips-solutions.com">http://www.opensips-solutions.com</a></pre>
<div class="moz-cite-prefix">On 22.11.2016 03:35, Alec J Fett wrote:<br>
</div>
<blockquote
cite="mid:CAKqasgLQgWO2P9Vak2mLcdWrQJOZT-CU16s+BitYmqGkeQ7e9Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<div style="font-size:12.8px">(resending with truncated log
since my last message was oversized for the list)</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Hi Devs,</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">I have an existing config with a
working rest_post() query without issue.</div>
<div style="font-size:12.8px">By simply wrapping with async()
and specifying a resume route, OpenSIPS is crashing every time
the request is made via https. </div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Interestingly, this seems to
happen only when making https request with async()... that is
a) http works with async() and b) both http and https work
without async().<br>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Logs and stack trace below.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Thanks,</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">Alec</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">
<div>version: opensips 2.2.2 (x86_64/linux)</div>
<div>flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</div>
<div>ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535</div>
<div>poll method support: poll, epoll_lt, epoll_et, sigio_rt,
select.</div>
<div>git revision: d250e1c</div>
<div>main.c compiled on with gcc 4.8</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div style="font-size:12.8px">[New LWP 8915]</div>
<div style="font-size:12.8px">[Thread debugging using
libthread_db enabled]</div>
<div style="font-size:12.8px">Using host libthread_db
library "/usr/grte/v4/lib64/libthread_<wbr>db.so.1".</div>
<div style="font-size:12.8px">Core was generated by
`opensips -w /tmp'.</div>
<div style="font-size:12.8px">Program terminated with signal
SIGSEGV, Segmentation fault.</div>
<div style="font-size:12.8px">#0 0x00007fdf4f8fd5b3 in ??
() from /usr/lib/x86_64-linux-gnu/libc<wbr>url-gnutls.so.4</div>
<div style="font-size:12.8px">(gdb) bt full</div>
<div style="font-size:12.8px">#0 0x00007fdf4f8fd5b3 in ??
() from /usr/lib/x86_64-linux-gnu/libc<wbr>url-gnutls.so.4</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#1 0x00007fdf4f8d38df in ??
() from /usr/lib/x86_64-linux-gnu/libc<wbr>url-gnutls.so.4</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#2 0x00007fdf4f8d4dff in ??
() from /usr/lib/x86_64-linux-gnu/libc<wbr>url-gnutls.so.4</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#3 0x00007fdf4f8e66db in ??
() from /usr/lib/x86_64-linux-gnu/libc<wbr>url-gnutls.so.4</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#4 0x00007fdf4f8f6047 in ??
() from /usr/lib/x86_64-linux-gnu/libc<wbr>url-gnutls.so.4</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#5 0x00007fdf4f8f6671 in
curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libc<wbr>url-gnutls.so.4</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#6 0x00007fdf4fb2d01d in
resume_async_http_req () from
/usr/lib/x86_64-linux-gnu/open<wbr>sips/modules/rest_client.so</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#7 0x00007fdf511efc49 in
t_resume_async () from /usr/lib/x86_64-linux-gnu/open<wbr>sips/modules/tm.so</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#8 0x0000000000519f7a in ??
()</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#9 0x000000000051bf47 in
tcp_worker_proc ()</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#10 0x0000000000514c88 in
tcp_start_processes ()</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">#11 0x00000000004197da in main
()</div>
<div style="font-size:12.8px">No symbol table info
available.</div>
<div style="font-size:12.8px">(gdb) </div>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px">
<div style="font-size:12.8px">2016-11-19T02:35:25.626123+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8915]: DBG:tm:io_watch_add: [TCP_worker]
io_watch_add op (27 on 6) (0x7e4020, 27, 16,
0x7fdf51b77df0,1), fd_no=3/1024</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.626135+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8915]: DBG:core:destroy_avp_list: destroying list
(nil)</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.626139+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8915]: DBG:core:receive_msg: cleaning up</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.626160+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8915]: DBG:core:tcp_read_req: tcp_read_req end</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.627576+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8915]: DBG:tm:t_resume_async: resuming on fd 27,
transaction 0x7fdf51b72d60 </div>
<div style="font-size:12.8px">2016-11-19T02:35:25.628876+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8915]: DBG:rest_client:resume_async_<wbr>http_req:
running handles: 1</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.672898+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8915]: DBG:tm:t_resume_async: resuming on fd 27,
transaction 0x7fdf51b72d60 </div>
<div style="font-size:12.8px">2016-11-19T02:35:25.828337+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: DBG:core:io_wait_loop_epoll: [TCP_main]
EPOLLHUP on IN ->connection closed by the remote peer!</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.828363+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: CRITICAL:core:receive_fd: EOF on 25</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.828367+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: DBG:core:handle_worker: dead child 12, pid
8915 (shutting down?)</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.828372+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: DBG:core:io_watch_del: [TCP_main]
io_watch_del op on index 12 25 (0x7e4020, 25, 12, 0x0,0x1)
fd_no=29 called</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.828468+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: DBG:core:io_wait_loop_epoll: [TCP_main]
EPOLLHUP on IN ->connection closed by the remote peer!</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.828485+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: DBG:core:handle_tcp_worker: dead tcp
worker 0 (pid 8915, no 0) (shutting down?)</div>
<div style="font-size:12.8px">2016-11-19T02:35:25.828490+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: DBG:core:io_watch_del: [TCP_main]
io_watch_del op on index 20 23 (0x7e4020, 23, 20, 0x0,0x1)
fd_no=28 called</div>
<div style="font-size:12.8px">2016-11-19T02:35:26.140655+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8903]: DBG:core:handle_sigs: status = 139</div>
<div style="font-size:12.8px">2016-11-19T02:35:26.140759+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8903]: <a class="moz-txt-link-freetext" href="INFO:core:handle_sigs">INFO:core:handle_sigs</a>: child process 8915
exited by a signal 11</div>
<div style="font-size:12.8px">2016-11-19T02:35:26.140805+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8903]: <a class="moz-txt-link-freetext" href="INFO:core:handle_sigs">INFO:core:handle_sigs</a>: core was generated</div>
<div style="font-size:12.8px">2016-11-19T02:35:26.140850+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8903]: <a class="moz-txt-link-freetext" href="INFO:core:handle_sigs">INFO:core:handle_sigs</a>: terminating due to
SIGCHLD</div>
<div style="font-size:12.8px">2016-11-19T02:35:26.141012+00:<wbr>00
<a moz-do-not-send="true"
href="http://cc-foosipas1.foo.voip.example.com"
target="_blank">cc-foosipas1.foo.voip.example.<wbr>com</a>
opensips[8924]: <a class="moz-txt-link-freetext" href="INFO:core:sig_usr">INFO:core:sig_usr</a>: signal 15 received</div>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px"><br>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Devel@lists.opensips.org">Devel@lists.opensips.org</a>
<a class="moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/devel">http://lists.opensips.org/cgi-bin/mailman/listinfo/devel</a>
</pre>
</blockquote>
<br>
</body>
</html>