<div dir="ltr"><div>I'm not sure what to look for to match the previous issue so I pasted a couple of crashes from the last two days</div><div><br></div><a href="https://pastebin.com/aAFsgXEm">https://pastebin.com/aAFsgXEm</a><br><div><a href="https://pastebin.com/2Eq8JZ1A">https://pastebin.com/2Eq8JZ1A</a><br></div><div><a href="https://pastebin.com/reDexmqY">https://pastebin.com/reDexmqY</a><br></div><div><br></div><div>Let me know if this helps.</div><div><br></div><div>Thanks,</div><div>Tito</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 11, 2018 at 12:50 AM, Tito Cumpen <span dir="ltr"><<a href="mailto:tito@xsvoce.com" target="_blank">tito@xsvoce.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Razvan,<div>Here's an output of show origin</div><div> </div><div><div>git remote show origin</div><div>* remote origin</div><div> Fetch URL: <a href="https://github.com/OpenSIPS/opensips.git" target="_blank">https://github.com/OpenSIPS/<wbr>opensips.git</a></div><div> Push URL: <a href="https://github.com/OpenSIPS/opensips.git" target="_blank">https://github.com/OpenSIPS/<wbr>opensips.git</a></div><div> HEAD branch: master</div><div> Remote branches:</div><div> 1.10 tracked</div><div> 1.11 tracked</div><div> 1.4 tracked</div><div> 1.5 tracked</div><div> 1.6 tracked</div><div> 1.7 tracked</div><div> 1.8 tracked</div><div> 1.9 tracked</div><div> 2.1 tracked</div><div> 2.2 tracked</div><div> 2.3 tracked</div><div> 3.0-dev tracked</div><div> coverity_scan tracked</div><div> feature/usrloc-kv-contact-<wbr>storage new (next fetch will store in remotes/origin)</div><div> master tracked</div><div> Local branch configured for 'git pull':</div><div> 2.3 merges with remote 2.3</div><div> Local ref configured for 'git push':</div><div> 2.3 pushes to 2.3 (local out of date)</div></div><div><br></div><div><div>git describe --tags</div><div>2.3.2-82-ga3b332c</div></div><div><br></div><div>Does this help? </div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 11, 2018 at 12:19 AM, Răzvan Crainea <span dir="ltr"><<a href="mailto:razvan@opensips.org" target="_blank">razvan@opensips.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<tt>Hi, Tito!<br>
<br>
I noticed that you have posted a different backtrace. Please post
files on pastebin, or somewhere else, to avoid overloading
everybody's mailing box.<br>
Also, I didn't find anywhere the </tt><tt>a3b332c commit - are
you using the official sources?<br>
<br>
Best regards,<br>
</tt><span>
<pre class="m_-5160439995009183529m_6610100584891220169moz-signature" cols="72">Răzvan Crainea
OpenSIPS Developer
<a class="m_-5160439995009183529m_6610100584891220169moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
</span><div><div class="m_-5160439995009183529h5"><div class="m_-5160439995009183529m_6610100584891220169moz-cite-prefix">On 01/09/2018 10:57 AM, Răzvan Crainea
wrote:<br>
</div>
<blockquote type="cite">
<tt>Hi, Tito!<br>
<br>
Can you try to compile with memory debugging (compiling with
QM_MALLOC and DBG_MALOC macros)? You can do that either
manually[1] or by using the menuconfig tool, where you should go
to Compile Flags and enable the two macros, then recompile your
sources.<br>
<br>
[1] <a class="m_-5160439995009183529m_6610100584891220169moz-txt-link-freetext" href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem" target="_blank">http://www.opensips.org/Docume<wbr>ntation/TroubleShooting-OutOfM<wbr>em</a><br>
<br>
Best regards,<br>
</tt>
<pre class="m_-5160439995009183529m_6610100584891220169moz-signature" cols="72">Răzvan Crainea
OpenSIPS Developer
<a class="m_-5160439995009183529m_6610100584891220169moz-txt-link-abbreviated" href="http://www.opensips-solutions.com" target="_blank">www.opensips-solutions.com</a></pre>
<div class="m_-5160439995009183529m_6610100584891220169moz-cite-prefix">On 01/09/2018 12:12 AM, Tito Cumpen
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Group,</div>
<div><br>
</div>
<div>I've migrated to opensips 2.3.2 git revision a3b332c and
I am experiencing a crash </div>
<div><br>
</div>
<div>I am using </div>
<div>OpenSSL 1.0.2k-fips if that helps.</div>
<div>gdb opensips core.11677 </div>
<div>GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7</div>
<div>Copyright (C) 2013 Free Software Foundation, Inc.</div>
<div>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.h<wbr>tml</a>></div>
<div>This is free software: you are free to change and
redistribute it.</div>
<div>There is NO WARRANTY, to the extent permitted by law.
Type "show copying"</div>
<div>and "show warranty" for details.</div>
<div>This GDB was configured as "x86_64-redhat-linux-gnu".</div>
<div>For bug reporting instructions, please see:</div>
<div><<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/g<wbr>db/bugs/</a>>...</div>
<div>Reading symbols from /usr/sbin/opensips...done.</div>
<div>[New LWP 11677]</div>
<div>[Thread debugging using libthread_db enabled]</div>
<div>Using host libthread_db library
"/lib64/libthread_db.so.1".</div>
<div>Core was generated by `/sbin/opensips -P
/var/run/opensips.pid -u opensips -g opensips -M 1024 -m
256'.</div>
<div>Program terminated with signal 11, Segmentation fault.</div>
<div>#0 0x00000000005313b2 in fm_remove_free
(n=0x7f6a1fa7bb50, qm=0x7f6a1f681000) at mem/f_malloc.c:200</div>
<div>200<span style="white-space:pre-wrap"> </span>*pf=n->u.nxt_free;</div>
<div>(gdb) bt full</div>
<div>#0 0x00000000005313b2 in fm_remove_free
(n=0x7f6a1fa7bb50, qm=0x7f6a1f681000) at mem/f_malloc.c:200</div>
<div> pf = 0x0</div>
<div> hash = 10</div>
<div>#1 fm_malloc (qm=0x7f6a1f681000, size=size@entry=80) at
mem/f_malloc.c:428</div>
<div> frag = 0x7f6a1fa7bb50</div>
<div> n = <optimized out></div>
<div> hash = 10</div>
<div> __FUNCTION__ = "fm_malloc"</div>
<div>#2 0x00007f6a1f2670d9 in shm_malloc (size=80) at
../../mem/shm_mem.h:437</div>
<div> p = <optimized out></div>
<div>#3 os_malloc (size=80) at tls.h:85</div>
<div>No locals.</div>
<div>#4 0x00007f6a1ebeb3b8 in CRYPTO_malloc
(num=num@entry=80, file=file@entry=0x7f6a1ed636fb
"pmeth_lib.c", line=line@entry=292) at mem.c:346</div>
<div> ret = 0x0</div>
<div>#5 0x00007f6a1ecb55ec in EVP_PKEY_CTX_dup
(pctx=0x7f6a1fa7bb68) at pmeth_lib.c:292</div>
<div> rctx = <optimized out></div>
<div>#6 0x00007f6a1eca7160 in EVP_MD_CTX_copy_ex
(out=0x7ffc9d7b5560, in=0x7ffc9d7b5660) at digest.c:373</div>
<div> tmp_buf = 0x0</div>
<div>#7 0x00007f6a1ecb6e0c in EVP_DigestSignFinal
(ctx=ctx@entry=0x7ffc9d7b5660,
sigret=sigret@entry=0x7f6a1fb0<wbr>8022 "\256v?p\257\244",
siglen=siglen@entry=0x7ffc9d7b<wbr>5658) at m_sigver.c:153</div>
<div> tmp_ctx = {digest = 0x7f6a1efc5d80
<sha1_md>, engine = 0x0, flags = 256, md_data = 0x0,
pctx = 0x7f6a1fa7bb68, update = 0x7f6a1ec12570
<int_update>}</div>
<div> md =
"\001\000\000\000\000\000\000\<wbr>000\300C\246\037j\177\000\000@<wbr>\342\256\037j\177\000\000\327'<wbr>\301\036j\177\000\000\070\356\<wbr>260\037j\177\000\000h\273\247\<wbr>037j\177\000\000\000\000\000\0<wbr>00\000\000\000\000\213V\313\03<wbr>6j\177\000"</div>
<div> mdlen = 32618</div>
<div> sctx = 1</div>
<div> r = 0</div>
<div> pctx = <optimized out></div>
<div>#8 0x00007f6a1f016a9b in tls1_mac (ssl=0x7f6a1faed638,
md=0x7f6a1fb08022 "\256v?p\257\244", send=1) at
t1_enc.c:1102</div>
<div> rec = 0x7f6a1fafa130</div>
<div> seq = 0x7f6a1fafa030 ""</div>
<div> hash = <optimized out></div>
<div> md_size = 20</div>
<div> orig_len = 22</div>
<div> i = <optimized out></div>
<div> hmac = {digest = 0x7f6a1efc5d80 <sha1_md>,
engine = 0x0, flags = 256, md_data = 0x0, pctx =
0x7f6a1fa7bb68, update = 0x7f6a1ec12570 <int_update>}</div>
<div> mac_ctx = 0x7ffc9d7b5660</div>
<div> header =
"\000\000\000\000\000\000\000I<wbr>\027\003\001\000\002"</div>
<div> stream_mac = 0</div>
<div> t = <optimized out></div>
<div>#9 0x00007f6a1f008bfd in do_ssl3_write
(s=s@entry=0x7f6a1faed638, type=type@entry=23,
buf=buf@entry=0x7f6a15d2508f "\r\n", len=2,
create_empty_fragment=create_e<wbr>mpty_fragment@entry=0) at
s3_pkt.c:1030</div>
<div> p = 0x7f6a1fb08020 "\r\n\256v?p\257\244"</div>
<div> plen = 0x7f6a1fb0801e ""</div>
<div> i = <optimized out></div>
<div> mac_size = 20</div>
<div> clear = <optimized out></div>
<div> prefix_len = <optimized out></div>
<div> eivlen = <optimized out></div>
<div> align = <optimized out></div>
<div> wr = 0x7f6a1fafa130</div>
<div> wb = 0x7f6a1fafa0e0</div>
<div> sess = <optimized out></div>
<div>#10 0x00007f6a1f008e36 in ssl3_write_bytes
(s=0x7f6a1faed638, type=<optimized out>,
buf_=0x7f6a15d2508f, len=<optimized out>) at
s3_pkt.c:833</div>
<div> buf = 0x7f6a15d2508f "\r\n"</div>
<div> tot = 0</div>
<div> n = 2</div>
<div> nw = <optimized out></div>
<div> max_send_fragment = <optimized out></div>
<div> wb = 0x7f6a1fafa0e0</div>
<div> i = <optimized out></div>
<div>#11 0x00007f6a15d20f69 in tls_write (fd=<optimized
out>, poll_events=0x7ffc9d7b5844, len=2,
buf=0x7f6a15d2508f, c=0x7f6a1faee0b8) at
../tls_mgm/tls_conn_server.h:5<wbr>04</div>
<div> ret = <optimized out></div>
<div> err = <optimized out></div>
<div>---Type <return> to continue, or q <return>
to quit---</div>
<div> ssl = 0x7f6a1faed638</div>
<div>#12 tls_blocking_write (c=c@entry=0x7f6a1faee0b8,
fd=<optimized out>, buf=buf@entry=0x7f6a15d2508f
"\r\n", len=len@entry=2, api=0x7f6a15f397e0
<tls_mgm_api>, api=0x7f6a15f397e0 <tls_mgm_api>)</div>
<div> at ../tls_mgm/tls_conn_server.h:5<wbr>78</div>
<div> written = 0</div>
<div> n = 0</div>
<div> timeout = <optimized out></div>
<div> retries = 0</div>
<div> pf = {fd = 11, events = 0, revents = 0}</div>
<div>#13 0x00007f6a15d241bc in w_tls_blocking_write
(buf=0x7f6a15d2508f "\r\n", len=2, fd=<optimized out>,
c=0x7f6a1faee0b8) at proto_tls.c:124</div>
<div> ret = <optimized out></div>
<div>#14 tcp_handle_req (_max_msg_chunks=<optimized
out>, con=<optimized out>, req=0x7f6a2f71b6d8) at
../../net/proto_tcp/tcp_common<wbr>.h:386</div>
<div> local_rcv = {src_ip = {af = 2, len = 4, u =
{addrl = {3070, 0}, addr32 = {3070, 0, 0, 0}, addr16 =
{3070, 0, 0, 0, 0, 0, 0, 0}, addr = "\376\v", '\000'
<repeats 13 times>}}, dst_ip = {af = 2, </div>
<div> len = 4, u = {addrl = {4, 0}, addr32 = {4, 0,
0, 0}, addr16 = {4, 0, 0, 0, 0, 0, 0, 0}, addr = "\004",
'\000' <repeats 14 times>}}, src_port = 0, dst_port =
0, proto = 0, </div>
<div> proto_reserved1 = <a href="tel:(208)%20732-6500" value="+12087326500" target="_blank">2087326500</a>, proto_reserved2 =
0, src_su = {s = {sa_family = 2, sa_data = "\333Yl#
\374\200\260\b\267\000\000\000<wbr>"}, sin = {sin_family = 2,
sin_port = 23003, sin_addr = {</div>
<div> s_addr = 4229964652}, sin_zero =
"\200\260\b\267\000\000\000"}, sin6 = {sin6_family = 2,
sin6_port = 23003, sin6_flowinfo = 4229964652, sin6_addr =
{__in6_u = {</div>
<div> __u6_addr8 =
"\200\260\b\267\000\000\000\00<wbr>0 \000\000\000\000\000\000",
__u6_addr16 = {45184, 46856, 0, 0, 32, 0, 0, 0}, __u6_addr32
= {3070800000, 0, 32, 0}}}, </div>
<div> sin6_scope_id = 4294967291}}, bind_address
= 0x1}</div>
<div> msg_buf = <optimized out></div>
<div> msg_len = <optimized out></div>
<div> c = 0 '\000'</div>
<div> size = 0</div>
<div>#15 tls_read_req (con=0x7f6a1faee0b8,
bytes_read=0x7ffc9d7b5a20) at proto_tls.c:616</div>
<div> ret = 0</div>
<div> bytes = <optimized out></div>
<div> total_bytes = 3</div>
<div> req = 0x7f6a2f71b6d8</div>
<div> data = <optimized out></div>
<div> __FUNCTION__ = "tls_read_req"</div>
<div>#16 0x00000000005e4ba7 in handle_io (event_type=1, idx=1,
fm=<optimized out>) at net/net_tcp_proc.c:231</div>
<div> con = 0x7f6a1faee0b8</div>
<div> ret = 0</div>
<div> n = <optimized out></div>
<div> rw = <optimized out></div>
<div> response = {140093774569184, 1}</div>
<div> s = 13</div>
<div> resp = <optimized out></div>
<div>#17 io_wait_loop_epoll (repeat=<optimized out>,
t=<optimized out>, h=<optimized out>) at
net/../io_wait_loop.h:280</div>
<div> ret = 1</div>
<div> n = 1</div>
<div> r = 1</div>
<div> i = <optimized out></div>
<div> e = <optimized out></div>
<div> ep_event = {events = 565320, data = {ptr =
0x531afb00000000, fd = 0, u32 = 0, u64 = 23392088406097920}}</div>
<div> fd = <optimized out></div>
<div>#18 0x00000000005e665b in tcp_worker_proc_loop () at
net/net_tcp_proc.c:359</div>
<div> __FUNCTION__ = "tcp_worker_proc_loop"</div>
<div>#19 0x00000000005f31ad in tcp_start_processes
(chd_rank=chd_rank@entry=0x89f<wbr>3e0 <chd_rank.11332>,
startup_done=startup_done@entr<wbr>y=0x7f6a1fa4e870) at
net/net_tcp.c:1837</div>
<div> r = 0</div>
<div> reader_fd = {90, 93}</div>
<div> pid = <optimized out></div>
<div> load_p = 0x7f6a1fa4f638</div>
<div> __FUNCTION__ = "tcp_start_processes"</div>
<div>#20 0x000000000041c704 in main_loop () at main.c:683</div>
<div> startup_done = 0x7f6a1fa4e870</div>
<div> chd_rank = 17</div>
<div>#21 main (argc=<optimized out>, argv=<optimized
out>) at main.c:1283</div>
<div> cfg_stream = <optimized out></div>
<div> c = <optimized out></div>
<div> r = <optimized out></div>
<div> tmp = 0x7ffc9d7b6f65 ""</div>
<div> tmp_len = <optimized out></div>
<div>---Type <return> to continue, or q <return>
to quit---</div>
<div> port = <optimized out></div>
<div> proto = <optimized out></div>
<div> protos_no = <optimized out></div>
<div> options = 0x632f18
"f:cCm:M:b:l:n:N:rRvdDFETSVhw:<wbr>t:u:g:P:G:W:o:"</div>
<div> ret = -1</div>
<div> seed = <a href="tel:(270)%20646-4427" value="+12706464427" target="_blank">2706464427</a></div>
<div> rfd = <optimized out></div>
<div> __FUNCTION__ = "main"</div>
<div><br>
</div>
</div>
<br>
<fieldset class="m_-5160439995009183529m_6610100584891220169mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Users mailing list
<a class="m_-5160439995009183529m_6610100584891220169moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="m_-5160439995009183529m_6610100584891220169moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="m_-5160439995009183529m_6610100584891220169mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Users mailing list
<a class="m_-5160439995009183529m_6610100584891220169moz-txt-link-abbreviated" href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a class="m_-5160439995009183529m_6610100584891220169moz-txt-link-freetext" href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a><br>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.opensips.org/cgi-<wbr>bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>