<div dir="ltr">On a related note, however, is this of any concern:<div><br></div><div><div><font face="courier new, monospace">May 27 15:35:02 b2b-dev1 /usr/sbin/opensips[19455]: CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here</font></div>
</div><div><br></div><div style>Or for that matter, any of these:</div><div style><br></div><div style><div><font face="courier new, monospace">May 27 15:35:22 b2b-dev1 /usr/sbin/opensips[19455]: WARNING:b2b_logic:b2bl_delete_entity: entity [0x7f8840c171d8]->[] not found for tuple [383.0]</font></div>
<div><div><font face="courier new, monospace">May 27 15:35:19 b2b-dev1 /usr/sbin/opensips[19458]: ERROR:b2b_logic:process_bridge_negreply: unexpected entity_no [2] for tuple [0x7f8840c19378]</font></div><div><font face="courier new, monospace">May 27 15:35:19 b2b-dev1 /usr/sbin/opensips[19458]: ERROR:b2b_logic:b2b_logic_notify_reply: Failed to process negative reply while in bridging state</font></div>
</div><div><br></div></div><div><br></div><div>--</div><div style>Tolga</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 27, 2013 at 3:32 PM, Tolga Tarhan <span dir="ltr"><<a href="mailto:tolga@netbrains.com" target="_blank">tolga@netbrains.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Bogdan,<br><div><br></div><div>We've tried out the patch and it seems to resolve the issue. We will continue to test to see if we can reproduce the problem, but for now, it appears fixed.</div>
<div>
<br></div><div>Thanks so much for your help on this.</div><div><br></div><div>--</div><div>Tolga</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 27, 2013 at 2:31 PM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>Hi Tolga,</div><div><br></div><div>Yes the patch is correct, please give it a try.</div><div><br></div><div>
Thanks and regards,</div>
<div>Bogdan</div><div><br></div><div><br></div><div><div style="font-size:75%;color:#575757">Sent from Samsung Mobile</div></div><div><div> <br><br><br>-------- Original message --------<br>From: Tolga Tarhan <<a href="mailto:tolga@netbrains.com" target="_blank">tolga@netbrains.com</a>> <br>
Date: <br>To: Bogdan-Andrei Iancu <<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>> <br>Cc: OpenSIPS users mailling list <<a href="mailto:users@lists.opensips.org" target="_blank">users@lists.opensips.org</a>> <br>
Subject: Re: [OpenSIPS-Users] B2BUA Segfault <br> <br><br><div dir="ltr">Bogdan,<div><br></div><div>The patch doesn't build:</div><div><br></div><div><div><font face="courier new, monospace">Compiling logic.c</font></div>
<div><font face="courier new, monospace">logic.c: In function 'process_bridge_200OK':</font></div>
<div><font face="courier new, monospace">logic.c:741: error: invalid operands to binary == (have 'char *' and 'str')</font></div><div><font face="courier new, monospace">logic.c:742: error: incompatible types when assigning to type 'str' from type 'int'</font></div>
<div><font face="courier new, monospace">make[1]: *** [logic.o] Error 1</font></div><div><br></div></div><div>I've attached what I believe is a corrected patch. Let me know if this is what you intended.</div><div>
<br></div><div>I'm trying the (corrected) patch now to see if it resolves the original issue. I'll let you know on my findings shortly.</div><div><br></div><div>Thanks,</div><div>Tolga</div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 27, 2013 at 6:27 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div text="#000000" bgcolor="#ffffff">
<tt>Hello Tolga,<br>
<br>
Please test the attached patch.<br>
<br>
Regards,<br>
</tt><div>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
<br></div><div><div>
On 05/24/2013 08:33 PM, Tolga Tarhan wrote:
<blockquote type="cite">
<div dir="ltr">
<div>Sure, here you go:</div>
<div><br>
</div>
<div>version: opensips 1.9.0-tls (x86_64/linux)</div>
<div>flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS,
DISABLE_NAGLE, USE_MCAST, SHM_MEM, SHM_MMAP, PKG_MALLOC,
DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT</div>
<div>ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535</div>
<div>poll method support: poll, epoll_lt, epoll_et, sigio_rt,
select.</div>
<div>svnrevision: unknown</div>
<div>
@(#) $Id: main.c 9790 2013-02-15 10:14:34Z bogdan_iancu $</div>
<div>main.c compiled on 12:39:26 May 22 2013 with gcc 4.4.7</div>
<div><br>
</div>
<div>It also has the couple patches that
I've previously submitted (and you've since merged) in the
build (the three related to GRUU bugs). They aren't a factor,
as this particular instance is not acting as a registrar at
all.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tolga</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, May 24, 2013 at 9:33 AM,
Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> <tt>Hi Tolga,<br>
<br>
Thanks for the info.<br>
<br>
What exact OpenSIPs version/revision are you using ? I
need to correlate logs with sources.<br>
<br>
Regards,<br>
</tt>
<div>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
<br>
</div>
<div>
<div> On 05/22/2013 11:02 PM, Tolga Tarhan
wrote:
<blockquote type="cite">
<div dir="ltr">Sorry for the self-reply -- here's
the (same) stacktraces with line numbers and
params:
<div><br>
</div>
<div>
<div><font face="courier new, monospace">#0
0x0000003564c328a5 in raise () from
/lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#1
0x0000003564c34085 in abort () from
/lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#2
0x000000000049d370 in qm_free (qm=<value
optimized out>, p=<value optimized
out>, file=0x7ffccb25df64 "logic.c",
func=<value optimized out>, line=755)
at mem/q_malloc.c:450</font></div>
<div><font face="courier new, monospace">#3
0x00007ffccb253804 in process_bridge_200OK
(msg=0x7ffcccdd2600, extra_headers=0xd5,
body=<value optimized out>,
tuple=0x7ffcc9518de0, entity=<value
optimized out>) at logic.c:755</font></div>
<div><font face="courier new, monospace">#4
0x00007ffccb254ba2 in
b2b_logic_notify_reply (src=<value
optimized out>, msg=0x7ffcccdd2600,
key=<value optimized out>,
body=0x7fffc2ce15d0,
extra_headers=0x7fffc2ce15c0,
b2bl_key=0x7fffc2ce23f0, hash_index=649,
local_index=1)</font></div>
<div><font face="courier new, monospace"> at
logic.c:1133</font></div>
<div><font face="courier new, monospace">#5
0x00007ffccb2565e1 in b2b_logic_notify
(src=1, msg=0x7ffcccdd2600,
key=0x7ffcc9525e40, type=1,
param=0x7fffc2ce23f0) at logic.c:2040</font></div>
<div><font face="courier new, monospace">#6
0x00007ffccb47a7a7 in b2b_tm_cback
(t=0x7ffcc951c110, htable=0x7ffcc94f38d0,
ps=<value optimized out>) at
dlg.c:2678</font></div>
<div><font face="courier new, monospace">#7
0x00007ffccc744b71 in run_trans_callbacks
(type=256, trans=0x7ffcc951c110,
req=<value optimized out>,
rpl=<value optimized out>,
code=<value optimized out>) at
t_hooks.c:212</font></div>
<div><font face="courier new, monospace">#8
0x00007ffccc74fa12 in local_reply
(t=0x7ffcc951c110, p_msg=<value optimized
out>, branch=<value optimized out>,
msg_status=<value optimized out>,
cancel_bitmap=<value optimized out>)
at t_reply.c:1391</font></div>
<div><font face="courier new, monospace">#9
0x00007ffccc750cc5 in reply_received
(p_msg=0x7ffcccdd2600) at t_reply.c:1540</font></div>
<div><font face="courier new, monospace">#10
0x00000000004266da in forward_reply
(msg=0x7ffcccdd2600) at forward.c:574</font></div>
<div><font face="courier new, monospace">#11
0x0000000000452ad8 in receive_msg
(buf=<value optimized out>,
len=<value optimized out>,
rcv_info=0x7fffc2ce28c0) at receive.c:207</font></div>
<div><font face="courier new, monospace">#12
0x0000000000496c95 in udp_rcv_loop () at
udp_server.c:424</font></div>
<div><font face="courier new, monospace">#13
0x000000000042d763 in main_loop
(argc=<value optimized out>,
argv=<value optimized out>) at
main.c:884</font></div>
<div><font face="courier new, monospace">#14
main (argc=<value optimized out>,
argv=<value optimized out>) at
main.c:1557</font></div>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div><font face="courier new, monospace">#0
0x0000003564c328a5 in raise () from
/lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#1
0x0000003564c34085 in abort () from
/lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#2
0x000000000049d370 in qm_free (qm=<value
optimized out>, p=<value optimized
out>, file=0x7ffccb262d75 "records.c",
func=<value optimized out>, line=595)
at mem/q_malloc.c:450</font></div>
<div><font face="courier new, monospace">#3
0x00007ffccb25a003 in b2bl_delete
(tuple=0x7ffcc9518de0, hash_index=<value
optimized out>, not_del_b2be=1) at
records.c:595</font></div>
<div><font face="courier new, monospace">#4
0x00007ffccb25a3d5 in destroy_b2bl_htable
() at records.c:705</font></div>
<div><font face="courier new, monospace">#5
0x000000000046dbf2 in destroy_modules () at
sr_module.c:371</font></div>
<div><font face="courier new, monospace">#6
0x00000000004298a1 in cleanup
(show_status=1) at main.c:348</font></div>
<div><font face="courier new, monospace">#7
0x000000000042a360 in handle_sigs () at
main.c:549</font></div>
<div><font face="courier new, monospace">#8
0x000000000042db66 in main_loop
(argc=<value optimized out>,
argv=<value optimized out>) at
main.c:1024</font></div>
<div><font face="courier new, monospace">#9
main (argc=<value optimized out>,
argv=<value optimized out>) at
main.c:1557</font></div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tolga</div>
</div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On Wed, May 22, 2013 at
1:00 PM, Tolga Tarhan <span dir="ltr"><<a href="mailto:tolga@netbrains.com" target="_blank">tolga@netbrains.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Thank you -- I've recompiled
and enabled the memory debug. I have the log
file from the whole experience available
here:
<div><br>
</div>
<div><a href="http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log" target="_blank">http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log</a><br>
</div>
<div><br>
</div>
<div>(note - real phone numbers and domain
names in the log have been replaced with
placeholders)</div>
<div><br>
</div>
<div>The key item seems to be:</div>
<div><br>
</div>
<div>CRITICAL:core:qm_free: freeing already
freed pointer, first free: logic.c:
process_bridge_200OK(740) - aborting<br>
</div>
<div><br>
</div>
<div>Although this appears to be after we've
already decided we're going to crash, as I
see "INFO:core:cleanup: cleanup" and
"NFO:core:handle_sigs: child process 24788
exited by a signal 6" above this part of
the log.</div>
<div><br>
</div>
<div>Also worth noting is the existance of
"CRITICAL:b2b_logic:b2bl_drop_entity: we
should never end up here" throughout the
log.</div>
<div>
<div><br>
</div>
<div>Also, here's the stack trace at crash
time. Note that there were two core
files generated for the same crash, so
this is the backtrace from each:</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">#0
0x0000003564c328a5 in raise () from
/lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#1
0x0000003564c34085 in abort () from
/lib64/libc.so.6</font></div>
<div><font face="courier new, monospace">#2
0x000000000049d370 in qm_free ()</font></div>
<div><font face="courier new, monospace">#3
0x00007ffccb253804 in
process_bridge_200OK () from
/usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new, monospace">#4
0x00007ffccb254ba2 in
b2b_logic_notify_reply () from
/usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new, monospace">#5
0x00007ffccb2565e1 in
b2b_logic_notify () from
/usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new, monospace">#6
0x00007ffccb47a7a7 in b2b_tm_cback
() from
/usr/lib64/opensips/modules/b2b_entities.so</font></div>
<div><font face="courier new, monospace">#7
0x00007ffccc744b71 in
run_trans_callbacks () from
/usr/lib64/opensips/modules/tm.so</font></div>
<div><font face="courier new, monospace">#8
0x00007ffccc74fa12 in local_reply
() from
/usr/lib64/opensips/modules/tm.so</font></div>
<div><font face="courier new, monospace">#9
0x00007ffccc750cc5 in
reply_received () from
/usr/lib64/opensips/modules/tm.so</font></div>
<div><font face="courier new, monospace">#10
0x00000000004266da in forward_reply
()</font></div>
<div><font face="courier new, monospace">#11
0x0000000000452ad8 in receive_msg ()</font></div>
<div><font face="courier new, monospace">#12
0x0000000000496c95 in udp_rcv_loop
()</font></div>
<div><font face="courier new, monospace">#13
0x000000000042d763 in main ()</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div><font face="courier new,
monospace">#0 0x0000003564c328a5
in raise () from /lib64/libc.so.6</font></div>
<div><font face="courier new,
monospace">#1 0x0000003564c34085
in abort () from /lib64/libc.so.6</font></div>
<div><font face="courier new,
monospace">#2 0x000000000049d370
in qm_free ()</font></div>
<div><font face="courier new,
monospace">#3 0x00007ffccb25a003
in b2bl_delete () from
/usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new,
monospace">#4 0x00007ffccb25a3d5
in destroy_b2bl_htable () from
/usr/lib64/opensips/modules/b2b_logic.so</font></div>
<div><font face="courier new,
monospace">#5 0x000000000046dbf2
in destroy_modules ()</font></div>
<div><font face="courier new,
monospace">#6 0x00000000004298a1
in cleanup ()</font></div>
<div><font face="courier new,
monospace">#7 0x000000000042a360
in handle_sigs ()</font></div>
<div><font face="courier new,
monospace">#8 0x000000000042db66
in main ()</font></div>
</div>
<div><br>
</div>
<div>I am unfamiliar with this codebase.
Can anyone garner anything useful from
the logs?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tolga</div>
<div><br>
</div>
</div>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, May 22,
2013 at 9:02 AM, Bogdan-Andrei Iancu <span dir="ltr"><<a href="mailto:bogdan@opensips.org" target="_blank">bogdan@opensips.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> <tt>Hello
Tolga,<br>
<br>
The crash seems to be in the
memory manager, most probably
because of memory corruption. To
troubleshoot such issues you
need to compile-in the memory
debugger - see <a href="http://www.opensips.org/Documentation/TroubleShooting-OutOfMem" target="_blank">http://www.opensips.org/Documentation/TroubleShooting-OutOfMem</a>
.<br>
<br>
Using memlog=1 + memdump=10 you
should get a lot of logs related
to mem ops, including a final
report + abort() when the
corruption is detected.<br>
<br>
Regards,<br>
</tt>
<pre cols="72">Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
<a href="http://www.opensips-solutions.com" target="_blank">http://www.opensips-solutions.com</a></pre>
<div>
<div> <br>
On 05/22/2013 01:29 AM, Tolga
Tarhan wrote: </div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>While using the B2BUA
module in
OpenSIPS 1.9.0,
we've encountered a
consistent segfault. We
are using a refer
scenario just like the
one in the B2BUA sample
docs, and after several
REFERs for the same call
(to different
destinations), OpenSIPS
crashes with a segfault.
The core file indicates
the following backtrace:</div>
<div><br>
</div>
<div>
<div><font face="courier
new, monospace">#0
0x000000000049a334
in fm_malloc ()</font></div>
<div><font face="courier
new, monospace">#1
0x00007fdaecd96230
in shm_malloc_unsafe
(type=B2B_CLIENT,
entity_id=0x7fdaee8ec750,
to_uri=0x7fff2d346360,
from_uri=0x7fff2d346320,
from_dname=0x0,
ssid=<value
optimized out>,
msg=0x0) at
../../mem/shm_mem.h:248</font></div>
<div><font face="courier
new, monospace">#2
shm_malloc
(type=B2B_CLIENT,
entity_id=0x7fdaee8ec750,
to_uri=0x7fff2d346360,
from_uri=0x7fff2d346320,
from_dname=0x0,
ssid=<value
optimized out>,
msg=0x0) at
../../mem/shm_mem.h:258</font></div>
<div><font face="courier
new, monospace">#3
b2bl_create_new_entity
(type=B2B_CLIENT,
entity_id=0x7fdaee8ec750,
to_uri=0x7fff2d346360,
from_uri=0x7fff2d346320,
from_dname=0x0,
ssid=<value
optimized out>,
msg=0x0) at
logic.c:293</font></div>
<div><font face="courier
new, monospace">#4
0x00007fdaecd96882
in b2bl_new_client
(to_uri=<value
optimized out>,
from_uri=<value
optimized out>,
tuple=<value
optimized out>,
ssid=0x7fdaeb026c00,
msg=<value
optimized out>)
at logic.c:607</font></div>
<div><font face="courier
new, monospace">#5
0x00007fdaecda3579
in
process_bridge_200OK
(msg=0x7fdaee8e8b30,
extra_headers=0x7fdaeb03d578,
body=<value
optimized out>,
tuple=0x7fdaeb01ada8,
entity=<value
optimized out>)
at logic.c:816</font></div>
<div><font face="courier
new, monospace">#6
0x00007fdaecda46c2
in
b2b_logic_notify_reply
(src=<value
optimized out>,
msg=0x7fdaee8e8b30,
key=<value
optimized out>,
body=0x7fff2d3468b0,
extra_headers=0x7fff2d3468a0,
b2bl_key=0x7fff2d3476d0,
hash_index=649,
local_index=0)</font></div>
<div><font face="courier
new, monospace">
at logic.c:1133</font></div>
<div><font face="courier
new, monospace">#7
0x00007fdaecda6081
in b2b_logic_notify
(src=1,
msg=0x7fdaee8e8b30,
key=0x7fdaeb03d500,
type=1,
param=0x7fff2d3476d0)
at logic.c:2040</font></div>
<div><font face="courier
new, monospace">#8
0x00007fdaecfca7ad
in b2b_tm_cback
(t=0x7fdaeb054118,
htable=0x7fdaeb014630,
ps=<value
optimized out>)
at dlg.c:2678</font></div>
<div><font face="courier
new, monospace">#9
0x00007fdaee291441
in
run_trans_callbacks
(type=256,
trans=0x7fdaeb054118,
req=<value
optimized out>,
rpl=<value
optimized out>,
code=<value
optimized out>)
at t_hooks.c:212</font></div>
<div><font face="courier
new, monospace">#10
0x00007fdaee29c0e2
in local_reply
(t=0x7fdaeb054118,
p_msg=<value
optimized out>,
branch=<value
optimized out>,
msg_status=<value
optimized out>,
cancel_bitmap=<value
optimized out>)
at t_reply.c:1391</font></div>
<div><font face="courier
new, monospace">#11
0x00007fdaee29d31d
in reply_received
(p_msg=0x7fdaee8e8b30)
at t_reply.c:1540</font></div>
<div><font face="courier
new, monospace">#12
0x000000000042625a
in forward_reply ()</font></div>
<div><font face="courier
new, monospace">#13
0x0000000000451c28
in receive_msg ()</font></div>
<div><font face="courier
new, monospace">#14
0x0000000000494e45
in udp_rcv_loop ()</font></div>
<div><font face="courier
new, monospace">#15
0x000000000042d1a3
in main ()</font></div>
<div><br>
</div>
<div>I'm not really sure
how to diagnose this
one. Any
hints/fixes/suggestions
would be very
appreciated.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Tolga</div>
</div>
</div>
</div>
</div>
<pre><fieldset></fieldset>
_______________________________________________
Users mailing list
<a href="mailto:Users@lists.opensips.org" target="_blank">Users@lists.opensips.org</a>
<a href="http://lists.opensips.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.opensips.org/cgi-bin/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div></div></div>
</blockquote></div><br></div>
</div></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>