[OpenSIPS-Users] sched_yield()
Andrei Dragus
adragus at opensips.org
Fri Jan 22 12:16:37 CET 2010
Hi,
The new f_malloc will not do anything extra when compared to the old one
until memory usage goes way up.
I've added a warning in mem/f_malloc.c so you can see when defrag
starts. If you get this warning then it is clear that the problem is
from high memory usage.
1 GB for 4k calls seems a lot ( 250k per call). You can try to use
"opensipsctl fifo get_statistics shmem:" and see what the memory usage
is for diferent number of concurrent calls ( 1k,2k,3k,4k), and if indeed
the memory usage is that high we should investigate the cause.
Alex Massover wrote:
> Hi,
>
> Now shared memory is 1G (-m 1024), and all memory is dedicated to the virtual machine (it was shared till now).
> But it still happens, just not so often.
>
> I originate the calls for this stress test in Asterisk with the same resources and looks like Asterisk performs much better than OpenSIPS. How can it be?
> In my stress OpenSIPS does no blocking/slow requests. And it's just 4K concurrent calls, each one is 2-3 min.
>
> Maybe OpenSIPS does too much low level memory management and virtual machine is not suitable for it (despite that Asterisk runs well over VMware)?
>
> I'm not sure but I have a feeling that 1.4 performed better. What can cause performance degradation in 1.6? Storing vars on dialog, new malloc()?
>
> gdb) bt
> #0 0xb78ad424 in __kernel_vsyscall ()
> #1 0xb77e841c in sched_yield () from /lib/i686/cmov/libc.so.6
> #2 0x080bf23d in new_avp ()
> #3 0x080bf53f in add_avp ()
> #4 0x08080e6e in pv_set_avp ()
> #5 0x0808229c in pv_set_value ()
> #6 0x08053c9d in do_assign ()
> #7 0x0805447a in do_action ()
> #8 0x08053ebf in run_action_list ()
> #9 0x08056e7a in do_action ()
> #10 0x08053ebf in run_action_list ()
> #11 0x08056e7a in do_action ()
> #12 0x08053ebf in run_action_list ()
> #13 0x080569d8 in do_action ()
> #14 0x08053ebf in run_action_list ()
> #15 0x08056e7a in do_action ()
> #16 0x08053ebf in run_action_list ()
> #17 0x08057d99 in run_top_route ()
> #18 0x0808ad6c in receive_msg ()
> #19 0x080bd2f2 in udp_rcv_loop ()
> #20 0x08069339 in main ()
> (gdb) quit
>
>
> (gdb) bt
> #0 0xb78ad424 in __kernel_vsyscall ()
> #1 0xb77e841c in sched_yield () from /lib/i686/cmov/libc.so.6
> #2 0xb76f52cd in build_cell () from /usr/lib/opensips/modules/tm.so
> #3 0xb770ac4a in t_newtran () from /usr/lib/opensips/modules/tm.so
> #4 0xb76ff7b8 in t_relay_to () from /usr/lib/opensips/modules/tm.so
> #5 0xb770c501 in ?? () from /usr/lib/opensips/modules/tm.so
> #6 0x08055030 in do_action ()
> #7 0x08053ebf in run_action_list ()
> #8 0x08095cf2 in eval_expr ()
> #9 0x080958d9 in eval_expr ()
> #10 0x08095919 in eval_expr ()
> #11 0x080554e2 in do_action ()
> #12 0x08053ebf in run_action_list ()
> #13 0x080569d8 in do_action ()
> #14 0x08053ebf in run_action_list ()
> #15 0x08056e7a in do_action ()
> #16 0x08053ebf in run_action_list ()
> #17 0x08057d99 in run_top_route ()
> #18 0x0808ad6c in receive_msg ()
> #19 0x080bd2f2 in udp_rcv_loop ()
> #20 0x08069339 in main ()
>
>
> --
> Best Regards,
> Alex Massover
> VoIP R&D TL
> Jajah Inc.
>
>
>> -----Original Message-----
>> From: users-bounces at lists.opensips.org [mailto:users-
>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>> Sent: Thursday, January 21, 2010 3:43 PM
>> To: OpenSIPS users mailling list
>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>
>> My guess is that there is not enough shared memory. When an allocation
>> failes OpenSIPS tries to defragment memory to make room which takes a
>> lot of time and must be done under lock.
>>
>> Please try to increase the shared memory size and tell me if it
>> persists.
>>
>>
>> Alex Massover wrote:
>>
>>> Hi!
>>>
>>> Yes, with -DF_MALLOC.
>>>
>>> 1.6.1 from sources, I build deb package.
>>> I use 128M of shared and 10*1024*1024 private memory (can increase -
>>>
>> no problem).
>>
>>> Hmmmm, "opensipsctl fifo get_statistics all" crashes/stops the
>>>
>> opensips.
>>
>>> 'fifo uptime' or 'fifo debug' are OK.
>>>
>>> strace while 'fifo get_statistics all':
>>> Process 9509 attached - interrupt to quit
>>> pause() = ? ERESTARTNOHAND (To be
>>>
>> restarted)
>>
>>> --- SIGUSR2 (User defined signal 2) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> pause() = ? ERESTARTNOHAND (To be
>>>
>> restarted)
>>
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> waitpid(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGUSR2}], WNOHANG) =
>>>
>> 9520
>>
>>> waitpid(-1, 0xbf84b4c8, WNOHANG) = 0
>>> kill(0, SIGTERM) = 0
>>> --- SIGTERM (Terminated) @ 0 (0) ---
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [TERM])
>>> sigreturn() = ? (mask now [])
>>> rt_sigaction(SIGALRM, {0x8065920, [ALRM], SA_RESTART}, {SIG_DFL}, 8)
>>>
>> = 0
>>
>>> alarm(60) = 0
>>> wait4(-1, NULL, 0, NULL) = 9514
>>> wait4(-1, NULL, 0, NULL) = 9519
>>> wait4(-1, NULL, 0, NULL) = 9521
>>> wait4(-1, NULL, 0, NULL) = 9522
>>> wait4(-1, NULL, 0, NULL) = 9512
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> wait4(-1, NULL, 0, NULL) = 9510
>>> wait4(-1, NULL, 0, NULL) = 9516
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> wait4(-1, NULL, 0, NULL) = 9515
>>> wait4(-1, NULL, 0, NULL) = 9517
>>> wait4(-1, NULL, 0, NULL) = 9524
>>> wait4(-1, NULL, 0, NULL) = 9525
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>> sigreturn() = ? (mask now [])
>>> wait4(-1, NULL, 0, NULL) = 9511
>>> wait4(-1, NULL, 0, NULL) = 9513
>>> wait4(-1, NULL, 0, NULL) = 9518
>>> wait4(-1, NULL, 0, NULL) = 9523
>>> wait4(-1, NULL, 0, NULL) = -1 ECHILD (No child
>>>
>> processes)
>>
>>> rt_sigaction(SIGALRM, {0x8066080, [ALRM], SA_RESTART}, {0x8065920,
>>>
>> [ALRM], SA_RESTART}, 8) = 0
>>
>>> stat64("/tmp/opensips_fifo", {st_mode=S_IFIFO|0660, st_size=0, ...})
>>>
>> = 0
>>
>>> unlink("/tmp/opensips_fifo") = 0
>>> munmap(0xaed25000, 134217728) = 0
>>> unlink("/var/run/opensips/opensips.pid") = 0
>>> alarm(0) = 60
>>> rt_sigaction(SIGALRM, {SIG_IGN}, {0x8066080, [ALRM], SA_RESTART}, 8)
>>>
>> = 0
>>
>>> exit_group(0) = ?
>>> Process 9509 detached
>>>
>>> --
>>> Best Regards,
>>> Alex Massover
>>> VoIP R&D TL
>>> Jajah Inc.
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>>>> Sent: Thursday, January 21, 2010 3:09 PM
>>>> To: OpenSIPS users mailling list
>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>
>>>>
>>>> Hi,
>>>>
>>>> Since all the backtraces are in allocation routines my guess is that
>>>> the
>>>> shared memory lock might be causing a problem.
>>>>
>>>> Are you compiling with -DF_MALLOC?
>>>> What version of OpenSIPS are you using?
>>>> What is the total shared memory pool you are allocating?
>>>> What amount of memory are you using? ( Use : opensipsctl fifo
>>>> get_statistics all )
>>>>
>>>> Alex Massover wrote:
>>>>
>>>>
>>>>> Some more,
>>>>>
>>>>> (gdb) bt
>>>>> #0 0xb78dc424 in __kernel_vsyscall ()
>>>>> #1 0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
>>>>> #2 0xb73d77fd in build_new_dlg () from
>>>>>
>>>>>
>>>> /usr/lib/opensips/modules/dialog.so
>>>>
>>>>
>>>>> #3 0xb73d4b81 in dlg_create_dialog () from
>>>>>
>>>>>
>>>> /usr/lib/opensips/modules/dialog.so
>>>>
>>>>
>>>>> #4 0xb73c9c9e in ?? () from /usr/lib/opensips/modules/dialog.so
>>>>> #5 0x08055030 in do_action ()
>>>>> #6 0x08053ebf in run_action_list ()
>>>>> #7 0x08056e7a in do_action ()
>>>>> #8 0x08053ebf in run_action_list ()
>>>>> #9 0x08057d99 in run_top_route ()
>>>>> #10 0x0808ad6c in receive_msg ()
>>>>> #11 0x080bd2f2 in udp_rcv_loop ()
>>>>> #12 0x08069339 in main ()
>>>>>
>>>>>
>>>>> (gdb) bt
>>>>> #0 0xb78dc424 in __kernel_vsyscall ()
>>>>> #1 0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
>>>>> #2 0xb77242cd in build_cell () from
>>>>>
>> /usr/lib/opensips/modules/tm.so
>>
>>>>> #3 0xb7739c4a in t_newtran () from /usr/lib/opensips/modules/tm.so
>>>>> #4 0xb772e7b8 in t_relay_to () from
>>>>>
>> /usr/lib/opensips/modules/tm.so
>>
>>>>> #5 0xb773b501 in ?? () from /usr/lib/opensips/modules/tm.so
>>>>> #6 0x08055030 in do_action ()
>>>>> #7 0x08053ebf in run_action_list ()
>>>>> #8 0x08095cf2 in eval_expr ()
>>>>> #9 0x080958d9 in eval_expr ()
>>>>> #10 0x08095919 in eval_expr ()
>>>>> #11 0x080554e2 in do_action ()
>>>>> #12 0x08053ebf in run_action_list ()
>>>>> #13 0x080569d8 in do_action ()
>>>>> #14 0x08053ebf in run_action_list ()
>>>>> #15 0x08056e7a in do_action ()
>>>>> #16 0x08053ebf in run_action_list ()
>>>>> #17 0x08057d99 in run_top_route ()
>>>>> #18 0x0808ad6c in receive_msg ()
>>>>> #19 0x080bd2f2 in udp_rcv_loop ()
>>>>> #20 0x08069339 in main ()
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Alex Massover
>>>>> VoIP R&D TL
>>>>> Jajah Inc.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>> bounces at lists.opensips.org] On Behalf Of Alex Massover
>>>>>> Sent: Thursday, January 21, 2010 2:24 PM
>>>>>> To: OpenSIPS users mailling list
>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Another one.. It hangs for a number of seconds (but it's enough to
>>>>>> cause to SIP timeouts - MSG queue jumps to 260K), it's hard to
>>>>>>
>> make
>>
>>>> a
>>>>
>>>>
>>>>>> bt at the right moment.
>>>>>> This one looks better because there's sched_yield() there :)
>>>>>>
>>>>>> (gdb) bt
>>>>>> #0 0xb77d5424 in __kernel_vsyscall ()
>>>>>> #1 0xb771041c in sched_yield () from /lib/i686/cmov/libc.so.6
>>>>>> #2 0x080bf23d in new_avp ()
>>>>>> #3 0x080bf53f in add_avp ()
>>>>>> #4 0xb72c1c9c in ?? () from /usr/lib/opensips/modules/dialog.so
>>>>>> #5 0x08055030 in do_action ()
>>>>>> #6 0x08053ebf in run_action_list ()
>>>>>> #7 0x08056e7a in do_action ()
>>>>>> #8 0x08053ebf in run_action_list ()
>>>>>> #9 0x08056e7a in do_action ()
>>>>>> #10 0x08053ebf in run_action_list ()
>>>>>> #11 0x08056e7a in do_action ()
>>>>>> #12 0x08053ebf in run_action_list ()
>>>>>> #13 0x08057d99 in run_top_route ()
>>>>>> #14 0x0808ad6c in receive_msg ()
>>>>>> #15 0x080bd2f2 in udp_rcv_loop ()
>>>>>> #16 0x08069339 in main ()
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Alex Massover
>>>>>> VoIP R&D TL
>>>>>> Jajah Inc.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>> bounces at lists.opensips.org] On Behalf Of Alex Massover
>>>>>>> Sent: Thursday, January 21, 2010 2:05 PM
>>>>>>> To: OpenSIPS users mailling list
>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>>
>>>>>>> Hi Andrei,
>>>>>>> Hopefully this is it (with FASTLOCK)
>>>>>>>
>>>>>>> #0 0xb77d5424 in __kernel_vsyscall ()
>>>>>>> #1 0xb772babb in poll () from /lib/i686/cmov/libc.so.6
>>>>>>> #2 0xb77ba83a in ?? () from /lib/i686/cmov/libresolv.so.2
>>>>>>> #3 0xb77b8946 in __libc_res_nquery () from
>>>>>>> /lib/i686/cmov/libresolv.so.2
>>>>>>> #4 0xb77b8fdb in ?? () from /lib/i686/cmov/libresolv.so.2
>>>>>>> #5 0xb77b92ae in __libc_res_nsearch () from
>>>>>>> /lib/i686/cmov/libresolv.so.2
>>>>>>> #6 0xb77b96d4 in __res_nsearch () from
>>>>>>>
>>>>>>>
>>>> /lib/i686/cmov/libresolv.so.2
>>>>
>>>>
>>>>>>> #7 0xb77b808a in res_search () from
>>>>>>>
>> /lib/i686/cmov/libresolv.so.2
>>
>>>>>>> #8 0x0808c613 in get_record ()
>>>>>>> #9 0x0808cf05 in ?? ()
>>>>>>> #10 0x0808e385 in sip_resolvehost ()
>>>>>>> #11 0x0807a26c in mk_proxy ()
>>>>>>> #12 0xb7627d39 in t_relay_to () from
>>>>>>>
>>>>>>>
>>>> /usr/lib/opensips/modules/tm.so
>>>>
>>>>
>>>>>>> #13 0xb7634501 in ?? () from /usr/lib/opensips/modules/tm.so
>>>>>>> #14 0x08055030 in do_action ()
>>>>>>> #15 0x08053ebf in run_action_list ()
>>>>>>> #16 0x08095cf2 in eval_expr ()
>>>>>>> #17 0x080958d9 in eval_expr ()
>>>>>>> #18 0x08095919 in eval_expr ()
>>>>>>> #19 0x080554e2 in do_action ()
>>>>>>> #20 0x08053ebf in run_action_list ()
>>>>>>> #21 0x08056e7a in do_action ()
>>>>>>> #22 0x08053ebf in run_action_list ()
>>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>> #23 0x080569d8 in do_action ()
>>>>>>> #24 0x08053ebf in run_action_list ()
>>>>>>> #25 0x08056e7a in do_action ()
>>>>>>> #26 0x08053ebf in run_action_list ()
>>>>>>> #27 0x08057d99 in run_top_route ()
>>>>>>> #28 0x0808ad6c in receive_msg ()
>>>>>>> #29 0x080bd2f2 in udp_rcv_loop ()
>>>>>>> #30 0x08069339 in main ()
>>>>>>> (gdb)
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Alex Massover
>>>>>>> VoIP R&D TL
>>>>>>> Jajah Inc.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>>>>>>>> Sent: Wednesday, January 20, 2010 2:58 PM
>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I think that there is a lock that is being held more than it
>>>>>>>>
>>>>>>>>
>>>> should
>>>>
>>>>
>>>>>>> be
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> and that's what causes starvation. It would help us if you could
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> attach
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> to a process using gdb and give us a full backtrace.
>>>>>>>>
>>>>>>>> Temporary solutions which should work would be to reduce the
>>>>>>>>
>>>>>>>>
>>>> number
>>>>
>>>>
>>>>>>> of
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> processes to 4-6 or to recompile replacing -DFAST_LOCK with one
>>>>>>>>
>> of
>>
>>>>>>>>
>>>>>>> the
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> other options (-DUSE_POSIX_SEM or -DUSE_PTHREAD_MUTEX) but we
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> should
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> see
>>>>>>>> where this is from to fix it.
>>>>>>>>
>>>>>>>> Alex Massover wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> Yes, from the source on debian, I build deb package. (I did
>>>>>>>>>
>> some
>>
>>>>>>>>>
>>>>>>>> minor changes to the source, but the problem happens also
>>>>>>>>
>> without
>>
>>>>>>>>
>>>>>> my
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> changes)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> 16 children on 4 cores.
>>>>>>>>>
>>>>>>>>> What do you suggest to reduce it to 4? It runs on 2.6.32 on
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> VMware
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> ESX.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I'm also trying now sleep(0) instead of sched_yield().
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Alex Massover
>>>>>>>>> VoIP R&D TL
>>>>>>>>> Jajah Inc.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>>>>>>>>>> Sent: Wednesday, January 20, 2010 1:05 PM
>>>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>>>>>
>>>>>>>>>> Hi Alex,
>>>>>>>>>>
>>>>>>>>>> Are you building OpenSIPS from source?
>>>>>>>>>> How many processes do you have and on how many cores?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Alex Massover wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hello!
>>>>>>>>>>>
>>>>>>>>>>> I'm facing a strange problem, sometimes under a stress
>>>>>>>>>>>
>> OpenSIPS
>>
>>>>>>>>>>> "locks" - load average jumps, SIP processing delays, opensips
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>> msg
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>>>> queue fills with a lot of sip messages, opensips processes
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>> start
>>>>>>
>>>>>>
>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>>> comsume a lot of CPU.
>>>>>>>>>>>
>>>>>>>>>>> And strace shows:
>>>>>>>>>>>
>>>>>>>>>>> sched_yield()
>>>>>>>>>>>
>>>>>>>>>>> sched_yield()
>>>>>>>>>>>
>>>>>>>>>>> sched_yield()
>>>>>>>>>>>
>>>>>>>>>>> sched_yield()
>>>>>>>>>>>
>>>>>>>>>>> ....
>>>>>>>>>>>
>>>>>>>>>>> for all processes.
>>>>>>>>>>>
>>>>>>>>>>> If I stop the stress - after a while (not immediately) - it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> unlocks,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>> also suddenly, I can see in top that all opensips processes
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>> stop
>>>>>>
>>>>>>
>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>>> consume CPU.
>>>>>>>>>>>
>>>>>>>>>>> What can it be? Some kind of starvation?
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Best Regards,
>>>>>>>>>>>
>>>>>>>>>>> Alex Massover
>>>>>>>>>>>
>>>>>>>>>>> VoIP R&D TL
>>>>>>>>>>>
>>>>>>>>>>> Jajah Inc.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>>>>> -------------------------------------------------------------
>>>>>>>>>>>
>> --
>>
>>>>>>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Andrei Dragus
>>>>>>>>>> www.voice-system.ro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>> This mail was received via Mail-SeCure System.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at lists.opensips.org
>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Andrei Dragus
>>>>>>>> www.voice-system.ro
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at lists.opensips.org
>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>> This mail was received via Mail-SeCure System.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at lists.opensips.org
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>> This mail was received via Mail-SeCure System.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at lists.opensips.org
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>> This mail was received via Mail-SeCure System.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> This mail was sent via Mail-SeCure System.
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opensips.org
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Andrei Dragus
>>>> www.voice-system.ro
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.opensips.org
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>> This mail was received via Mail-SeCure System.
>>>>
>>>>
>>>>
>>> This mail was sent via Mail-SeCure System.
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opensips.org
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>> --
>> Andrei Dragus
>> www.voice-system.ro
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>> This mail was received via Mail-SeCure System.
>>
>>
>
>
> This mail was sent via Mail-SeCure System.
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
--
Andrei Dragus
www.voice-system.ro
More information about the Users
mailing list