[OpenSIPS-Users] sched_yield()

Bogdan-Andrei Iancu bogdan at voice-system.ro
Fri Jan 22 18:42:46 CET 2010


Hi Alex,

Bug was fixed - update from SVN.

Regarding your observation on"forking" versus "no-forking" - in some 
cases (when not doing any blocking ops), a single proc may be faster 
that multiple procs on a single core machine - because the CPU power is 
the same and maximum used (no blocking), but in forking mode you have 
the overhead of proc switching and the loocking/synchronizing dead-times.

Regards,
Bogdan

Alex Massover wrote:
> Hi,
>
> Unfortunately 'fifo get_statistics' crashes opensips, I opened a bug.
> But no chance that 1G is not enough, only about 400M is used for all linux processes:
> Mem:   3115120k total,   398360k used,  2716760k free,      536k buffers
>
> Maybe sched_yield() just cause problems on 2.3.62 or on vmware or on SMP?
>
> I'm trying now with fork=yes and children=1.
> If I have only one working child, does it suppose to lock and shed_yeild() itself from any reason?
>
> Meanwhile with single child OpenSIPS easily handles 4K of concurrent calls at 15cps, load average is 0.00 (!) and CPU is about 96% idle.
>
> I wonder if single working child also hangs.
>
> --
> Best Regards,
> Alex Massover
> VoIP R&D TL
> Jajah Inc.
>   
>> -----Original Message-----
>> From: users-bounces at lists.opensips.org [mailto:users-
>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>> Sent: Friday, January 22, 2010 1:17 PM
>> To: OpenSIPS users mailling list
>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>
>> Hi,
>>
>> The new f_malloc will not do anything extra when compared to the old
>> one
>> until memory usage goes way up.
>> I've added a warning in mem/f_malloc.c so you can see when defrag
>> starts. If you get this warning then it is clear that the problem is
>> from high memory usage.
>>
>> 1 GB for 4k calls seems a lot ( 250k per call). You can try to use
>> "opensipsctl fifo get_statistics shmem:" and see what the memory usage
>> is for diferent number of concurrent calls ( 1k,2k,3k,4k), and if
>> indeed
>> the memory usage is that high we should investigate the cause.
>>
>>
>> Alex Massover wrote:
>>     
>>> Hi,
>>>
>>> Now shared memory is 1G (-m 1024), and all memory  is dedicated to
>>>       
>> the virtual machine (it was shared till now).
>>     
>>> But it still happens, just not so often.
>>>
>>> I originate the calls for this stress test in Asterisk with the same
>>>       
>> resources and looks like Asterisk performs much better than OpenSIPS.
>> How can it be?
>>     
>>> In my stress OpenSIPS does no blocking/slow requests. And it's just
>>>       
>> 4K concurrent calls, each one is 2-3 min.
>>     
>>> Maybe OpenSIPS does too much low level memory management and virtual
>>>       
>> machine is not suitable for it (despite that Asterisk runs well over
>> VMware)?
>>     
>>> I'm not sure but I have a feeling that 1.4 performed better. What can
>>>       
>> cause performance degradation in 1.6? Storing vars on dialog, new
>> malloc()?
>>     
>>> gdb) bt
>>> #0  0xb78ad424 in __kernel_vsyscall ()
>>> #1  0xb77e841c in sched_yield () from /lib/i686/cmov/libc.so.6
>>> #2  0x080bf23d in new_avp ()
>>> #3  0x080bf53f in add_avp ()
>>> #4  0x08080e6e in pv_set_avp ()
>>> #5  0x0808229c in pv_set_value ()
>>> #6  0x08053c9d in do_assign ()
>>> #7  0x0805447a in do_action ()
>>> #8  0x08053ebf in run_action_list ()
>>> #9  0x08056e7a in do_action ()
>>> #10 0x08053ebf in run_action_list ()
>>> #11 0x08056e7a in do_action ()
>>> #12 0x08053ebf in run_action_list ()
>>> #13 0x080569d8 in do_action ()
>>> #14 0x08053ebf in run_action_list ()
>>> #15 0x08056e7a in do_action ()
>>> #16 0x08053ebf in run_action_list ()
>>> #17 0x08057d99 in run_top_route ()
>>> #18 0x0808ad6c in receive_msg ()
>>> #19 0x080bd2f2 in udp_rcv_loop ()
>>> #20 0x08069339 in main ()
>>> (gdb) quit
>>>
>>>
>>> (gdb) bt
>>> #0  0xb78ad424 in __kernel_vsyscall ()
>>> #1  0xb77e841c in sched_yield () from /lib/i686/cmov/libc.so.6
>>> #2  0xb76f52cd in build_cell () from /usr/lib/opensips/modules/tm.so
>>> #3  0xb770ac4a in t_newtran () from /usr/lib/opensips/modules/tm.so
>>> #4  0xb76ff7b8 in t_relay_to () from /usr/lib/opensips/modules/tm.so
>>> #5  0xb770c501 in ?? () from /usr/lib/opensips/modules/tm.so
>>> #6  0x08055030 in do_action ()
>>> #7  0x08053ebf in run_action_list ()
>>> #8  0x08095cf2 in eval_expr ()
>>> #9  0x080958d9 in eval_expr ()
>>> #10 0x08095919 in eval_expr ()
>>> #11 0x080554e2 in do_action ()
>>> #12 0x08053ebf in run_action_list ()
>>> #13 0x080569d8 in do_action ()
>>> #14 0x08053ebf in run_action_list ()
>>> #15 0x08056e7a in do_action ()
>>> #16 0x08053ebf in run_action_list ()
>>> #17 0x08057d99 in run_top_route ()
>>> #18 0x0808ad6c in receive_msg ()
>>> #19 0x080bd2f2 in udp_rcv_loop ()
>>> #20 0x08069339 in main ()
>>>
>>>
>>> --
>>> Best Regards,
>>> Alex Massover
>>> VoIP R&D TL
>>> Jajah Inc.
>>>
>>>
>>>       
>>>> -----Original Message-----
>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>>>> Sent: Thursday, January 21, 2010 3:43 PM
>>>> To: OpenSIPS users mailling list
>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>
>>>> My guess is that there is not enough shared memory. When an
>>>>         
>> allocation
>>     
>>>> failes OpenSIPS tries to defragment memory to make room which takes
>>>>         
>> a
>>     
>>>> lot of time and must be done under lock.
>>>>
>>>> Please try to increase the shared memory size and tell me if it
>>>> persists.
>>>>
>>>>
>>>> Alex Massover wrote:
>>>>
>>>>         
>>>>> Hi!
>>>>>
>>>>> Yes, with -DF_MALLOC.
>>>>>
>>>>> 1.6.1 from sources, I build deb package.
>>>>> I use 128M of shared and 10*1024*1024 private memory (can increase
>>>>>           
>> -
>>     
>>>> no problem).
>>>>
>>>>         
>>>>> Hmmmm, "opensipsctl fifo get_statistics all" crashes/stops the
>>>>>
>>>>>           
>>>> opensips.
>>>>
>>>>         
>>>>> 'fifo uptime' or 'fifo debug' are OK.
>>>>>
>>>>> strace while 'fifo get_statistics all':
>>>>> Process 9509 attached - interrupt to quit
>>>>> pause()                                 = ? ERESTARTNOHAND (To be
>>>>>
>>>>>           
>>>> restarted)
>>>>
>>>>         
>>>>> --- SIGUSR2 (User defined signal 2) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> pause()                                 = ? ERESTARTNOHAND (To be
>>>>>
>>>>>           
>>>> restarted)
>>>>
>>>>         
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> waitpid(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGUSR2}], WNOHANG)
>>>>>           
>> =
>>     
>>>> 9520
>>>>
>>>>         
>>>>> waitpid(-1, 0xbf84b4c8, WNOHANG)        = 0
>>>>> kill(0, SIGTERM)                        = 0
>>>>> --- SIGTERM (Terminated) @ 0 (0) ---
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [TERM])
>>>>> sigreturn()                             = ? (mask now [])
>>>>> rt_sigaction(SIGALRM, {0x8065920, [ALRM], SA_RESTART}, {SIG_DFL},
>>>>>           
>> 8)
>>     
>>>> = 0
>>>>
>>>>         
>>>>> alarm(60)                               = 0
>>>>> wait4(-1, NULL, 0, NULL)                = 9514
>>>>> wait4(-1, NULL, 0, NULL)                = 9519
>>>>> wait4(-1, NULL, 0, NULL)                = 9521
>>>>> wait4(-1, NULL, 0, NULL)                = 9522
>>>>> wait4(-1, NULL, 0, NULL)                = 9512
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> wait4(-1, NULL, 0, NULL)                = 9510
>>>>> wait4(-1, NULL, 0, NULL)                = 9516
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> wait4(-1, NULL, 0, NULL)                = 9515
>>>>> wait4(-1, NULL, 0, NULL)                = 9517
>>>>> wait4(-1, NULL, 0, NULL)                = 9524
>>>>> wait4(-1, NULL, 0, NULL)                = 9525
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> --- SIGCHLD (Child exited) @ 0 (0) ---
>>>>> sigreturn()                             = ? (mask now [])
>>>>> wait4(-1, NULL, 0, NULL)                = 9511
>>>>> wait4(-1, NULL, 0, NULL)                = 9513
>>>>> wait4(-1, NULL, 0, NULL)                = 9518
>>>>> wait4(-1, NULL, 0, NULL)                = 9523
>>>>> wait4(-1, NULL, 0, NULL)                = -1 ECHILD (No child
>>>>>
>>>>>           
>>>> processes)
>>>>
>>>>         
>>>>> rt_sigaction(SIGALRM, {0x8066080, [ALRM], SA_RESTART}, {0x8065920,
>>>>>
>>>>>           
>>>> [ALRM], SA_RESTART}, 8) = 0
>>>>
>>>>         
>>>>> stat64("/tmp/opensips_fifo", {st_mode=S_IFIFO|0660, st_size=0,
>>>>>           
>> ...})
>>     
>>>> = 0
>>>>
>>>>         
>>>>> unlink("/tmp/opensips_fifo")            = 0
>>>>> munmap(0xaed25000, 134217728)           = 0
>>>>> unlink("/var/run/opensips/opensips.pid") = 0
>>>>> alarm(0)                                = 60
>>>>> rt_sigaction(SIGALRM, {SIG_IGN}, {0x8066080, [ALRM], SA_RESTART},
>>>>>           
>> 8)
>>     
>>>> = 0
>>>>
>>>>         
>>>>> exit_group(0)                           = ?
>>>>> Process 9509 detached
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Alex Massover
>>>>> VoIP R&D TL
>>>>> Jajah Inc.
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>> -----Original Message-----
>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>>>>>> Sent: Thursday, January 21, 2010 3:09 PM
>>>>>> To: OpenSIPS users mailling list
>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Since all the backtraces are in allocation routines my guess is
>>>>>>             
>> that
>>     
>>>>>> the
>>>>>> shared memory lock might be causing a problem.
>>>>>>
>>>>>> Are you compiling with -DF_MALLOC?
>>>>>> What version of OpenSIPS are you using?
>>>>>> What is the total shared memory pool you are allocating?
>>>>>> What amount of memory are you using? ( Use : opensipsctl fifo
>>>>>> get_statistics all )
>>>>>>
>>>>>> Alex Massover wrote:
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> Some more,
>>>>>>>
>>>>>>> (gdb) bt
>>>>>>> #0  0xb78dc424 in __kernel_vsyscall ()
>>>>>>> #1  0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
>>>>>>> #2  0xb73d77fd in build_new_dlg () from
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> /usr/lib/opensips/modules/dialog.so
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> #3  0xb73d4b81 in dlg_create_dialog () from
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> /usr/lib/opensips/modules/dialog.so
>>>>>>
>>>>>>
>>>>>>             
>>>>>>> #4  0xb73c9c9e in ?? () from /usr/lib/opensips/modules/dialog.so
>>>>>>> #5  0x08055030 in do_action ()
>>>>>>> #6  0x08053ebf in run_action_list ()
>>>>>>> #7  0x08056e7a in do_action ()
>>>>>>> #8  0x08053ebf in run_action_list ()
>>>>>>> #9  0x08057d99 in run_top_route ()
>>>>>>> #10 0x0808ad6c in receive_msg ()
>>>>>>> #11 0x080bd2f2 in udp_rcv_loop ()
>>>>>>> #12 0x08069339 in main ()
>>>>>>>
>>>>>>>
>>>>>>>    (gdb) bt
>>>>>>> #0  0xb78dc424 in __kernel_vsyscall ()
>>>>>>> #1  0xb781741c in sched_yield () from /lib/i686/cmov/libc.so.6
>>>>>>> #2  0xb77242cd in build_cell () from
>>>>>>>
>>>>>>>               
>>>> /usr/lib/opensips/modules/tm.so
>>>>
>>>>         
>>>>>>> #3  0xb7739c4a in t_newtran () from
>>>>>>>               
>> /usr/lib/opensips/modules/tm.so
>>     
>>>>>>> #4  0xb772e7b8 in t_relay_to () from
>>>>>>>
>>>>>>>               
>>>> /usr/lib/opensips/modules/tm.so
>>>>
>>>>         
>>>>>>> #5  0xb773b501 in ?? () from /usr/lib/opensips/modules/tm.so
>>>>>>> #6  0x08055030 in do_action ()
>>>>>>> #7  0x08053ebf in run_action_list ()
>>>>>>> #8  0x08095cf2 in eval_expr ()
>>>>>>> #9  0x080958d9 in eval_expr ()
>>>>>>> #10 0x08095919 in eval_expr ()
>>>>>>> #11 0x080554e2 in do_action ()
>>>>>>> #12 0x08053ebf in run_action_list ()
>>>>>>> #13 0x080569d8 in do_action ()
>>>>>>> #14 0x08053ebf in run_action_list ()
>>>>>>> #15 0x08056e7a in do_action ()
>>>>>>> #16 0x08053ebf in run_action_list ()
>>>>>>> #17 0x08057d99 in run_top_route ()
>>>>>>> #18 0x0808ad6c in receive_msg ()
>>>>>>> #19 0x080bd2f2 in udp_rcv_loop ()
>>>>>>> #20 0x08069339 in main ()
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Alex Massover
>>>>>>> VoIP R&D TL
>>>>>>> Jajah Inc.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> -----Original Message-----
>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>> bounces at lists.opensips.org] On Behalf Of Alex Massover
>>>>>>>> Sent: Thursday, January 21, 2010 2:24 PM
>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Another one.. It hangs for a number of seconds (but it's enough
>>>>>>>>                 
>> to
>>     
>>>>>>>> cause to SIP timeouts - MSG queue jumps to 260K), it's hard to
>>>>>>>>
>>>>>>>>                 
>>>> make
>>>>
>>>>         
>>>>>> a
>>>>>>
>>>>>>
>>>>>>             
>>>>>>>> bt at the right moment.
>>>>>>>> This one looks better because there's sched_yield() there :)
>>>>>>>>
>>>>>>>> (gdb) bt
>>>>>>>> #0  0xb77d5424 in __kernel_vsyscall ()
>>>>>>>> #1  0xb771041c in sched_yield () from /lib/i686/cmov/libc.so.6
>>>>>>>> #2  0x080bf23d in new_avp ()
>>>>>>>> #3  0x080bf53f in add_avp ()
>>>>>>>> #4  0xb72c1c9c in ?? () from /usr/lib/opensips/modules/dialog.so
>>>>>>>> #5  0x08055030 in do_action ()
>>>>>>>> #6  0x08053ebf in run_action_list ()
>>>>>>>> #7  0x08056e7a in do_action ()
>>>>>>>> #8  0x08053ebf in run_action_list ()
>>>>>>>> #9  0x08056e7a in do_action ()
>>>>>>>> #10 0x08053ebf in run_action_list ()
>>>>>>>> #11 0x08056e7a in do_action ()
>>>>>>>> #12 0x08053ebf in run_action_list ()
>>>>>>>> #13 0x08057d99 in run_top_route ()
>>>>>>>> #14 0x0808ad6c in receive_msg ()
>>>>>>>> #15 0x080bd2f2 in udp_rcv_loop ()
>>>>>>>> #16 0x08069339 in main ()
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Alex Massover
>>>>>>>> VoIP R&D TL
>>>>>>>> Jajah Inc.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Alex Massover
>>>>>>>>> Sent: Thursday, January 21, 2010 2:05 PM
>>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>>>>
>>>>>>>>> Hi Andrei,
>>>>>>>>> Hopefully this is it (with FASTLOCK)
>>>>>>>>>
>>>>>>>>> #0  0xb77d5424 in __kernel_vsyscall ()
>>>>>>>>> #1  0xb772babb in poll () from /lib/i686/cmov/libc.so.6
>>>>>>>>> #2  0xb77ba83a in ?? () from /lib/i686/cmov/libresolv.so.2
>>>>>>>>> #3  0xb77b8946 in __libc_res_nquery () from
>>>>>>>>> /lib/i686/cmov/libresolv.so.2
>>>>>>>>> #4  0xb77b8fdb in ?? () from /lib/i686/cmov/libresolv.so.2
>>>>>>>>> #5  0xb77b92ae in __libc_res_nsearch () from
>>>>>>>>> /lib/i686/cmov/libresolv.so.2
>>>>>>>>> #6  0xb77b96d4 in __res_nsearch () from
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>> /lib/i686/cmov/libresolv.so.2
>>>>>>
>>>>>>
>>>>>>             
>>>>>>>>> #7  0xb77b808a in res_search () from
>>>>>>>>>
>>>>>>>>>                   
>>>> /lib/i686/cmov/libresolv.so.2
>>>>
>>>>         
>>>>>>>>> #8  0x0808c613 in get_record ()
>>>>>>>>> #9  0x0808cf05 in ?? ()
>>>>>>>>> #10 0x0808e385 in sip_resolvehost ()
>>>>>>>>> #11 0x0807a26c in mk_proxy ()
>>>>>>>>> #12 0xb7627d39 in t_relay_to () from
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>> /usr/lib/opensips/modules/tm.so
>>>>>>
>>>>>>
>>>>>>             
>>>>>>>>> #13 0xb7634501 in ?? () from /usr/lib/opensips/modules/tm.so
>>>>>>>>> #14 0x08055030 in do_action ()
>>>>>>>>> #15 0x08053ebf in run_action_list ()
>>>>>>>>> #16 0x08095cf2 in eval_expr ()
>>>>>>>>> #17 0x080958d9 in eval_expr ()
>>>>>>>>> #18 0x08095919 in eval_expr ()
>>>>>>>>> #19 0x080554e2 in do_action ()
>>>>>>>>> #20 0x08053ebf in run_action_list ()
>>>>>>>>> #21 0x08056e7a in do_action ()
>>>>>>>>> #22 0x08053ebf in run_action_list ()
>>>>>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>>>>> #23 0x080569d8 in do_action ()
>>>>>>>>> #24 0x08053ebf in run_action_list ()
>>>>>>>>> #25 0x08056e7a in do_action ()
>>>>>>>>> #26 0x08053ebf in run_action_list ()
>>>>>>>>> #27 0x08057d99 in run_top_route ()
>>>>>>>>> #28 0x0808ad6c in receive_msg ()
>>>>>>>>> #29 0x080bd2f2 in udp_rcv_loop ()
>>>>>>>>> #30 0x08069339 in main ()
>>>>>>>>> (gdb)
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Alex Massover
>>>>>>>>> VoIP R&D TL
>>>>>>>>> Jajah Inc.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>>>>>>>>>> Sent: Wednesday, January 20, 2010 2:58 PM
>>>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I think that there is a lock that is being held more than it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> should
>>>>>>
>>>>>>
>>>>>>             
>>>>>>>>> be
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> and that's what causes starvation. It would help us if you
>>>>>>>>>>                     
>> could
>>     
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> attach
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> to a process using gdb and give us a full backtrace.
>>>>>>>>>>
>>>>>>>>>> Temporary solutions which should work would be to reduce the
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>> number
>>>>>>
>>>>>>
>>>>>>             
>>>>>>>>> of
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> processes to 4-6 or to recompile replacing -DFAST_LOCK with
>>>>>>>>>>                     
>> one
>>     
>>>> of
>>>>
>>>>         
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> other options (-DUSE_POSIX_SEM or -DUSE_PTHREAD_MUTEX) but we
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>> should
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> see
>>>>>>>>>> where this is from to fix it.
>>>>>>>>>>
>>>>>>>>>> Alex Massover wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> Hi!
>>>>>>>>>>>
>>>>>>>>>>> Yes, from the source on debian, I build deb package. (I did
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>> some
>>>>
>>>>         
>>>>>>>>>> minor changes to the source, but the problem happens also
>>>>>>>>>>
>>>>>>>>>>                     
>>>> without
>>>>
>>>>         
>>>>>>>> my
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> changes)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> 16 children on 4 cores.
>>>>>>>>>>>
>>>>>>>>>>> What do you suggest to reduce it to 4? It runs on 2.6.32 on
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>> VMware
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> ESX.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> I'm also trying now sleep(0) instead of sched_yield().
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Alex Massover
>>>>>>>>>>> VoIP R&D TL
>>>>>>>>>>> Jajah Inc.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: users-bounces at lists.opensips.org [mailto:users-
>>>>>>>>>>>> bounces at lists.opensips.org] On Behalf Of Andrei Dragus
>>>>>>>>>>>> Sent: Wednesday, January 20, 2010 1:05 PM
>>>>>>>>>>>> To: OpenSIPS users mailling list
>>>>>>>>>>>> Subject: Re: [OpenSIPS-Users] sched_yield()
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Alex,
>>>>>>>>>>>>
>>>>>>>>>>>> Are you building OpenSIPS from source?
>>>>>>>>>>>> How many processes do you have and on how many cores?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Alex Massover wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                         
>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm facing a strange problem, sometimes under a stress
>>>>>>>>>>>>>
>>>>>>>>>>>>>                           
>>>> OpenSIPS
>>>>
>>>>         
>>>>>>>>>>>>> "locks" - load average jumps, SIP processing delays,
>>>>>>>>>>>>>                           
>> opensips
>>     
>>>>>>>>>>>>>
>>>>>>>>>>>>>                           
>>>>>>>> msg
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>>>>> queue fills with a lot of sip messages, opensips processes
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                           
>>>>>>>> start
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>>>>> comsume a lot of CPU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> And strace shows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> sched_yield()
>>>>>>>>>>>>>
>>>>>>>>>>>>> sched_yield()
>>>>>>>>>>>>>
>>>>>>>>>>>>> sched_yield()
>>>>>>>>>>>>>
>>>>>>>>>>>>> sched_yield()
>>>>>>>>>>>>>
>>>>>>>>>>>>> ....
>>>>>>>>>>>>>
>>>>>>>>>>>>> for all processes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If I stop the stress - after a while (not immediately) - it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                           
>>>>>>>>>> unlocks,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>>>> also suddenly, I can see in top that all opensips processes
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                           
>>>>>>>> stop
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>>>>> consume CPU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What can it be? Some kind of starvation?
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alex Massover
>>>>>>>>>>>>>
>>>>>>>>>>>>> VoIP R&D TL
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jajah Inc.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>>>>>>> -----------------------------------------------------------
>>>>>>>>>>>>>                           
>> --
>>     
>>>> --
>>>>
>>>>         
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>>> ---
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                         
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                           
>>>>>>>>>>>> --
>>>>>>>>>>>> Andrei Dragus
>>>>>>>>>>>> www.voice-system.ro
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>>
>>>>>>>>>>>> This mail was received via Mail-SeCure System.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                         
>>>>>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list
>>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> --
>>>>>>>>>> Andrei Dragus
>>>>>>>>>> www.voice-system.ro
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list
>>>>>>>>>> Users at lists.opensips.org
>>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>> This mail was received via Mail-SeCure System.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at lists.opensips.org
>>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>> This mail was received via Mail-SeCure System.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at lists.opensips.org
>>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>> This mail was received via Mail-SeCure System.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> This mail was sent via Mail-SeCure System.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at lists.opensips.org
>>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> --
>>>>>> Andrei Dragus
>>>>>> www.voice-system.ro
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at lists.opensips.org
>>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>> This mail was received via Mail-SeCure System.
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> This mail was sent via Mail-SeCure System.
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opensips.org
>>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>>           
>>>> --
>>>> Andrei Dragus
>>>> www.voice-system.ro
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.opensips.org
>>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>> This mail was received via Mail-SeCure System.
>>>>
>>>>
>>>>         
>>> This mail was sent via Mail-SeCure System.
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opensips.org
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>       
>> --
>> Andrei Dragus
>> www.voice-system.ro
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>> This mail was received via Mail-SeCure System.
>>
>>     
>
>
> This mail was sent via Mail-SeCure System.
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>   


-- 
Bogdan-Andrei Iancu
www.voice-system.ro




More information about the Users mailing list