[OpenSIPS-Users] Major performance issues question

Schneur Rosenberg rosenberg11219 at gmail.com
Wed Dec 21 13:22:36 EST 2016


No this server is running only OpenSIPS and mysql, here is a report of
iotop which I took when the resources was going down

Actual DISK READ:       0.00 B/s | Actual DISK WRITE:      78.77 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  153 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.08 % [jbd2/vda1-8]
 1681 be/4 mysql       0.00 B/s    7.88 K/s  0.00 %  0.06 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
 1711 be/4 mysql       0.00 B/s    3.94 K/s  0.00 %  0.05 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
  647 be/4 root        0.00 B/s    3.94 K/s  0.00 %  0.00 % rsyslogd -n
[rs:main Q:Reg]
 2048 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
 2052 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
    5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
    6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u16:0]
    7 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
    8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
    9 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
   10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
   11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
   12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
   13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
 1038 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
   15 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/1:0H]
   16 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
   17 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
   18 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
 2051 be/4 opensips    0.00 B/s    0.00 B/s  0.00 %  0.00 % opensips -P
/var/run/opensips/opensips.pid -m 128 -M 4 -u opensips -g opensips -f
/etc/opensips/opensips.cfg
   20 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/2:0H]
   21 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
   22 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
   23 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/3]
 2072 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
   25 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/3:0H]
   26 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
   27 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
   28 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/4]
 2077 be/4 opensips    0.00 B/s    0.00 B/s  0.00 %  0.00 % opensips -P
/var/run/opensips/opensips.pid -m 128 -M 4 -u opensips -g opensips -f
/etc/opensips/opensips.cfg
   30 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/4:0H]
   31 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/5]
   32 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]
   33 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/5]
   34 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/5:0]
   35 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/5:0H]
   36 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/6]


On Wed, Dec 21, 2016 at 7:25 PM, Liviu Chircu <liviu at opensips.org> wrote:

> Okay, so it seems lots of CPU cycles are spent in the kernel!
>
> Are you sure there isn't any intensive I/O going on? For example, maybe
> you are pushing lots of gateway pings. You could run "iostat 1" to assess
> this.
>
> Regards,
>
> Liviu Chircu
> OpenSIPS Developerhttp://www.opensips-solutions.com
>
> On 21.12.2016 18:58, Schneur Rosenberg wrote:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ------cpu-----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
> wa st
>  7  0      0 9966412 296728 5239152    0    0     3    60    0    0  0  0
> 99  0  0
>  6  0      0 9966684 296728 5239152    0    0     0    20 2075 1569 12 64
> 24  0  0
>  7  0      0 9966588 296728 5239152    0    0     0    20 2023 1408 12 64
> 24  0  0
>  4  0      0 9966492 296728 5239152    0    0     0   108 2027 1434 11 63
> 26  0  0
>  5  0      0 9966588 296728 5239152    0    0     0    44 1687 1232 11 52
> 38  0  0
>  5  0      0 9966620 296728 5239156    0    0     0    72 1704 1136  9 53
> 37  0  0
>  6  0      0 9966492 296728 5239156    0    0     0    40 1874 1133 11 64
> 25  0  0
>  6  0      0 9966524 296728 5239156    0    0     0    20 1794  853 11 65
> 25  0  0
>  6  0      0 9966556 296728 5239156    0    0     0    56 1810  951 11 64
> 25  0  0
>  6  0      0 9966620 296728 5239156    0    0     0     0 1802  934 10 65
> 25  0  0
>  7  0      0 9966556 296728 5239160    0    0     0  2328 2356 1489 12 76
> 12  0  0
>  7  0      0 9966460 296728 5239160    0    0     0     0 2074  929 12 76
> 12  0  0
>  7  0      0 9966588 296728 5239164    0    0     0     0 2022  728 11 76
> 12  0  0
>  7  0      0 9966620 296728 5239164    0    0     0     0 2015  802 14 74
> 12  0  0
>  5  0      0 9966652 296728 5239164    0    0     0    72 2275 1049 14 81
>  5  0  0
>
>
> On Wed, Dec 21, 2016 at 6:42 PM, Liviu Chircu <liviu at opensips.org> wrote:
>
>> Hi Schneur,
>>
>> Those warnings are due to a rather harmless bug, which was fixed and
>> backported ~ two weeks ago.
>>
>> Regarding 100% CPU usage though, could you please run "vmstat 1" for 10
>> seconds and paste the full output? Thanks!
>>
>> Regards,
>>
>> Liviu Chircu
>> OpenSIPS Developerhttp://www.opensips-solutions.com
>>
>> On 21.12.2016 18:17, Schneur Rosenberg wrote:
>>
>> Hi, I'm running a instance of OpenSIPS (just signalling no RTP on this
>> machine) on a DigitalOcean VM, it was running fine for a while and it does
>> not process lots of CPS (4-5 max) and today I was getting complaints of
>> dropped calls, I checked the logs and I saw lots of
>>  WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for
>> 3956760 ms (now 3956860 ms), it may overlap
>> and   WARNING:core:handle_timer_job: utimer job <tm-utimer> has a 30000
>> us delay in execution
>> I had 4 children and I upped it to 15, I think it helped but not
>> completely solved the problem. When running top I see all of the sudden a
>> few opensips processes jumping to 100% and the idle resources slowly going
>> down sometimes even to 0% then it recovers, I haven't changed anything in
>> my cfg file lately and the traffic has not increased etc. Is there anyway
>> to see in the logs whats taking so much resources? I'm also wondering if
>> its can be a DigitalOcean issue where other users on the same Hyperviser
>> are using very high resources and its stealing my resources.
>> thank you
>>
>> _______________________________________________
>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>> _______________________________________________ Users mailing list
>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>> bin/mailman/listinfo/users
>
> _______________________________________________
> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20161221/9c0e67a9/attachment-0001.html>


More information about the Users mailing list