[OpenSIPS-Users] Major performance issues question
Schneur Rosenberg
rosenberg11219 at gmail.com
Wed Dec 21 13:22:36 EST 2016
No this server is running only OpenSIPS and mysql, here is a report of
iotop which I took when the resources was going down
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 78.77 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
153 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.08 % [jbd2/vda1-8]
1681 be/4 mysql 0.00 B/s 7.88 K/s 0.00 % 0.06 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
1711 be/4 mysql 0.00 B/s 3.94 K/s 0.00 % 0.05 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
647 be/4 root 0.00 B/s 3.94 K/s 0.00 % 0.00 % rsyslogd -n
[rs:main Q:Reg]
2048 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
2052 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
6 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/u16:0]
7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
9 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
10 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0]
11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1]
12 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1]
13 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1]
1038 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
15 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H]
16 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/2]
17 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/2]
18 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/2]
2051 be/4 opensips 0.00 B/s 0.00 B/s 0.00 % 0.00 % opensips -P
/var/run/opensips/opensips.pid -m 128 -M 4 -u opensips -g opensips -f
/etc/opensips/opensips.cfg
20 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/2:0H]
21 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/3]
22 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/3]
23 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/3]
2072 be/4 mysql 0.00 B/s 0.00 B/s 0.00 % 0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
25 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/3:0H]
26 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/4]
27 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/4]
28 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/4]
2077 be/4 opensips 0.00 B/s 0.00 B/s 0.00 % 0.00 % opensips -P
/var/run/opensips/opensips.pid -m 128 -M 4 -u opensips -g opensips -f
/etc/opensips/opensips.cfg
30 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/4:0H]
31 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/5]
32 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/5]
33 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/5]
34 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/5:0]
35 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/5:0H]
36 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/6]
On Wed, Dec 21, 2016 at 7:25 PM, Liviu Chircu <liviu at opensips.org> wrote:
> Okay, so it seems lots of CPU cycles are spent in the kernel!
>
> Are you sure there isn't any intensive I/O going on? For example, maybe
> you are pushing lots of gateway pings. You could run "iostat 1" to assess
> this.
>
> Regards,
>
> Liviu Chircu
> OpenSIPS Developerhttp://www.opensips-solutions.com
>
> On 21.12.2016 18:58, Schneur Rosenberg wrote:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ------cpu-----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa st
> 7 0 0 9966412 296728 5239152 0 0 3 60 0 0 0 0
> 99 0 0
> 6 0 0 9966684 296728 5239152 0 0 0 20 2075 1569 12 64
> 24 0 0
> 7 0 0 9966588 296728 5239152 0 0 0 20 2023 1408 12 64
> 24 0 0
> 4 0 0 9966492 296728 5239152 0 0 0 108 2027 1434 11 63
> 26 0 0
> 5 0 0 9966588 296728 5239152 0 0 0 44 1687 1232 11 52
> 38 0 0
> 5 0 0 9966620 296728 5239156 0 0 0 72 1704 1136 9 53
> 37 0 0
> 6 0 0 9966492 296728 5239156 0 0 0 40 1874 1133 11 64
> 25 0 0
> 6 0 0 9966524 296728 5239156 0 0 0 20 1794 853 11 65
> 25 0 0
> 6 0 0 9966556 296728 5239156 0 0 0 56 1810 951 11 64
> 25 0 0
> 6 0 0 9966620 296728 5239156 0 0 0 0 1802 934 10 65
> 25 0 0
> 7 0 0 9966556 296728 5239160 0 0 0 2328 2356 1489 12 76
> 12 0 0
> 7 0 0 9966460 296728 5239160 0 0 0 0 2074 929 12 76
> 12 0 0
> 7 0 0 9966588 296728 5239164 0 0 0 0 2022 728 11 76
> 12 0 0
> 7 0 0 9966620 296728 5239164 0 0 0 0 2015 802 14 74
> 12 0 0
> 5 0 0 9966652 296728 5239164 0 0 0 72 2275 1049 14 81
> 5 0 0
>
>
> On Wed, Dec 21, 2016 at 6:42 PM, Liviu Chircu <liviu at opensips.org> wrote:
>
>> Hi Schneur,
>>
>> Those warnings are due to a rather harmless bug, which was fixed and
>> backported ~ two weeks ago.
>>
>> Regarding 100% CPU usage though, could you please run "vmstat 1" for 10
>> seconds and paste the full output? Thanks!
>>
>> Regards,
>>
>> Liviu Chircu
>> OpenSIPS Developerhttp://www.opensips-solutions.com
>>
>> On 21.12.2016 18:17, Schneur Rosenberg wrote:
>>
>> Hi, I'm running a instance of OpenSIPS (just signalling no RTP on this
>> machine) on a DigitalOcean VM, it was running fine for a while and it does
>> not process lots of CPS (4-5 max) and today I was getting complaints of
>> dropped calls, I checked the logs and I saw lots of
>> WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for
>> 3956760 ms (now 3956860 ms), it may overlap
>> and WARNING:core:handle_timer_job: utimer job <tm-utimer> has a 30000
>> us delay in execution
>> I had 4 children and I upped it to 15, I think it helped but not
>> completely solved the problem. When running top I see all of the sudden a
>> few opensips processes jumping to 100% and the idle resources slowly going
>> down sometimes even to 0% then it recovers, I haven't changed anything in
>> my cfg file lately and the traffic has not increased etc. Is there anyway
>> to see in the logs whats taking so much resources? I'm also wondering if
>> its can be a DigitalOcean issue where other users on the same Hyperviser
>> are using very high resources and its stealing my resources.
>> thank you
>>
>> _______________________________________________
>> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>> _______________________________________________ Users mailing list
>> Users at lists.opensips.org http://lists.opensips.org/cgi-
>> bin/mailman/listinfo/users
>
> _______________________________________________
> Users mailing listUsers at lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20161221/9c0e67a9/attachment-0001.html>
More information about the Users
mailing list