Services validation aptitude
Sauter à la navigation
Sauter à la recherche
Le titre est bonnie mais en réalité, c'est un résumé de la session de tests de perfs.
=====================================================
== Ma machine pour comparer =========================
=====================================================
fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio => 118489KB/s
hdparm -t => 120.19 MB/sec (Mécanique)
hdparm -t => 383.54 MB/sec (SSD)
=====================================================
=== Host (services) =================================
=====================================================
# Config
Les disques de services sont des Western Caviar Green de 1.5To, ayant tous pour ref WDC WD15EADS-00P8B0 (fw: 01.00A01) (hdparm -i)
Au niveau raid c'est du classique RAID5 (3 disques) via mdadm.
/dev/md1:
Version : 1.2
Creation Time : Mon Jul 15 10:35:52 2013
Raid Level : raid5
Array Size : 2929816576 (2794.09 GiB 3000.13 GB)
Used Dev Size : 1464908288 (1397.05 GiB 1500.07 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Wed Apr 23 11:05:35 2014
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : services:1
UUID : 1ce82a81:c375123b:3660348b:6963a293
Events : 274
Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 5 1 active sync /dev/sda5
2 8 21 2 active sync /dev/sdb5
Au niveau tunning raid, le Read ahead à été modifié (256 valeur par défaut)
$ blockdev --setra 4096 /dev/md1
Et le stripe_cache_size à été aussi modifié
$ cat /sys/block/md1/md/stripe_cache_size
8192
# Tests
hdparm -t
-------------------------------
sda 84.89 MB/sec
sdb 95.32 MB/sec
sdc 91.49 MB/sec
services-root 171.78MB/sec
fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio : 2188.4MB/s (!!) puis 175994KB/s et 243656KB/s
fio --size=2g --bs=64k --rw=read --ioengine=libaio --name=fio : 157929KB/s, 169302KB/s, 173691KB/s, 164250KB/s
gcorona@services:~/temp$ sudo chown gcorona .
gcorona@services:~/temp$ /usr/sbin/bonnie++ -s 50g -d $(pwd)
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
services.ldn-fa 50G 2468 87 97946 7 43630 2 3235 54 163693 4 254.1 8
Latency 16586us 3616ms 2074ms 307ms 237ms 153ms
Version 1.96 ------Sequential Create------ --------Random Create--------
services.ldn-fai.ne -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 4071 33 +++++ +++ 13655 2 11034 83 +++++ +++ +++++ +++
Latency 12353us 129us 72us 29677us 51us 366us
1.96,1.96,services.ldn-fai.net,1,1398185044,50G,,2468,87,97946,7,43630,2,3235,54,163693,4,254.1,8,16,,,,,4071,33,+++++,+++,13655,2,11034,83,+++++,+++,+++++,+++,16586us,3616ms,2074ms,307ms,237ms,153ms,12353us,129us,72us,29677us,51us,366us
=====================================================
=== Depuis la VM (WTF???) ===
=====================================================
dvb est une partition avec une seule couche de LVM
/dev/vdb:
Timing buffered disk reads: 352 MB in 3.06 seconds = 114.90 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vda
/dev/vda:
Timing buffered disk reads: 464 MB in 3.01 seconds = 154.15 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vda
/dev/vda:
Timing buffered disk reads: 460 MB in 3.01 seconds = 152.83 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 622 MB in 3.02 seconds = 205.67 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 1000 MB in 3.01 seconds = 332.63 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 1252 MB in 3.04 seconds = 412.51 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 1558 MB in 3.05 seconds = 510.64 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 1722 MB in 3.01 seconds = 571.82 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 1972 MB in 3.02 seconds = 653.72 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 2118 MB in 3.00 seconds = 705.32 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 2324 MB in 3.00 seconds = 773.74 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb
/dev/vdb:
Timing buffered disk reads: 2552 MB in 3.02 seconds = 845.06 MB/sec
gcorona@blender:~/temp$ sudo hdparm -t /dev/vda
gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio
fio: (g=0): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 process
fio: Laying out IO file(s) (1 file(s) / 2048MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/27300K /s] [0 /426 iops] [eta 00m:00s]
fio: (groupid=0, jobs=1): err= 0: pid=25865
write: io=2048.0MB, bw=25773KB/s, iops=402 , runt= 81371msec
slat (usec): min=15 , max=464309 , avg=2477.75, stdev=5778.96
clat (usec): min=0 , max=2234 , avg= 1.10, stdev=12.37
lat (usec): min=16 , max=464311 , avg=2480.10, stdev=5780.05
clat percentiles (usec):
| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],
| 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1],
| 70.00th=[ 1], 80.00th=[ 2], 90.00th=[ 2], 95.00th=[ 2],
| 99.00th=[ 3], 99.50th=[ 4], 99.90th=[ 10], 99.95th=[ 11],
| 99.99th=[ 29]
bw (KB/s) : min= 4353, max=189849, per=100.00%, avg=25810.33, stdev=21757.74
lat (usec) : 2=78.34%, 4=21.14%, 10=0.33%, 20=0.16%, 50=0.02%
lat (usec) : 100=0.01%
lat (msec) : 4=0.01%
cpu : usr=0.08%, sys=1.83%, ctx=12144, majf=1, minf=20
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=32768/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2048.0MB, aggrb=25772KB/s, minb=25772KB/s, maxb=25772KB/s, mint=81371msec, maxt=81371msec
Disk stats (read/write):
vdb: ios=0/4289, merge=0/103, ticks=0/12596244, in_queue=12592400, util=100.00%
gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio
fio: (g=0): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/19884K /s] [0 /310 iops] [eta 00m:00s]
fio: (groupid=0, jobs=1): err= 0: pid=26397
write: io=2048.0MB, bw=19908KB/s, iops=311 , runt=105344msec
slat (usec): min=15 , max=130181 , avg=2726.25, stdev=5445.86
clat (usec): min=0 , max=121 , avg= 1.02, stdev= 1.35
lat (usec): min=15 , max=130183 , avg=2728.25, stdev=5447.08
clat percentiles (usec):
| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],
| 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1],
| 70.00th=[ 1], 80.00th=[ 2], 90.00th=[ 2], 95.00th=[ 2],
| 99.00th=[ 3], 99.50th=[ 3], 99.90th=[ 10], 99.95th=[ 27],
| 99.99th=[ 36]
bw (KB/s) : min= 4, max=804844, per=100.00%, avg=23385.34, stdev=59855.44
lat (usec) : 2=79.57%, 4=19.97%, 10=0.30%, 20=0.09%, 50=0.06%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=0.05%, sys=1.43%, ctx=7570, majf=0, minf=22
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=32768/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2048.0MB, aggrb=19907KB/s, minb=19907KB/s, maxb=19907KB/s, mint=105344msec, maxt=105344msec
Disk stats (read/write):
vdb: ios=0/3740, merge=0/98, ticks=0/14862032, in_queue=15315168, util=100.00%
gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=read --ioengine=libaio --name=fio
fio: (g=0): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 process
Jobs: 1 (f=1)
fio: (groupid=0, jobs=1): err= 0: pid=28358
read : io=2048.0MB, bw=2000.0MB/s, iops=32000 , runt= 1024msec
slat (usec): min=6 , max=3004 , avg=26.12, stdev=53.72
clat (usec): min=0 , max=175 , avg= 0.71, stdev= 1.65
lat (usec): min=6 , max=3007 , avg=27.24, stdev=53.99
clat percentiles (usec):
| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],
| 30.00th=[ 0], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1],
| 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 1],
| 99.00th=[ 2], 99.50th=[ 5], 99.90th=[ 21], 99.95th=[ 23],
| 99.99th=[ 35]
bw (MB/s) : min= 1884, max= 2217, per=100.00%, avg=2051.01, stdev=235.96
lat (usec) : 2=97.81%, 4=1.68%, 10=0.10%, 20=0.28%, 50=0.12%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=7.43%, sys=79.37%, ctx=7342, majf=0, minf=38
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=32768/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=2048.0MB, aggrb=2000.0MB/s, minb=2000.0MB/s, maxb=2000.0MB/s, mint=1024msec, maxt=1024msec
Disk stats (read/write):
vdb: ios=7828/0, merge=0/0, ticks=724/0, in_queue=720, util=63.33%
gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=read --ioengine=libaio --name=fio
fio: (g=0): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1
2.0.8
Starting 1 process
Jobs: 1 (f=1)
fio: (groupid=0, jobs=1): err= 0: pid=28381
read : io=2048.0MB, bw=1961.8MB/s, iops=31386 , runt= 1044msec
slat (usec): min=5 , max=37994 , avg=26.70, stdev=215.98
clat (usec): min=0 , max=52 , avg= 0.69, stdev= 1.39
lat (usec): min=6 , max=37997 , avg=27.77, stdev=216.04
clat percentiles (usec):
| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],
| 30.00th=[ 0], 40.00th=[ 0], 50.00th=[ 1], 60.00th=[ 1],
| 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 1],
| 99.00th=[ 2], 99.50th=[ 9], 99.90th=[ 20], 99.95th=[ 23],
| 99.99th=[ 32]
bw (MB/s) : min= 1745, max= 2284, per=100.00%, avg=2015.10, stdev=381.59
lat (usec) : 2=98.46%, 4=0.99%, 10=0.10%, 20=0.30%, 50=0.13%
lat (usec) : 100=0.01%
cpu : usr=6.14%, sys=76.32%, ctx=7542, majf=0, minf=37
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=32768/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=2048.0MB, aggrb=1961.8MB/s, minb=1961.8MB/s, maxb=1961.8MB/s, mint=1044msec, maxt=1044msec
Disk stats (read/write):
vdb: ios=7664/0, merge=0/0, ticks=796/0, in_queue=788, util=63.26%
======================================================
== Résultats précédents (avec les VMS qui tournent) ==
======================================================
FS \ système | host guest
---------------------+--------------------
host (au dessus de 4)|87029KB/s ?
guest (8) |71712KB/s 46717KB/s
Résultats de "fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio"
FS \ système |host guest
---------------------+-------------------
host (au dessus de 4)|05.71s ?
guest (8) |06.27s 35.s
Résultats de "time dd bs=1M count=256 if=/dev/zero of=ff.out conv=fdatasync" (durée)
FS \ système |host guest
---------------------+-------------------
host (au dessus de 4)|45MB/s ?
guest (8) |41MB/s 7MB/s
Résultats de "time dd bs=1M count=256 if=/dev/zero of=ff.out conv=fdatasync" (débit)
NCQ
cat /sys/block/sdX/device/queue_depth if you get a number greater than 1 it is enabled enable via echo 31 > /sys/block/sdX/device/queue_depth disable via echo 1 > /sys/block/sdX/device/queue_depth
Tunning
- https://ata.wiki.kernel.org/index.php/Libata_FAQ
- http://chiliproject.tetaneutral.net/projects/tetaneutral/wiki/Cluster_Ganeti
#!bin/sh
# http://chiliproject.tetaneutral.net/projects/tetaneutral/wiki/Cluster_Ganeti
KVM_DISKS="WDC_WD15EADS-00_WD-WCAVU0237611 WDC_WD15EADS-00_WD-WMAVU0297159 WDC_WD15EADS-00_WD-WMAVU0892520"
for diskname in $KVM_DISKS ; do
disk=$(basename $(readlink -e /dev/disk/by-id/scsi-SATA_$diskname) 2>/dev/null)
[ -z "$disk" -o ! -d "/sys/block/$disk" ] && continue
echo deadline > /sys/block/${disk}/queue/scheduler
echo 1 > /sys/block/${disk}/queue/iosched/fifo_batch
echo 0 > /sys/block/${disk}/queue/iosched/front_merges
echo 2 > /sys/block/${disk}/queue/iosched/read_expire
echo 2 > /sys/block/${disk}/queue/iosched/write_expire
echo 1 > /sys/block/${disk}/queue/iosched/writes_starved
done
Tests
- test avant (cfq)
root@graphite-4:~# cat /sys/block/sda/queue/scheduler noop deadline [cfq] root@graphite-4:/tmp/foo# time tar xzf wheezy-x64-prod-1.7.tgz real 0m48.315s user 0m44.711s sys 0m11.737s root@graphite-4:/tmp/foo# du -sh wheezy-x64-prod-1.7.tgz 1.1G wheezy-x64-prod-1.7.tgz
- changement de scheduler
root@graphite-4:~# bash tunning.sh + KVM_DISKS='wwn-0x50015178f36802eb wwn-0x50015178f368041d' + for diskname in '$KVM_DISKS' +++ readlink -e /dev/disk/by-id/wwn-0x50015178f36802eb ++ basename /dev/sda + disk=sda + '[' -z sda -o '!' -d /sys/block/sda ']' + echo deadline + echo 1 + echo 0 + echo 2 + echo 2 + echo 1 + for diskname in '$KVM_DISKS' +++ readlink -e /dev/disk/by-id/wwn-0x50015178f368041d ++ basename /dev/sdb + disk=sdb + '[' -z sdb -o '!' -d /sys/block/sdb ']' + echo deadline + echo 1 + echo 0 + echo 2 + echo 2 + echo 1 root@graphite-4:~# cat /sys/block/sda/queue/scheduler noop [deadline] cfq
- tests deadline
root@graphite-4:/tmp/foo2# sync root@graphite-4:/tmp/foo2# time tar xzf wheezy-x64-prod-1.7.tgz real 0m47.260s user 0m45.443s sys 0m12.101s
Benchmark scheduler
Recommandations KVM d'IBM
Source :