Services validation aptitude
Sauter à la navigation
Sauter à la recherche
Le titre est bonnie mais en réalité, c'est un résumé de la session de tests de perfs. ===================================================== == Ma machine pour comparer ========================= ===================================================== fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio => 118489KB/s hdparm -t => 120.19 MB/sec (Mécanique) hdparm -t => 383.54 MB/sec (SSD) ===================================================== === Host (services) ================================= ===================================================== # Config Les disques de services sont des Western Caviar Green de 1.5To, ayant tous pour ref WDC WD15EADS-00P8B0 (fw: 01.00A01) (hdparm -i) Au niveau raid c'est du classique RAID5 (3 disques) via mdadm. /dev/md1: Version : 1.2 Creation Time : Mon Jul 15 10:35:52 2013 Raid Level : raid5 Array Size : 2929816576 (2794.09 GiB 3000.13 GB) Used Dev Size : 1464908288 (1397.05 GiB 1500.07 GB) Raid Devices : 3 Total Devices : 3 Persistence : Superblock is persistent Update Time : Wed Apr 23 11:05:35 2014 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : services:1 UUID : 1ce82a81:c375123b:3660348b:6963a293 Events : 274 Number Major Minor RaidDevice State 0 8 34 0 active sync /dev/sdc2 1 8 5 1 active sync /dev/sda5 2 8 21 2 active sync /dev/sdb5 Au niveau tunning raid, le Read ahead à été modifié (256 valeur par défaut) $ blockdev --setra 4096 /dev/md1 Et le stripe_cache_size à été aussi modifié $ cat /sys/block/md1/md/stripe_cache_size 8192 # Tests hdparm -t ------------------------------- sda 84.89 MB/sec sdb 95.32 MB/sec sdc 91.49 MB/sec services-root 171.78MB/sec fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio : 2188.4MB/s (!!) puis 175994KB/s et 243656KB/s fio --size=2g --bs=64k --rw=read --ioengine=libaio --name=fio : 157929KB/s, 169302KB/s, 173691KB/s, 164250KB/s gcorona@services:~/temp$ sudo chown gcorona . gcorona@services:~/temp$ /usr/sbin/bonnie++ -s 50g -d $(pwd) Writing a byte at a time...done Writing intelligently...done Rewriting...done Reading a byte at a time...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP services.ldn-fa 50G 2468 87 97946 7 43630 2 3235 54 163693 4 254.1 8 Latency 16586us 3616ms 2074ms 307ms 237ms 153ms Version 1.96 ------Sequential Create------ --------Random Create-------- services.ldn-fai.ne -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 4071 33 +++++ +++ 13655 2 11034 83 +++++ +++ +++++ +++ Latency 12353us 129us 72us 29677us 51us 366us 1.96,1.96,services.ldn-fai.net,1,1398185044,50G,,2468,87,97946,7,43630,2,3235,54,163693,4,254.1,8,16,,,,,4071,33,+++++,+++,13655,2,11034,83,+++++,+++,+++++,+++,16586us,3616ms,2074ms,307ms,237ms,153ms,12353us,129us,72us,29677us,51us,366us ===================================================== === Depuis la VM (WTF???) === ===================================================== dvb est une partition avec une seule couche de LVM /dev/vdb: Timing buffered disk reads: 352 MB in 3.06 seconds = 114.90 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vda /dev/vda: Timing buffered disk reads: 464 MB in 3.01 seconds = 154.15 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vda /dev/vda: Timing buffered disk reads: 460 MB in 3.01 seconds = 152.83 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 622 MB in 3.02 seconds = 205.67 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 1000 MB in 3.01 seconds = 332.63 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 1252 MB in 3.04 seconds = 412.51 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 1558 MB in 3.05 seconds = 510.64 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 1722 MB in 3.01 seconds = 571.82 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 1972 MB in 3.02 seconds = 653.72 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 2118 MB in 3.00 seconds = 705.32 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 2324 MB in 3.00 seconds = 773.74 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vdb /dev/vdb: Timing buffered disk reads: 2552 MB in 3.02 seconds = 845.06 MB/sec gcorona@blender:~/temp$ sudo hdparm -t /dev/vda gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio fio: (g=0): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 2.0.8 Starting 1 process fio: Laying out IO file(s) (1 file(s) / 2048MB) Jobs: 1 (f=1): [W] [100.0% done] [0K/27300K /s] [0 /426 iops] [eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=25865 write: io=2048.0MB, bw=25773KB/s, iops=402 , runt= 81371msec slat (usec): min=15 , max=464309 , avg=2477.75, stdev=5778.96 clat (usec): min=0 , max=2234 , avg= 1.10, stdev=12.37 lat (usec): min=16 , max=464311 , avg=2480.10, stdev=5780.05 clat percentiles (usec): | 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0], | 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1], | 70.00th=[ 1], 80.00th=[ 2], 90.00th=[ 2], 95.00th=[ 2], | 99.00th=[ 3], 99.50th=[ 4], 99.90th=[ 10], 99.95th=[ 11], | 99.99th=[ 29] bw (KB/s) : min= 4353, max=189849, per=100.00%, avg=25810.33, stdev=21757.74 lat (usec) : 2=78.34%, 4=21.14%, 10=0.33%, 20=0.16%, 50=0.02% lat (usec) : 100=0.01% lat (msec) : 4=0.01% cpu : usr=0.08%, sys=1.83%, ctx=12144, majf=1, minf=20 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=32768/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=2048.0MB, aggrb=25772KB/s, minb=25772KB/s, maxb=25772KB/s, mint=81371msec, maxt=81371msec Disk stats (read/write): vdb: ios=0/4289, merge=0/103, ticks=0/12596244, in_queue=12592400, util=100.00% gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio fio: (g=0): rw=write, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 2.0.8 Starting 1 process Jobs: 1 (f=1): [W] [100.0% done] [0K/19884K /s] [0 /310 iops] [eta 00m:00s] fio: (groupid=0, jobs=1): err= 0: pid=26397 write: io=2048.0MB, bw=19908KB/s, iops=311 , runt=105344msec slat (usec): min=15 , max=130181 , avg=2726.25, stdev=5445.86 clat (usec): min=0 , max=121 , avg= 1.02, stdev= 1.35 lat (usec): min=15 , max=130183 , avg=2728.25, stdev=5447.08 clat percentiles (usec): | 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0], | 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1], | 70.00th=[ 1], 80.00th=[ 2], 90.00th=[ 2], 95.00th=[ 2], | 99.00th=[ 3], 99.50th=[ 3], 99.90th=[ 10], 99.95th=[ 27], | 99.99th=[ 36] bw (KB/s) : min= 4, max=804844, per=100.00%, avg=23385.34, stdev=59855.44 lat (usec) : 2=79.57%, 4=19.97%, 10=0.30%, 20=0.09%, 50=0.06% lat (usec) : 100=0.01%, 250=0.01% cpu : usr=0.05%, sys=1.43%, ctx=7570, majf=0, minf=22 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=32768/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=2048.0MB, aggrb=19907KB/s, minb=19907KB/s, maxb=19907KB/s, mint=105344msec, maxt=105344msec Disk stats (read/write): vdb: ios=0/3740, merge=0/98, ticks=0/14862032, in_queue=15315168, util=100.00% gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=read --ioengine=libaio --name=fio fio: (g=0): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 2.0.8 Starting 1 process Jobs: 1 (f=1) fio: (groupid=0, jobs=1): err= 0: pid=28358 read : io=2048.0MB, bw=2000.0MB/s, iops=32000 , runt= 1024msec slat (usec): min=6 , max=3004 , avg=26.12, stdev=53.72 clat (usec): min=0 , max=175 , avg= 0.71, stdev= 1.65 lat (usec): min=6 , max=3007 , avg=27.24, stdev=53.99 clat percentiles (usec): | 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0], | 30.00th=[ 0], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1], | 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 1], | 99.00th=[ 2], 99.50th=[ 5], 99.90th=[ 21], 99.95th=[ 23], | 99.99th=[ 35] bw (MB/s) : min= 1884, max= 2217, per=100.00%, avg=2051.01, stdev=235.96 lat (usec) : 2=97.81%, 4=1.68%, 10=0.10%, 20=0.28%, 50=0.12% lat (usec) : 100=0.01%, 250=0.01% cpu : usr=7.43%, sys=79.37%, ctx=7342, majf=0, minf=38 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=32768/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=2048.0MB, aggrb=2000.0MB/s, minb=2000.0MB/s, maxb=2000.0MB/s, mint=1024msec, maxt=1024msec Disk stats (read/write): vdb: ios=7828/0, merge=0/0, ticks=724/0, in_queue=720, util=63.33% gcorona@blender:~/temp$ fio --size=2g --bs=64k --rw=read --ioengine=libaio --name=fio fio: (g=0): rw=read, bs=64K-64K/64K-64K, ioengine=libaio, iodepth=1 2.0.8 Starting 1 process Jobs: 1 (f=1) fio: (groupid=0, jobs=1): err= 0: pid=28381 read : io=2048.0MB, bw=1961.8MB/s, iops=31386 , runt= 1044msec slat (usec): min=5 , max=37994 , avg=26.70, stdev=215.98 clat (usec): min=0 , max=52 , avg= 0.69, stdev= 1.39 lat (usec): min=6 , max=37997 , avg=27.77, stdev=216.04 clat percentiles (usec): | 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0], | 30.00th=[ 0], 40.00th=[ 0], 50.00th=[ 1], 60.00th=[ 1], | 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 1], | 99.00th=[ 2], 99.50th=[ 9], 99.90th=[ 20], 99.95th=[ 23], | 99.99th=[ 32] bw (MB/s) : min= 1745, max= 2284, per=100.00%, avg=2015.10, stdev=381.59 lat (usec) : 2=98.46%, 4=0.99%, 10=0.10%, 20=0.30%, 50=0.13% lat (usec) : 100=0.01% cpu : usr=6.14%, sys=76.32%, ctx=7542, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=32768/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=2048.0MB, aggrb=1961.8MB/s, minb=1961.8MB/s, maxb=1961.8MB/s, mint=1044msec, maxt=1044msec Disk stats (read/write): vdb: ios=7664/0, merge=0/0, ticks=796/0, in_queue=788, util=63.26% ====================================================== == Résultats précédents (avec les VMS qui tournent) == ====================================================== FS \ système | host guest ---------------------+-------------------- host (au dessus de 4)|87029KB/s ? guest (8) |71712KB/s 46717KB/s Résultats de "fio --size=2g --bs=64k --rw=write --ioengine=libaio --name=fio" FS \ système |host guest ---------------------+------------------- host (au dessus de 4)|05.71s ? guest (8) |06.27s 35.s Résultats de "time dd bs=1M count=256 if=/dev/zero of=ff.out conv=fdatasync" (durée) FS \ système |host guest ---------------------+------------------- host (au dessus de 4)|45MB/s ? guest (8) |41MB/s 7MB/s Résultats de "time dd bs=1M count=256 if=/dev/zero of=ff.out conv=fdatasync" (débit)
NCQ
cat /sys/block/sdX/device/queue_depth if you get a number greater than 1 it is enabled enable via echo 31 > /sys/block/sdX/device/queue_depth disable via echo 1 > /sys/block/sdX/device/queue_depth
Tunning
- https://ata.wiki.kernel.org/index.php/Libata_FAQ
- http://chiliproject.tetaneutral.net/projects/tetaneutral/wiki/Cluster_Ganeti
#!bin/sh # http://chiliproject.tetaneutral.net/projects/tetaneutral/wiki/Cluster_Ganeti KVM_DISKS="WDC_WD15EADS-00_WD-WCAVU0237611 WDC_WD15EADS-00_WD-WMAVU0297159 WDC_WD15EADS-00_WD-WMAVU0892520" for diskname in $KVM_DISKS ; do disk=$(basename $(readlink -e /dev/disk/by-id/scsi-SATA_$diskname) 2>/dev/null) [ -z "$disk" -o ! -d "/sys/block/$disk" ] && continue echo deadline > /sys/block/${disk}/queue/scheduler echo 1 > /sys/block/${disk}/queue/iosched/fifo_batch echo 0 > /sys/block/${disk}/queue/iosched/front_merges echo 2 > /sys/block/${disk}/queue/iosched/read_expire echo 2 > /sys/block/${disk}/queue/iosched/write_expire echo 1 > /sys/block/${disk}/queue/iosched/writes_starved done
Tests
- test avant (cfq)
root@graphite-4:~# cat /sys/block/sda/queue/scheduler noop deadline [cfq] root@graphite-4:/tmp/foo# time tar xzf wheezy-x64-prod-1.7.tgz real 0m48.315s user 0m44.711s sys 0m11.737s root@graphite-4:/tmp/foo# du -sh wheezy-x64-prod-1.7.tgz 1.1G wheezy-x64-prod-1.7.tgz
- changement de scheduler
root@graphite-4:~# bash tunning.sh + KVM_DISKS='wwn-0x50015178f36802eb wwn-0x50015178f368041d' + for diskname in '$KVM_DISKS' +++ readlink -e /dev/disk/by-id/wwn-0x50015178f36802eb ++ basename /dev/sda + disk=sda + '[' -z sda -o '!' -d /sys/block/sda ']' + echo deadline + echo 1 + echo 0 + echo 2 + echo 2 + echo 1 + for diskname in '$KVM_DISKS' +++ readlink -e /dev/disk/by-id/wwn-0x50015178f368041d ++ basename /dev/sdb + disk=sdb + '[' -z sdb -o '!' -d /sys/block/sdb ']' + echo deadline + echo 1 + echo 0 + echo 2 + echo 2 + echo 1 root@graphite-4:~# cat /sys/block/sda/queue/scheduler noop [deadline] cfq
- tests deadline
root@graphite-4:/tmp/foo2# sync root@graphite-4:/tmp/foo2# time tar xzf wheezy-x64-prod-1.7.tgz real 0m47.260s user 0m45.443s sys 0m12.101s
Benchmark scheduler
Recommandations KVM d'IBM
Source :