== sanity test 398c: run fio to test AIO ================= 05:00:50 (1734688850) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.218202 s, 192 MB/s osc.lustre-OST0000-osc-ffff8800b3674000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=28661: Fri Dec 20 05:01:09 2024 write: IOPS=159, BW=639KiB/s (655kB/s)(10.0MiB/16021msec) slat (usec): min=12, max=10999, avg=122.37, stdev=417.57 clat (msec): min=7, max=163, avg=99.30, stdev=22.16 lat (msec): min=8, max=164, avg=99.42, stdev=22.16 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 64], 10.00th=[ 71], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 101], 60.00th=[ 104], | 70.00th=[ 110], 80.00th=[ 116], 90.00th=[ 130], 95.00th=[ 138], | 99.00th=[ 157], 99.50th=[ 159], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 165] bw ( KiB/s): min= 391, max= 910, per=24.83%, avg=634.77, stdev=115.78, samples=31 iops : min= 97, max= 227, avg=158.48, stdev=28.99, samples=31 lat (msec) : 10=0.08%, 20=0.04%, 50=0.35%, 100=50.16%, 250=49.38% cpu : usr=0.09%, sys=2.53%, ctx=931, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28662: Fri Dec 20 05:01:09 2024 write: IOPS=159, BW=639KiB/s (655kB/s)(10.0MiB/16017msec) slat (usec): min=12, max=5415, avg=112.61, stdev=300.06 clat (msec): min=13, max=187, avg=99.41, stdev=21.98 lat (msec): min=13, max=187, avg=99.53, stdev=22.00 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 65], 10.00th=[ 72], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 105], | 70.00th=[ 110], 80.00th=[ 117], 90.00th=[ 129], 95.00th=[ 138], | 99.00th=[ 159], 99.50th=[ 163], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 188] bw ( KiB/s): min= 423, max= 896, per=24.87%, avg=635.71, stdev=111.76, samples=31 iops : min= 105, max= 224, avg=158.81, stdev=27.97, samples=31 lat (msec) : 20=0.04%, 50=0.16%, 100=52.15%, 250=47.66% cpu : usr=0.14%, sys=2.49%, ctx=897, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28663: Fri Dec 20 05:01:09 2024 write: IOPS=160, BW=642KiB/s (658kB/s)(10.0MiB/15939msec) slat (usec): min=11, max=3819, avg=116.99, stdev=284.21 clat (msec): min=20, max=203, avg=99.13, stdev=23.05 lat (msec): min=22, max=203, avg=99.25, stdev=23.06 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 99], 60.00th=[ 105], | 70.00th=[ 110], 80.00th=[ 120], 90.00th=[ 128], 95.00th=[ 138], | 99.00th=[ 157], 99.50th=[ 165], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 203] bw ( KiB/s): min= 502, max= 870, per=25.13%, avg=642.23, stdev=107.35, samples=31 iops : min= 125, max= 217, avg=160.42, stdev=26.85, samples=31 lat (msec) : 50=1.25%, 100=51.21%, 250=47.54% cpu : usr=0.58%, sys=2.33%, ctx=984, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28664: Fri Dec 20 05:01:09 2024 write: IOPS=160, BW=641KiB/s (656kB/s)(10.0MiB/15974msec) slat (usec): min=15, max=4145, avg=146.21, stdev=346.55 clat (msec): min=14, max=206, avg=99.23, stdev=23.45 lat (msec): min=14, max=206, avg=99.38, stdev=23.45 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 79], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 104], | 70.00th=[ 110], 80.00th=[ 117], 90.00th=[ 129], 95.00th=[ 138], | 99.00th=[ 163], 99.50th=[ 169], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 207] bw ( KiB/s): min= 512, max= 902, per=25.08%, avg=641.10, stdev=108.81, samples=31 iops : min= 128, max= 225, avg=160.10, stdev=27.19, samples=31 lat (msec) : 20=0.08%, 50=1.13%, 100=52.19%, 250=46.60% cpu : usr=0.69%, sys=2.56%, ctx=1376, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2557KiB/s (2618kB/s), 639KiB/s-642KiB/s (655kB/s-658kB/s), io=40.0MiB (41.9MB), run=15939-16021msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28687: Fri Dec 20 05:01:24 2024 read: IOPS=96, BW=384KiB/s (393kB/s)(5048KiB/13142msec) slat (usec): min=1252, max=20433, avg=5212.46, stdev=2263.43 clat (msec): min=33, max=156, avg=76.75, stdev=18.44 lat (msec): min=36, max=176, avg=81.97, stdev=19.19 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 81], | 70.00th=[ 86], 80.00th=[ 93], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 126], 99.50th=[ 133], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 239, max= 544, per=24.90%, avg=379.65, stdev=77.07, samples=26 iops : min= 59, max= 136, avg=94.65, stdev=19.30, samples=26 write: IOPS=98, BW=395KiB/s (405kB/s)(5192KiB/13142msec) slat (usec): min=16, max=7377, avg=124.10, stdev=308.52 clat (msec): min=31, max=151, avg=81.04, stdev=18.53 lat (msec): min=32, max=151, avg=81.16, stdev=18.53 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 65], | 30.00th=[ 69], 40.00th=[ 74], 50.00th=[ 80], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 107], 95.00th=[ 114], | 99.00th=[ 127], 99.50th=[ 140], 99.90th=[ 148], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 296, max= 487, per=24.72%, avg=392.58, stdev=46.63, samples=26 iops : min= 74, max= 121, avg=97.88, stdev=11.56, samples=26 lat (msec) : 50=2.70%, 100=82.46%, 250=14.84% cpu : usr=1.13%, sys=5.30%, ctx=2444, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28688: Fri Dec 20 05:01:24 2024 read: IOPS=94, BW=378KiB/s (387kB/s)(4976KiB/13156msec) slat (usec): min=1456, max=25956, avg=5230.07, stdev=2306.37 clat (msec): min=8, max=147, avg=77.97, stdev=20.48 lat (msec): min=10, max=167, avg=83.20, stdev=21.26 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 51], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 77], 60.00th=[ 82], | 70.00th=[ 89], 80.00th=[ 96], 90.00th=[ 106], 95.00th=[ 114], | 99.00th=[ 129], 99.50th=[ 136], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 272, max= 496, per=24.48%, avg=373.27, stdev=64.23, samples=26 iops : min= 68, max= 124, avg=93.12, stdev=16.12, samples=26 write: IOPS=100, BW=400KiB/s (410kB/s)(5264KiB/13156msec) slat (usec): min=14, max=3970, avg=129.80, stdev=269.49 clat (msec): min=7, max=151, avg=80.05, stdev=18.90 lat (msec): min=8, max=151, avg=80.19, stdev=18.90 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 79], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 96], 90.00th=[ 106], 95.00th=[ 112], | 99.00th=[ 129], 99.50th=[ 142], 99.90th=[ 148], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 247, max= 552, per=24.80%, avg=393.88, stdev=66.36, samples=26 iops : min= 61, max= 138, avg=98.27, stdev=16.64, samples=26 lat (msec) : 10=0.08%, 20=0.27%, 50=3.55%, 100=80.43%, 250=15.66% cpu : usr=1.13%, sys=5.12%, ctx=2344, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28689: Fri Dec 20 05:01:24 2024 read: IOPS=94, BW=380KiB/s (389kB/s)(4984KiB/13117msec) slat (usec): min=19, max=26882, avg=5294.86, stdev=2390.69 clat (msec): min=10, max=156, avg=78.53, stdev=20.51 lat (msec): min=10, max=163, avg=83.83, stdev=21.25 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 50], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 89], 80.00th=[ 96], 90.00th=[ 108], 95.00th=[ 114], | 99.00th=[ 130], 99.50th=[ 140], 99.90th=[ 144], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 262, max= 504, per=24.83%, avg=378.69, stdev=63.61, samples=26 iops : min= 65, max= 126, avg=94.46, stdev=16.03, samples=26 write: IOPS=100, BW=401KiB/s (410kB/s)(5256KiB/13117msec) slat (usec): min=16, max=2679, avg=117.05, stdev=228.75 clat (msec): min=38, max=162, avg=79.88, stdev=18.10 lat (msec): min=38, max=162, avg=80.00, stdev=18.10 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 55], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 73], 50.00th=[ 79], 60.00th=[ 84], | 70.00th=[ 89], 80.00th=[ 96], 90.00th=[ 106], 95.00th=[ 112], | 99.00th=[ 124], 99.50th=[ 129], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 256, max= 542, per=25.03%, avg=397.42, stdev=79.58, samples=26 iops : min= 64, max= 135, avg=99.15, stdev=19.93, samples=26 lat (msec) : 20=0.12%, 50=3.79%, 100=80.59%, 250=15.51% cpu : usr=0.80%, sys=5.32%, ctx=2366, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28690: Fri Dec 20 05:01:24 2024 read: IOPS=96, BW=385KiB/s (394kB/s)(5060KiB/13144msec) slat (usec): min=1182, max=26646, avg=5321.28, stdev=2321.69 clat (msec): min=12, max=155, avg=78.30, stdev=19.76 lat (msec): min=18, max=161, avg=83.62, stdev=20.50 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 51], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 67], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 89], 80.00th=[ 95], 90.00th=[ 105], 95.00th=[ 113], | 99.00th=[ 126], 99.50th=[ 134], 99.90th=[ 150], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 239, max= 504, per=25.04%, avg=381.85, stdev=60.69, samples=26 iops : min= 59, max= 126, avg=95.35, stdev=15.29, samples=26 write: IOPS=98, BW=394KiB/s (404kB/s)(5180KiB/13144msec) slat (usec): min=17, max=4578, avg=119.21, stdev=280.16 clat (msec): min=27, max=148, avg=79.78, stdev=18.86 lat (msec): min=27, max=148, avg=79.90, stdev=18.86 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 90], 80.00th=[ 97], 90.00th=[ 107], 95.00th=[ 112], | 99.00th=[ 127], 99.50th=[ 138], 99.90th=[ 146], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 296, max= 560, per=24.57%, avg=390.15, stdev=74.95, samples=26 iops : min= 74, max= 140, avg=97.42, stdev=18.72, samples=26 lat (msec) : 20=0.08%, 50=3.59%, 100=80.16%, 250=16.17% cpu : usr=0.62%, sys=5.68%, ctx=2388, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1525KiB/s (1562kB/s), 378KiB/s-385KiB/s (387kB/s-394kB/s), io=19.6MiB (20.5MB), run=13117-13156msec WRITE: bw=1588KiB/s (1626kB/s), 394KiB/s-401KiB/s (404kB/s-410kB/s), io=20.4MiB (21.4MB), run=13117-13156msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=28705: Fri Dec 20 05:01:26 2024 read: IOPS=3, BW=147MiB/s (154MB/s)(40.0MiB/273msec) slat (nsec): min=9501.7k, max=9501.7k, avg=9501734.00, stdev= 0.00 clat (nsec): min=255318k, max=255318k, avg=255317597.00, stdev= 0.00 lat (nsec): min=264846k, max=264846k, avg=264846499.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.94%, ctx=12, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=40.0MiB (41.9MB), run=273-273msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout