== sanity test 398c: run fio to test AIO ================= 09:53:56 (1784123636) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.139813 s, 300 MB/s osc.lustre-OST0000-osc-ffff9c3fb89ce800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=159280: Wed Jul 15 09:54:28 2026 write: IOPS=87, BW=349KiB/s (357kB/s)(10.0MiB/29375msec); 0 zone resets slat (usec): min=42, max=1037, avg=218.37, stdev=96.77 clat (msec): min=38, max=297, avg=183.15, stdev=33.01 lat (msec): min=39, max=297, avg=183.37, stdev=33.01 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 190], | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 264], 99.50th=[ 268], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 264, max= 523, per=24.94%, avg=347.62, stdev=45.53, samples=58 iops : min= 66, max= 130, avg=86.88, stdev=11.35, samples=58 lat (msec) : 50=0.04%, 100=1.48%, 250=95.59%, 500=2.89% cpu : usr=0.12%, sys=1.57%, ctx=2972, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159281: Wed Jul 15 09:54:28 2026 write: IOPS=87, BW=349KiB/s (357kB/s)(10.0MiB/29338msec); 0 zone resets slat (usec): min=53, max=1083, avg=210.76, stdev=92.53 clat (msec): min=38, max=303, avg=182.96, stdev=33.19 lat (msec): min=39, max=303, avg=183.17, stdev=33.19 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 264], 99.50th=[ 271], 99.90th=[ 292], 99.95th=[ 300], | 99.99th=[ 305] bw ( KiB/s): min= 264, max= 535, per=24.99%, avg=348.38, stdev=46.27, samples=58 iops : min= 66, max= 133, avg=87.07, stdev=11.53, samples=58 lat (msec) : 50=0.12%, 100=1.48%, 250=95.86%, 500=2.54% cpu : usr=0.12%, sys=1.54%, ctx=2881, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159282: Wed Jul 15 09:54:28 2026 write: IOPS=87, BW=349KiB/s (357kB/s)(10.0MiB/29375msec); 0 zone resets slat (usec): min=35, max=973, avg=212.71, stdev=95.60 clat (msec): min=56, max=303, avg=183.13, stdev=33.38 lat (msec): min=56, max=303, avg=183.34, stdev=33.38 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 266], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 516, per=24.94%, avg=347.64, stdev=48.11, samples=58 iops : min= 64, max= 129, avg=86.90, stdev=12.05, samples=58 lat (msec) : 100=1.48%, 250=95.82%, 500=2.70% cpu : usr=0.16%, sys=1.50%, ctx=2900, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159283: Wed Jul 15 09:54:28 2026 write: IOPS=87, BW=350KiB/s (358kB/s)(10.0MiB/29281msec); 0 zone resets slat (usec): min=50, max=832, avg=219.02, stdev=95.11 clat (msec): min=23, max=313, avg=182.66, stdev=35.31 lat (msec): min=23, max=313, avg=182.88, stdev=35.30 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 134], 10.00th=[ 146], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 268], 99.50th=[ 279], 99.90th=[ 300], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 272, max= 617, per=24.98%, avg=348.28, stdev=51.24, samples=58 iops : min= 68, max= 154, avg=87.05, stdev=12.80, samples=58 lat (msec) : 50=0.70%, 100=1.52%, 250=94.84%, 500=2.93% cpu : usr=0.17%, sys=1.54%, ctx=2975, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1394KiB/s (1428kB/s), 349KiB/s-350KiB/s (357kB/s-358kB/s), io=40.0MiB (41.9MB), run=29281-29375msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=159297: Wed Jul 15 09:54:49 2026 read: IOPS=71, BW=285KiB/s (292kB/s)(5048KiB/17732msec) slat (usec): min=34, max=5623, avg=2183.35, stdev=705.70 clat (msec): min=3, max=189, avg=69.69, stdev=14.28 lat (msec): min=5, max=191, avg=71.87, stdev=14.35 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 50], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 100], 99.50th=[ 103], 99.90th=[ 176], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 200, max= 368, per=25.28%, avg=285.11, stdev=46.50, samples=35 iops : min= 50, max= 92, avg=71.26, stdev=11.61, samples=35 write: IOPS=73, BW=293KiB/s (300kB/s)(5192KiB/17732msec); 0 zone resets slat (usec): min=34, max=2156, avg=166.15, stdev=91.02 clat (msec): min=23, max=304, avg=148.31, stdev=32.65 lat (msec): min=23, max=304, avg=148.48, stdev=32.65 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 100], 10.00th=[ 109], 20.00th=[ 122], | 30.00th=[ 132], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 203], | 99.00th=[ 234], 99.50th=[ 249], 99.90th=[ 296], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 232, max= 352, per=24.85%, avg=292.03, stdev=30.49, samples=35 iops : min= 58, max= 88, avg=73.00, stdev= 7.61, samples=35 lat (msec) : 4=0.04%, 10=0.08%, 20=0.31%, 50=2.46%, 100=48.71% lat (msec) : 250=48.24%, 500=0.16% cpu : usr=0.25%, sys=2.28%, ctx=4652, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159298: Wed Jul 15 09:54:49 2026 read: IOPS=70, BW=280KiB/s (287kB/s)(4976KiB/17744msec) slat (usec): min=944, max=5913, avg=2206.90, stdev=689.41 clat (msec): min=2, max=154, avg=69.92, stdev=12.80 lat (msec): min=6, max=155, avg=72.13, stdev=12.81 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 138], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 176, max= 384, per=24.94%, avg=281.29, stdev=50.15, samples=35 iops : min= 44, max= 96, avg=70.31, stdev=12.53, samples=35 write: IOPS=74, BW=297KiB/s (304kB/s)(5264KiB/17744msec); 0 zone resets slat (usec): min=40, max=701, avg=168.57, stdev=78.03 clat (msec): min=26, max=279, avg=147.01, stdev=32.86 lat (msec): min=26, max=279, avg=147.18, stdev=32.87 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 95], 10.00th=[ 108], 20.00th=[ 122], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 199], | 99.00th=[ 228], 99.50th=[ 239], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 208, max= 392, per=25.20%, avg=296.09, stdev=35.56, samples=35 iops : min= 52, max= 98, avg=74.00, stdev= 8.88, samples=35 lat (msec) : 4=0.04%, 10=0.04%, 20=0.31%, 50=1.91%, 100=49.14% lat (msec) : 250=48.44%, 500=0.12% cpu : usr=0.19%, sys=2.33%, ctx=4681, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159299: Wed Jul 15 09:54:49 2026 read: IOPS=70, BW=280KiB/s (287kB/s)(4984KiB/17780msec) slat (usec): min=970, max=7947, avg=2199.70, stdev=679.11 clat (msec): min=2, max=109, avg=70.18, stdev=12.24 lat (msec): min=4, max=112, avg=72.38, stdev=12.24 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 89], | 99.00th=[ 100], 99.50th=[ 102], 99.90th=[ 105], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 176, max= 376, per=24.84%, avg=280.23, stdev=51.97, samples=35 iops : min= 44, max= 94, avg=70.06, stdev=12.99, samples=35 write: IOPS=73, BW=296KiB/s (303kB/s)(5256KiB/17780msec); 0 zone resets slat (usec): min=50, max=664, avg=167.34, stdev=71.50 clat (msec): min=25, max=238, avg=147.32, stdev=32.93 lat (msec): min=25, max=238, avg=147.49, stdev=32.93 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 92], 10.00th=[ 107], 20.00th=[ 121], | 30.00th=[ 131], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 228], 99.50th=[ 232], 99.90th=[ 236], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 216, max= 376, per=24.85%, avg=291.97, stdev=32.66, samples=35 iops : min= 54, max= 94, avg=72.97, stdev= 8.15, samples=35 lat (msec) : 4=0.08%, 20=0.16%, 50=2.27%, 100=49.49%, 250=48.01% cpu : usr=0.27%, sys=2.28%, ctx=4604, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159300: Wed Jul 15 09:54:49 2026 read: IOPS=71, BW=285KiB/s (292kB/s)(5060KiB/17732msec) slat (usec): min=973, max=4812, avg=2177.85, stdev=672.94 clat (msec): min=2, max=152, avg=70.16, stdev=12.06 lat (msec): min=4, max=154, avg=72.34, stdev=12.08 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 89], | 99.00th=[ 96], 99.50th=[ 100], 99.90th=[ 150], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 192, max= 400, per=25.39%, avg=286.40, stdev=46.96, samples=35 iops : min= 48, max= 100, avg=71.60, stdev=11.74, samples=35 write: IOPS=73, BW=292KiB/s (299kB/s)(5180KiB/17732msec); 0 zone resets slat (usec): min=56, max=688, avg=168.25, stdev=76.99 clat (msec): min=55, max=280, avg=147.78, stdev=32.48 lat (msec): min=55, max=280, avg=147.95, stdev=32.49 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 97], 10.00th=[ 108], 20.00th=[ 121], | 30.00th=[ 131], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 236], 99.50th=[ 245], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 224, max= 392, per=24.80%, avg=291.37, stdev=33.71, samples=35 iops : min= 56, max= 98, avg=72.83, stdev= 8.42, samples=35 lat (msec) : 4=0.08%, 10=0.04%, 20=0.23%, 50=1.21%, 100=50.51% lat (msec) : 250=47.70%, 500=0.23% cpu : usr=0.18%, sys=2.34%, ctx=4765, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1129KiB/s (1156kB/s), 280KiB/s-285KiB/s (287kB/s-292kB/s), io=19.6MiB (20.5MB), run=17732-17780msec WRITE: bw=1175KiB/s (1203kB/s), 292KiB/s-297KiB/s (299kB/s-304kB/s), io=20.4MiB (21.4MB), run=17732-17780msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=159305: Wed Jul 15 09:54:53 2026 read: IOPS=8, BW=348MiB/s (365MB/s)(40.0MiB/115msec) slat (nsec): min=11655k, max=11655k, avg=11655374.00, stdev= 0.00 clat (nsec): min=98231k, max=98231k, avg=98231371.00, stdev= 0.00 lat (nsec): min=109910k, max=109910k, avg=109909516.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[98042], 5.00th=[98042], 10.00th=[98042], 20.00th=[98042], | 30.00th=[98042], 40.00th=[98042], 50.00th=[98042], 60.00th=[98042], | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.89%, ctx=41, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=40.0MiB (41.9MB), run=115-115msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout