== sanity test 398c: run fio to test AIO ================= 10:50:33 (1787842233) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.407465 s, 103 MB/s osc.lustre-OST0000-osc-ffff894ded93a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180253: Thu Aug 27 10:51:11 2026 write: IOPS=89, BW=360KiB/s (368kB/s)(10.0MiB/28476msec); 0 zone resets slat (usec): min=29, max=39611, avg=288.09, stdev=1318.47 clat (msec): min=32, max=316, avg=177.04, stdev=39.56 lat (msec): min=33, max=316, avg=177.33, stdev=39.59 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 116], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 296], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 208, max= 504, per=24.92%, avg=358.39, stdev=67.12, samples=56 iops : min= 52, max= 126, avg=89.21, stdev=16.76, samples=56 lat (msec) : 50=0.08%, 100=1.25%, 250=93.98%, 500=4.69% cpu : usr=0.23%, sys=1.10%, ctx=2631, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180254: Thu Aug 27 10:51:11 2026 write: IOPS=90, BW=361KiB/s (370kB/s)(10.0MiB/28334msec); 0 zone resets slat (usec): min=28, max=15714, avg=253.39, stdev=802.49 clat (msec): min=30, max=483, avg=176.63, stdev=42.36 lat (msec): min=30, max=484, avg=176.88, stdev=42.35 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 116], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 313], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 206, max= 567, per=25.06%, avg=360.37, stdev=73.71, samples=56 iops : min= 51, max= 141, avg=89.71, stdev=18.42, samples=56 lat (msec) : 50=0.16%, 100=1.99%, 250=93.63%, 500=4.22% cpu : usr=0.23%, sys=0.94%, ctx=2168, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180255: Thu Aug 27 10:51:11 2026 write: IOPS=89, BW=360KiB/s (368kB/s)(10.0MiB/28473msec); 0 zone resets slat (usec): min=25, max=16471, avg=240.16, stdev=851.94 clat (msec): min=26, max=313, avg=176.97, stdev=38.04 lat (msec): min=26, max=313, avg=177.21, stdev=38.04 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 122], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 309], | 99.99th=[ 313] bw ( KiB/s): min= 253, max= 506, per=24.89%, avg=357.93, stdev=62.34, samples=56 iops : min= 63, max= 126, avg=89.11, stdev=15.57, samples=56 lat (msec) : 50=0.31%, 100=0.55%, 250=95.00%, 500=4.14% cpu : usr=0.18%, sys=0.95%, ctx=2289, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180256: Thu Aug 27 10:51:11 2026 write: IOPS=89, BW=360KiB/s (368kB/s)(10.0MiB/28464msec); 0 zone resets slat (usec): min=31, max=27859, avg=279.52, stdev=1022.68 clat (msec): min=17, max=308, avg=176.95, stdev=40.06 lat (msec): min=17, max=308, avg=177.23, stdev=40.07 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 116], 10.00th=[ 129], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 240, max= 480, per=24.84%, avg=357.25, stdev=61.87, samples=56 iops : min= 60, max= 120, avg=88.91, stdev=15.44, samples=56 lat (msec) : 20=0.04%, 50=0.20%, 100=1.76%, 250=93.59%, 500=4.41% cpu : usr=0.21%, sys=1.14%, ctx=2715, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1438KiB/s (1473kB/s), 360KiB/s-361KiB/s (368kB/s-370kB/s), io=40.0MiB (41.9MB), run=28334-28476msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180271: Thu Aug 27 10:51:54 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(5048KiB/32940msec) slat (usec): min=96, max=62670, avg=13995.13, stdev=7475.29 clat (msec): min=32, max=377, avg=191.79, stdev=44.30 lat (msec): min=32, max=400, avg=205.80, stdev=45.49 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 124], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 266], | 99.00th=[ 317], 99.50th=[ 321], 99.90th=[ 359], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 61, max= 253, per=24.83%, avg=151.23, stdev=37.61, samples=65 iops : min= 15, max= 63, avg=37.26, stdev= 9.43, samples=65 write: IOPS=39, BW=158KiB/s (161kB/s)(5192KiB/32940msec); 0 zone resets slat (usec): min=31, max=17743, avg=325.35, stdev=1134.79 clat (msec): min=32, max=353, avg=202.51, stdev=42.14 lat (msec): min=32, max=354, avg=202.84, stdev=42.14 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 326], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 61, max= 215, per=24.67%, avg=156.43, stdev=29.59, samples=65 iops : min= 15, max= 53, avg=38.57, stdev= 7.40, samples=65 lat (msec) : 50=0.27%, 100=0.47%, 250=88.05%, 500=11.21% cpu : usr=0.29%, sys=1.76%, ctx=4510, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180272: Thu Aug 27 10:51:54 2026 read: IOPS=37, BW=151KiB/s (155kB/s)(4976KiB/32857msec) slat (usec): min=3105, max=47830, avg=13697.51, stdev=6898.43 clat (msec): min=46, max=343, avg=194.21, stdev=42.45 lat (msec): min=54, max=373, avg=207.91, stdev=43.86 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 249], 95.00th=[ 266], | 99.00th=[ 317], 99.50th=[ 326], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 46, max= 261, per=24.70%, avg=150.40, stdev=38.07, samples=65 iops : min= 11, max= 65, avg=37.06, stdev= 9.56, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5264KiB/32857msec); 0 zone resets slat (usec): min=31, max=16559, avg=297.61, stdev=873.79 clat (msec): min=61, max=367, avg=199.71, stdev=40.50 lat (msec): min=61, max=367, avg=200.01, stdev=40.45 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 220], 80.00th=[ 232], 90.00th=[ 255], 95.00th=[ 268], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 342], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 96, max= 271, per=25.23%, avg=159.94, stdev=37.17, samples=65 iops : min= 24, max= 67, avg=39.45, stdev= 9.31, samples=65 lat (msec) : 50=0.08%, 100=0.62%, 250=89.06%, 500=10.23% cpu : usr=0.21%, sys=1.81%, ctx=4478, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180273: Thu Aug 27 10:51:54 2026 read: IOPS=37, BW=152KiB/s (155kB/s)(4984KiB/32837msec) slat (usec): min=55, max=195860, avg=14015.57, stdev=8727.69 clat (msec): min=25, max=355, avg=194.48, stdev=45.23 lat (msec): min=43, max=374, avg=208.50, stdev=46.19 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 128], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 253], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 79, max= 285, per=24.88%, avg=151.49, stdev=38.19, samples=65 iops : min= 19, max= 71, avg=37.34, stdev= 9.60, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5256KiB/32837msec); 0 zone resets slat (usec): min=36, max=10849, avg=248.44, stdev=698.99 clat (msec): min=43, max=377, avg=201.40, stdev=41.80 lat (msec): min=43, max=377, avg=201.65, stdev=41.76 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 234], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 359], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 63, max= 247, per=25.07%, avg=158.95, stdev=35.15, samples=65 iops : min= 15, max= 61, avg=39.20, stdev= 8.81, samples=65 lat (msec) : 50=0.12%, 100=0.39%, 250=88.01%, 500=11.48% cpu : usr=0.27%, sys=1.72%, ctx=4381, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180274: Thu Aug 27 10:51:54 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(5060KiB/32882msec) slat (usec): min=3028, max=48184, avg=13973.93, stdev=6998.54 clat (msec): min=18, max=342, avg=192.72, stdev=41.35 lat (msec): min=22, max=366, avg=206.70, stdev=42.46 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 224], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 300], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 63, max= 261, per=25.12%, avg=152.97, stdev=35.08, samples=65 iops : min= 15, max= 65, avg=37.69, stdev= 8.87, samples=65 write: IOPS=39, BW=158KiB/s (161kB/s)(5180KiB/32882msec); 0 zone resets slat (usec): min=31, max=12754, avg=243.61, stdev=786.30 clat (msec): min=56, max=343, avg=201.33, stdev=42.06 lat (msec): min=56, max=343, avg=201.58, stdev=42.04 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 86, max= 231, per=24.64%, avg=156.22, stdev=31.89, samples=65 iops : min= 21, max= 57, avg=38.51, stdev= 7.93, samples=65 lat (msec) : 20=0.04%, 50=0.04%, 100=0.47%, 250=88.16%, 500=11.29% cpu : usr=0.25%, sys=1.77%, ctx=4354, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=609KiB/s (624kB/s), 151KiB/s-154KiB/s (155kB/s-158kB/s), io=19.6MiB (20.5MB), run=32837-32940msec WRITE: bw=634KiB/s (649kB/s), 158KiB/s-160KiB/s (161kB/s-164kB/s), io=20.4MiB (21.4MB), run=32837-32940msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180283: Thu Aug 27 10:52:04 2026 read: IOPS=2, BW=86.4MiB/s (90.6MB/s)(40.0MiB/463msec) slat (nsec): min=36641k, max=36641k, avg=36640583.00, stdev= 0.00 clat (nsec): min=404430k, max=404430k, avg=404430374.00, stdev= 0.00 lat (nsec): min=441075k, max=441075k, avg=441075301.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.03%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=86.4MiB/s (90.6MB/s), 86.4MiB/s-86.4MiB/s (90.6MB/s-90.6MB/s), io=40.0MiB (41.9MB), run=463-463msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout