== sanity test 398c: run fio to test AIO ================= 22:09:54 (1787796594) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.508567 s, 82.5 MB/s osc.lustre-OST0000-osc-ffff9b8cf4ef4800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180688: Wed Aug 26 22:10:32 2026 write: IOPS=93, BW=374KiB/s (383kB/s)(10.0MiB/27352msec); 0 zone resets slat (usec): min=23, max=24430, avg=183.85, stdev=855.93 clat (msec): min=43, max=319, avg=169.48, stdev=38.58 lat (msec): min=48, max=319, avg=169.66, stdev=38.56 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 118], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 268], 99.50th=[ 296], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 246, max= 510, per=24.88%, avg=372.50, stdev=75.92, samples=54 iops : min= 61, max= 127, avg=92.83, stdev=18.94, samples=54 lat (msec) : 50=0.04%, 100=1.84%, 250=95.82%, 500=2.30% cpu : usr=0.16%, sys=0.84%, ctx=1908, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180689: Wed Aug 26 22:10:32 2026 write: IOPS=94, BW=377KiB/s (386kB/s)(10.0MiB/27183msec); 0 zone resets slat (usec): min=22, max=16348, avg=157.33, stdev=659.16 clat (msec): min=22, max=320, avg=168.88, stdev=38.67 lat (msec): min=23, max=320, avg=169.04, stdev=38.67 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 115], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 236], | 99.00th=[ 271], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 244, max= 512, per=25.05%, avg=374.96, stdev=73.29, samples=54 iops : min= 61, max= 128, avg=93.48, stdev=18.34, samples=54 lat (msec) : 50=0.08%, 100=1.05%, 250=97.46%, 500=1.41% cpu : usr=0.16%, sys=0.70%, ctx=1379, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180690: Wed Aug 26 22:10:32 2026 write: IOPS=94, BW=379KiB/s (388kB/s)(10.0MiB/27013msec); 0 zone resets slat (usec): min=22, max=16062, avg=200.11, stdev=722.69 clat (msec): min=11, max=342, avg=168.37, stdev=39.29 lat (msec): min=12, max=342, avg=168.57, stdev=39.29 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 112], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 220], 95.00th=[ 239], | 99.00th=[ 266], 99.50th=[ 309], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 342] bw ( KiB/s): min= 255, max= 510, per=25.28%, avg=378.49, stdev=63.97, samples=53 iops : min= 63, max= 127, avg=94.32, stdev=16.08, samples=53 lat (msec) : 20=0.08%, 50=0.27%, 100=2.07%, 250=95.51%, 500=2.07% cpu : usr=0.24%, sys=1.00%, ctx=2618, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180691: Wed Aug 26 22:10:32 2026 write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27313msec); 0 zone resets slat (usec): min=22, max=20703, avg=214.59, stdev=884.91 clat (msec): min=26, max=313, avg=169.26, stdev=39.31 lat (msec): min=26, max=314, avg=169.48, stdev=39.30 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 114], 10.00th=[ 126], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 236], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 245, max= 504, per=25.02%, avg=374.52, stdev=61.78, samples=54 iops : min= 61, max= 126, avg=93.35, stdev=15.44, samples=54 lat (msec) : 50=0.08%, 100=2.03%, 250=94.77%, 500=3.12% cpu : usr=0.21%, sys=1.00%, ctx=2664, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1498KiB/s (1533kB/s), 374KiB/s-379KiB/s (383kB/s-388kB/s), io=40.0MiB (41.9MB), run=27013-27352msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180705: Wed Aug 26 22:11:15 2026 read: IOPS=38, BW=154KiB/s (157kB/s)(5048KiB/32849msec) slat (usec): min=2490, max=52175, avg=14061.23, stdev=8448.78 clat (msec): min=74, max=441, avg=192.46, stdev=54.84 lat (msec): min=78, max=449, avg=206.53, stdev=57.03 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 230], 90.00th=[ 266], 95.00th=[ 300], | 99.00th=[ 355], 99.50th=[ 397], 99.90th=[ 435], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 55, max= 302, per=25.03%, avg=152.68, stdev=44.86, samples=65 iops : min= 13, max= 75, avg=37.62, stdev=11.18, samples=65 write: IOPS=39, BW=158KiB/s (162kB/s)(5192KiB/32849msec); 0 zone resets slat (usec): min=33, max=21483, avg=259.24, stdev=1071.83 clat (msec): min=31, max=423, avg=200.47, stdev=51.32 lat (msec): min=32, max=423, avg=200.73, stdev=51.30 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 384], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 65, max= 247, per=24.76%, avg=157.46, stdev=39.98, samples=65 iops : min= 16, max= 61, avg=38.80, stdev= 9.99, samples=65 lat (msec) : 50=0.04%, 100=0.78%, 250=84.26%, 500=14.92% cpu : usr=0.21%, sys=1.60%, ctx=4080, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180706: Wed Aug 26 22:11:15 2026 read: IOPS=38, BW=152KiB/s (156kB/s)(4976KiB/32652msec) slat (usec): min=2688, max=65196, avg=14618.10, stdev=9089.63 clat (msec): min=48, max=424, avg=193.25, stdev=55.36 lat (msec): min=63, max=438, avg=207.88, stdev=57.72 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 118], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 199], | 70.00th=[ 213], 80.00th=[ 232], 90.00th=[ 264], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 372], 99.90th=[ 405], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 71, max= 255, per=24.85%, avg=151.56, stdev=35.93, samples=64 iops : min= 17, max= 63, avg=37.33, stdev= 8.97, samples=64 write: IOPS=40, BW=161KiB/s (165kB/s)(5264KiB/32652msec); 0 zone resets slat (usec): min=25, max=18799, avg=200.16, stdev=717.86 clat (msec): min=52, max=388, avg=197.81, stdev=50.61 lat (msec): min=53, max=388, avg=198.02, stdev=50.63 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 133], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 268], 95.00th=[ 300], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 55, max= 254, per=25.36%, avg=161.31, stdev=42.45, samples=64 iops : min= 13, max= 63, avg=39.77, stdev=10.63, samples=64 lat (msec) : 50=0.04%, 100=0.78%, 250=85.47%, 500=13.71% cpu : usr=0.25%, sys=1.63%, ctx=3979, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180707: Wed Aug 26 22:11:15 2026 read: IOPS=37, BW=152KiB/s (155kB/s)(4984KiB/32827msec) slat (usec): min=2574, max=75865, avg=14325.56, stdev=8936.86 clat (msec): min=58, max=403, avg=192.90, stdev=53.67 lat (msec): min=83, max=425, avg=207.23, stdev=55.89 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 232], 90.00th=[ 264], 95.00th=[ 300], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 397], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 71, max= 294, per=24.84%, avg=151.51, stdev=41.45, samples=65 iops : min= 17, max= 73, avg=37.32, stdev=10.38, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5256KiB/32827msec); 0 zone resets slat (usec): min=27, max=13979, avg=258.39, stdev=883.12 clat (msec): min=92, max=443, avg=200.12, stdev=52.74 lat (msec): min=92, max=444, avg=200.38, stdev=52.73 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 305], | 99.00th=[ 372], 99.50th=[ 393], 99.90th=[ 409], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 71, max= 254, per=25.00%, avg=159.00, stdev=42.54, samples=65 iops : min= 17, max= 63, avg=39.20, stdev=10.65, samples=65 lat (msec) : 100=0.62%, 250=84.96%, 500=14.41% cpu : usr=0.28%, sys=1.66%, ctx=4462, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180708: Wed Aug 26 22:11:15 2026 read: IOPS=39, BW=156KiB/s (160kB/s)(5060KiB/32420msec) slat (usec): min=55, max=67558, avg=14513.37, stdev=9041.51 clat (msec): min=14, max=408, avg=193.17, stdev=56.37 lat (msec): min=14, max=428, avg=207.70, stdev=58.88 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 118], 10.00th=[ 131], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 266], 95.00th=[ 300], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 70, max= 270, per=25.51%, avg=155.63, stdev=37.58, samples=64 iops : min= 17, max= 67, avg=38.34, stdev= 9.41, samples=64 write: IOPS=39, BW=160KiB/s (164kB/s)(5180KiB/32420msec); 0 zone resets slat (usec): min=33, max=20586, avg=242.75, stdev=961.14 clat (msec): min=46, max=402, avg=196.89, stdev=50.92 lat (msec): min=46, max=402, avg=197.13, stdev=50.87 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 232], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 388], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 55, max= 239, per=25.05%, avg=159.31, stdev=41.66, samples=64 iops : min= 13, max= 59, avg=39.28, stdev=10.40, samples=64 lat (msec) : 20=0.12%, 50=0.08%, 100=0.94%, 250=84.80%, 500=14.06% cpu : usr=0.27%, sys=1.66%, ctx=4236, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=611KiB/s (626kB/s), 152KiB/s-156KiB/s (155kB/s-160kB/s), io=19.6MiB (20.5MB), run=32420-32849msec WRITE: bw=636KiB/s (651kB/s), 158KiB/s-161KiB/s (162kB/s-165kB/s), io=20.4MiB (21.4MB), run=32420-32849msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180718: Wed Aug 26 22:11:24 2026 read: IOPS=3, BW=125MiB/s (131MB/s)(40.0MiB/320msec) slat (nsec): min=25358k, max=25358k, avg=25357831.00, stdev= 0.00 clat (nsec): min=273488k, max=273488k, avg=273487534.00, stdev= 0.00 lat (nsec): min=298851k, max=298851k, avg=298850837.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.76%, ctx=5, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=40.0MiB (41.9MB), run=320-320msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout