== sanity test 398c: run fio to test AIO ================= 14:14:05 (1780337645) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.5967 s, 70.3 MB/s osc.lustre-OST0000-osc-ffff97ae48ba2800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182259: Mon Jun 1 14:14:50 2026 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31055msec); 0 zone resets slat (usec): min=38, max=26241, avg=431.97, stdev=1287.61 clat (msec): min=16, max=409, avg=193.51, stdev=53.79 lat (msec): min=19, max=410, avg=193.95, stdev=53.83 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 201], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 376], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 150, max= 510, per=25.08%, avg=329.52, stdev=77.50, samples=61 iops : min= 37, max= 127, avg=81.98, stdev=19.31, samples=61 lat (msec) : 20=0.04%, 50=0.12%, 100=0.39%, 250=83.28%, 500=16.17% cpu : usr=0.22%, sys=1.38%, ctx=2946, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182260: Mon Jun 1 14:14:50 2026 write: IOPS=82, BW=329KiB/s (336kB/s)(10.0MiB/31165msec); 0 zone resets slat (usec): min=39, max=23278, avg=345.17, stdev=1080.26 clat (msec): min=53, max=386, avg=193.14, stdev=54.15 lat (msec): min=53, max=386, avg=193.49, stdev=54.16 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 118], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 126, max= 500, per=24.88%, avg=326.90, stdev=81.48, samples=61 iops : min= 31, max= 125, avg=81.30, stdev=20.31, samples=61 lat (msec) : 100=0.90%, 250=83.40%, 500=15.70% cpu : usr=0.13%, sys=1.16%, ctx=2144, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182261: Mon Jun 1 14:14:50 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31093msec); 0 zone resets slat (usec): min=36, max=25018, avg=342.64, stdev=1004.19 clat (msec): min=88, max=404, avg=193.32, stdev=52.45 lat (msec): min=90, max=405, avg=193.66, stdev=52.48 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 121], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 186], 60.00th=[ 201], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 296], | 99.00th=[ 351], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 401], | 99.99th=[ 405] bw ( KiB/s): min= 142, max= 500, per=25.01%, avg=328.67, stdev=85.98, samples=61 iops : min= 35, max= 125, avg=81.74, stdev=21.49, samples=61 lat (msec) : 100=0.47%, 250=87.15%, 500=12.38% cpu : usr=0.13%, sys=1.00%, ctx=2281, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182262: Mon Jun 1 14:14:50 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31139msec); 0 zone resets slat (usec): min=38, max=31428, avg=378.60, stdev=1219.72 clat (msec): min=34, max=399, avg=193.18, stdev=52.98 lat (msec): min=35, max=399, avg=193.56, stdev=53.01 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 120], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 201], | 70.00th=[ 220], 80.00th=[ 234], 90.00th=[ 262], 95.00th=[ 296], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 126, max= 493, per=24.97%, avg=328.13, stdev=84.63, samples=61 iops : min= 31, max= 123, avg=81.59, stdev=21.10, samples=61 lat (msec) : 50=0.08%, 100=0.70%, 250=86.52%, 500=12.70% cpu : usr=0.15%, sys=1.03%, ctx=2491, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1314KiB/s (1346kB/s), 329KiB/s-330KiB/s (336kB/s-338kB/s), io=40.0MiB (41.9MB), run=31055-31165msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182276: Mon Jun 1 14:15:29 2026 read: IOPS=39, BW=159KiB/s (163kB/s)(5048KiB/31705msec) slat (usec): min=2826, max=55813, avg=14201.71, stdev=7641.10 clat (msec): min=13, max=309, avg=187.75, stdev=42.69 lat (msec): min=24, max=338, avg=201.96, stdev=44.08 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 226], 90.00th=[ 243], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 78, max= 229, per=24.82%, avg=156.85, stdev=33.36, samples=62 iops : min= 19, max= 57, avg=38.66, stdev= 8.35, samples=62 write: IOPS=40, BW=164KiB/s (168kB/s)(5192KiB/31705msec); 0 zone resets slat (usec): min=53, max=6167, avg=336.99, stdev=670.29 clat (msec): min=8, max=314, avg=191.98, stdev=42.08 lat (msec): min=13, max=314, avg=192.32, stdev=42.03 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 124], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 249], 95.00th=[ 268], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 87, max= 238, per=24.68%, avg=162.40, stdev=34.72, samples=62 iops : min= 21, max= 59, avg=40.03, stdev= 8.73, samples=62 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=0.94%, 250=90.27% lat (msec) : 500=8.55% cpu : usr=0.21%, sys=1.94%, ctx=4184, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182277: Mon Jun 1 14:15:29 2026 read: IOPS=39, BW=157KiB/s (161kB/s)(4976KiB/31671msec) slat (usec): min=3316, max=60365, avg=14213.13, stdev=7493.26 clat (msec): min=13, max=336, avg=187.52, stdev=41.97 lat (msec): min=17, max=353, avg=201.74, stdev=43.21 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 123], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 220], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 71, max= 277, per=24.66%, avg=155.84, stdev=37.98, samples=62 iops : min= 17, max= 69, avg=38.40, stdev= 9.51, samples=62 write: IOPS=41, BW=166KiB/s (170kB/s)(5264KiB/31671msec); 0 zone resets slat (usec): min=52, max=20957, avg=387.64, stdev=1046.68 clat (msec): min=45, max=314, avg=192.23, stdev=40.40 lat (msec): min=46, max=314, avg=192.62, stdev=40.42 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 226], 90.00th=[ 245], 95.00th=[ 259], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 103, max= 247, per=25.25%, avg=166.11, stdev=32.71, samples=62 iops : min= 25, max= 61, avg=40.97, stdev= 8.23, samples=62 lat (msec) : 20=0.04%, 50=0.08%, 100=0.82%, 250=91.64%, 500=7.42% cpu : usr=0.22%, sys=2.07%, ctx=4481, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182278: Mon Jun 1 14:15:29 2026 read: IOPS=39, BW=159KiB/s (162kB/s)(4984KiB/31407msec) slat (usec): min=93, max=55144, avg=14269.87, stdev=7451.92 clat (msec): min=11, max=319, avg=186.58, stdev=43.44 lat (msec): min=11, max=335, avg=200.87, stdev=44.97 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 116], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 80, max= 222, per=25.08%, avg=158.48, stdev=35.58, samples=62 iops : min= 20, max= 55, avg=39.08, stdev= 8.93, samples=62 write: IOPS=41, BW=167KiB/s (171kB/s)(5256KiB/31407msec); 0 zone resets slat (usec): min=51, max=17624, avg=352.71, stdev=914.58 clat (msec): min=35, max=352, avg=191.14, stdev=41.78 lat (msec): min=35, max=353, avg=191.49, stdev=41.79 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 224], 90.00th=[ 245], 95.00th=[ 257], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 342], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 87, max= 267, per=25.25%, avg=166.16, stdev=37.70, samples=62 iops : min= 21, max= 66, avg=40.98, stdev= 9.48, samples=62 lat (msec) : 20=0.08%, 50=0.23%, 100=1.41%, 250=91.13%, 500=7.15% cpu : usr=0.23%, sys=1.91%, ctx=4315, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182279: Mon Jun 1 14:15:29 2026 read: IOPS=39, BW=160KiB/s (164kB/s)(5060KiB/31639msec) slat (usec): min=3254, max=53305, avg=14094.62, stdev=7126.70 clat (msec): min=72, max=322, avg=186.89, stdev=40.02 lat (msec): min=87, max=339, avg=200.99, stdev=41.16 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 222], 90.00th=[ 241], 95.00th=[ 257], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 95, max= 236, per=25.16%, avg=159.00, stdev=31.14, samples=62 iops : min= 23, max= 59, avg=39.19, stdev= 7.78, samples=62 write: IOPS=40, BW=164KiB/s (168kB/s)(5180KiB/31639msec); 0 zone resets slat (usec): min=54, max=10909, avg=316.59, stdev=694.74 clat (msec): min=36, max=347, avg=192.65, stdev=40.76 lat (msec): min=36, max=347, avg=192.97, stdev=40.74 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 228], 90.00th=[ 247], 95.00th=[ 264], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 342], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 79, max= 253, per=24.71%, avg=162.58, stdev=39.13, samples=62 iops : min= 19, max= 63, avg=40.08, stdev= 9.86, samples=62 lat (msec) : 50=0.04%, 100=0.94%, 250=91.48%, 500=7.54% cpu : usr=0.33%, sys=1.90%, ctx=4178, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=633KiB/s (648kB/s), 157KiB/s-160KiB/s (161kB/s-164kB/s), io=19.6MiB (20.5MB), run=31407-31705msec WRITE: bw=659KiB/s (675kB/s), 164KiB/s-167KiB/s (168kB/s-171kB/s), io=20.4MiB (21.4MB), run=31407-31705msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182289: Mon Jun 1 14:15:41 2026 read: IOPS=1, BW=73.7MiB/s (77.2MB/s)(40.0MiB/543msec) slat (nsec): min=34200k, max=34200k, avg=34199937.00, stdev= 0.00 clat (nsec): min=482565k, max=482565k, avg=482564862.00, stdev= 0.00 lat (nsec): min=516770k, max=516770k, avg=516770096.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.03%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=73.7MiB/s (77.2MB/s), 73.7MiB/s-73.7MiB/s (77.2MB/s-77.2MB/s), io=40.0MiB (41.9MB), run=543-543msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout