== sanity test 398c: run fio to test AIO ================= 19:52:42 (1784591562) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.330121 s, 127 MB/s osc.lustre-OST0000-osc-ffff940c805ee000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179190: Mon Jul 20 19:53:10 2026 write: IOPS=119, BW=476KiB/s (488kB/s)(10.0MiB/21501msec); 0 zone resets slat (usec): min=35, max=14020, avg=257.22, stdev=663.28 clat (msec): min=21, max=301, avg=133.38, stdev=30.80 lat (msec): min=21, max=302, avg=133.64, stdev=30.80 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 90], 10.00th=[ 97], 20.00th=[ 108], | 30.00th=[ 114], 40.00th=[ 123], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 218], 99.50th=[ 232], 99.90th=[ 247], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 351, max= 640, per=25.17%, avg=477.74, stdev=70.67, samples=42 iops : min= 87, max= 160, avg=119.14, stdev=17.67, samples=42 lat (msec) : 50=0.27%, 100=13.12%, 250=86.52%, 500=0.08% cpu : usr=0.23%, sys=1.50%, ctx=2637, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179191: Mon Jul 20 19:53:10 2026 write: IOPS=118, BW=475KiB/s (486kB/s)(10.0MiB/21577msec); 0 zone resets slat (usec): min=36, max=8085, avg=255.74, stdev=550.30 clat (msec): min=25, max=242, avg=133.77, stdev=30.55 lat (msec): min=25, max=242, avg=134.03, stdev=30.55 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 190], | 99.00th=[ 222], 99.50th=[ 224], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 331, max= 626, per=25.00%, avg=474.48, stdev=84.75, samples=42 iops : min= 82, max= 156, avg=118.31, stdev=21.18, samples=42 lat (msec) : 50=0.27%, 100=10.23%, 250=89.49% cpu : usr=0.24%, sys=1.48%, ctx=2929, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179192: Mon Jul 20 19:53:10 2026 write: IOPS=119, BW=478KiB/s (489kB/s)(10.0MiB/21437msec); 0 zone resets slat (usec): min=36, max=9701, avg=227.45, stdev=506.70 clat (msec): min=34, max=277, avg=133.58, stdev=29.79 lat (msec): min=35, max=278, avg=133.81, stdev=29.78 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 93], 10.00th=[ 100], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 171], 95.00th=[ 190], | 99.00th=[ 215], 99.50th=[ 224], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 279] bw ( KiB/s): min= 312, max= 638, per=25.15%, avg=477.43, stdev=81.60, samples=42 iops : min= 78, max= 159, avg=119.07, stdev=20.35, samples=42 lat (msec) : 50=0.04%, 100=10.43%, 250=89.41%, 500=0.12% cpu : usr=0.18%, sys=1.27%, ctx=2264, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179193: Mon Jul 20 19:53:10 2026 write: IOPS=118, BW=475KiB/s (486kB/s)(10.0MiB/21575msec); 0 zone resets slat (usec): min=35, max=9014, avg=274.00, stdev=582.73 clat (msec): min=16, max=235, avg=133.78, stdev=29.41 lat (msec): min=17, max=235, avg=134.06, stdev=29.42 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 109], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 215], 99.50th=[ 224], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 236] bw ( KiB/s): min= 319, max= 632, per=25.02%, avg=474.93, stdev=74.15, samples=42 iops : min= 79, max= 158, avg=118.45, stdev=18.53, samples=42 lat (msec) : 20=0.08%, 50=0.04%, 100=10.20%, 250=89.69% cpu : usr=0.26%, sys=1.60%, ctx=3008, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1898KiB/s (1944kB/s), 475KiB/s-478KiB/s (486kB/s-489kB/s), io=40.0MiB (41.9MB), run=21437-21577msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179206: Mon Jul 20 19:53:39 2026 read: IOPS=57, BW=232KiB/s (237kB/s)(5048KiB/21789msec) slat (usec): min=2164, max=56040, avg=9303.28, stdev=5309.80 clat (msec): min=22, max=252, avg=127.10, stdev=34.62 lat (msec): min=33, max=263, avg=136.41, stdev=36.58 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 82], 10.00th=[ 88], 20.00th=[ 97], | 30.00th=[ 105], 40.00th=[ 112], 50.00th=[ 122], 60.00th=[ 132], | 70.00th=[ 144], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 190], | 99.00th=[ 215], 99.50th=[ 226], 99.90th=[ 249], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 124, max= 374, per=25.05%, avg=230.72, stdev=60.44, samples=43 iops : min= 31, max= 93, avg=57.30, stdev=15.17, samples=43 write: IOPS=59, BW=238KiB/s (244kB/s)(5192KiB/21789msec); 0 zone resets slat (usec): min=47, max=29732, avg=278.22, stdev=1018.06 clat (msec): min=31, max=247, avg=133.52, stdev=36.25 lat (msec): min=31, max=247, avg=133.80, stdev=36.32 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 87], 10.00th=[ 93], 20.00th=[ 100], | 30.00th=[ 109], 40.00th=[ 118], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 153], 80.00th=[ 165], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 230], 99.50th=[ 234], 99.90th=[ 243], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 136, max= 352, per=24.83%, avg=237.86, stdev=55.04, samples=43 iops : min= 34, max= 88, avg=59.09, stdev=13.81, samples=43 lat (msec) : 50=0.20%, 100=21.60%, 250=78.16%, 500=0.04% cpu : usr=0.32%, sys=2.20%, ctx=4725, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179207: Mon Jul 20 19:53:39 2026 read: IOPS=57, BW=229KiB/s (235kB/s)(4976KiB/21702msec) slat (usec): min=2037, max=42673, avg=9290.84, stdev=5221.97 clat (msec): min=57, max=307, avg=127.94, stdev=36.60 lat (msec): min=61, max=314, avg=137.24, stdev=38.69 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 81], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 105], 40.00th=[ 113], 50.00th=[ 122], 60.00th=[ 133], | 70.00th=[ 144], 80.00th=[ 159], 90.00th=[ 178], 95.00th=[ 194], | 99.00th=[ 232], 99.50th=[ 241], 99.90th=[ 292], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 131, max= 360, per=24.76%, avg=228.05, stdev=61.84, samples=43 iops : min= 32, max= 90, avg=56.60, stdev=15.54, samples=43 write: IOPS=60, BW=243KiB/s (248kB/s)(5264KiB/21702msec); 0 zone resets slat (usec): min=37, max=10176, avg=264.99, stdev=576.49 clat (msec): min=64, max=259, avg=132.01, stdev=35.68 lat (msec): min=64, max=259, avg=132.28, stdev=35.70 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 101], | 30.00th=[ 108], 40.00th=[ 117], 50.00th=[ 126], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 163], 90.00th=[ 182], 95.00th=[ 203], | 99.00th=[ 239], 99.50th=[ 247], 99.90th=[ 251], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 144, max= 398, per=25.32%, avg=242.53, stdev=62.78, samples=43 iops : min= 36, max= 99, avg=60.26, stdev=15.68, samples=43 lat (msec) : 100=22.30%, 250=77.42%, 500=0.27% cpu : usr=0.28%, sys=2.44%, ctx=4762, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179208: Mon Jul 20 19:53:39 2026 read: IOPS=57, BW=231KiB/s (236kB/s)(4984KiB/21612msec) slat (usec): min=96, max=71881, avg=9166.22, stdev=5639.46 clat (msec): min=21, max=252, avg=129.13, stdev=36.46 lat (msec): min=27, max=264, avg=138.30, stdev=38.31 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 87], 20.00th=[ 97], | 30.00th=[ 107], 40.00th=[ 114], 50.00th=[ 125], 60.00th=[ 136], | 70.00th=[ 146], 80.00th=[ 161], 90.00th=[ 180], 95.00th=[ 197], | 99.00th=[ 228], 99.50th=[ 236], 99.90th=[ 251], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 143, max= 390, per=25.16%, avg=231.69, stdev=52.72, samples=42 iops : min= 35, max= 97, avg=57.55, stdev=13.20, samples=42 write: IOPS=60, BW=243KiB/s (249kB/s)(5256KiB/21612msec); 0 zone resets slat (usec): min=54, max=8927, avg=256.84, stdev=501.85 clat (msec): min=66, max=288, avg=131.27, stdev=35.07 lat (msec): min=66, max=288, avg=131.53, stdev=35.11 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 86], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 107], 40.00th=[ 117], 50.00th=[ 127], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 180], 95.00th=[ 197], | 99.00th=[ 232], 99.50th=[ 251], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 95, max= 406, per=25.36%, avg=242.93, stdev=68.19, samples=42 iops : min= 23, max= 101, avg=60.33, stdev=17.13, samples=42 lat (msec) : 50=0.20%, 100=21.41%, 250=78.09%, 500=0.31% cpu : usr=0.33%, sys=2.38%, ctx=4728, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179209: Mon Jul 20 19:53:39 2026 read: IOPS=58, BW=232KiB/s (238kB/s)(5060KiB/21776msec) slat (usec): min=2057, max=45483, avg=9156.27, stdev=5239.90 clat (msec): min=39, max=253, avg=127.82, stdev=34.61 lat (msec): min=50, max=266, avg=136.98, stdev=36.47 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 81], 10.00th=[ 87], 20.00th=[ 96], | 30.00th=[ 106], 40.00th=[ 115], 50.00th=[ 124], 60.00th=[ 134], | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 192], | 99.00th=[ 224], 99.50th=[ 236], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 118, max= 406, per=25.26%, avg=232.60, stdev=60.95, samples=43 iops : min= 29, max= 101, avg=57.77, stdev=15.30, samples=43 write: IOPS=59, BW=238KiB/s (244kB/s)(5180KiB/21776msec); 0 zone resets slat (usec): min=42, max=13804, avg=251.13, stdev=584.88 clat (msec): min=63, max=271, avg=132.59, stdev=35.87 lat (msec): min=63, max=271, avg=132.84, stdev=35.88 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 86], 10.00th=[ 91], 20.00th=[ 100], | 30.00th=[ 108], 40.00th=[ 117], 50.00th=[ 128], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 163], 90.00th=[ 184], 95.00th=[ 201], | 99.00th=[ 232], 99.50th=[ 243], 99.90th=[ 253], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 135, max= 383, per=24.72%, avg=236.81, stdev=59.63, samples=43 iops : min= 33, max= 95, avg=58.81, stdev=15.00, samples=43 lat (msec) : 50=0.04%, 100=22.34%, 250=77.46%, 500=0.16% cpu : usr=0.24%, sys=2.39%, ctx=4791, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=921KiB/s (943kB/s), 229KiB/s-232KiB/s (235kB/s-238kB/s), io=19.6MiB (20.5MB), run=21612-21789msec WRITE: bw=959KiB/s (982kB/s), 238KiB/s-243KiB/s (244kB/s-249kB/s), io=20.4MiB (21.4MB), run=21612-21789msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179218: Mon Jul 20 19:53:46 2026 read: IOPS=2, BW=90.1MiB/s (94.5MB/s)(40.0MiB/444msec) slat (nsec): min=27140k, max=27140k, avg=27139998.00, stdev= 0.00 clat (nsec): min=387646k, max=387646k, avg=387645817.00, stdev= 0.00 lat (nsec): min=414790k, max=414790k, avg=414790482.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.16%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=90.1MiB/s (94.5MB/s), 90.1MiB/s-90.1MiB/s (94.5MB/s-94.5MB/s), io=40.0MiB (41.9MB), run=444-444msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout