== sanity test 398c: run fio to test AIO ================= 11:10:43 (1761318643) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.376878 s, 111 MB/s osc.lustre-OST0000-osc-ffff9ece02a1b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149026: Fri Oct 24 11:11:07 2025 write: IOPS=147, BW=591KiB/s (606kB/s)(10.0MiB/17316msec); 0 zone resets slat (usec): min=24, max=4533, avg=159.54, stdev=327.30 clat (msec): min=13, max=177, avg=107.69, stdev=24.16 lat (msec): min=13, max=177, avg=107.85, stdev=24.17 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 86], | 30.00th=[ 94], 40.00th=[ 103], 50.00th=[ 109], 60.00th=[ 115], | 70.00th=[ 123], 80.00th=[ 130], 90.00th=[ 140], 95.00th=[ 146], | 99.00th=[ 161], 99.50th=[ 169], 99.90th=[ 174], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 396, max= 800, per=24.93%, avg=588.79, stdev=103.40, samples=34 iops : min= 99, max= 200, avg=146.88, stdev=25.89, samples=34 lat (msec) : 20=0.08%, 50=0.16%, 100=36.88%, 250=62.89% cpu : usr=0.31%, sys=1.73%, ctx=1514, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149027: Fri Oct 24 11:11:07 2025 write: IOPS=147, BW=591KiB/s (605kB/s)(10.0MiB/17339msec); 0 zone resets slat (usec): min=22, max=8204, avg=136.46, stdev=305.01 clat (msec): min=5, max=183, avg=107.92, stdev=24.28 lat (msec): min=8, max=183, avg=108.06, stdev=24.28 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 68], 10.00th=[ 78], 20.00th=[ 87], | 30.00th=[ 94], 40.00th=[ 102], 50.00th=[ 108], 60.00th=[ 115], | 70.00th=[ 122], 80.00th=[ 130], 90.00th=[ 140], 95.00th=[ 148], | 99.00th=[ 163], 99.50th=[ 176], 99.90th=[ 182], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 388, max= 784, per=24.95%, avg=589.24, stdev=107.33, samples=34 iops : min= 97, max= 196, avg=146.97, stdev=26.91, samples=34 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=38.67%, 250=61.17% cpu : usr=0.18%, sys=1.60%, ctx=1229, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149028: Fri Oct 24 11:11:07 2025 write: IOPS=147, BW=591KiB/s (605kB/s)(10.0MiB/17339msec); 0 zone resets slat (usec): min=24, max=38915, avg=176.74, stdev=841.30 clat (msec): min=47, max=183, avg=107.55, stdev=24.27 lat (msec): min=47, max=183, avg=107.73, stdev=24.26 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 77], 20.00th=[ 87], | 30.00th=[ 93], 40.00th=[ 101], 50.00th=[ 108], 60.00th=[ 114], | 70.00th=[ 122], 80.00th=[ 129], 90.00th=[ 140], 95.00th=[ 148], | 99.00th=[ 165], 99.50th=[ 174], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 381, max= 776, per=24.95%, avg=589.29, stdev=106.88, samples=34 iops : min= 95, max= 194, avg=147.00, stdev=26.78, samples=34 lat (msec) : 50=0.12%, 100=39.96%, 250=59.92% cpu : usr=0.29%, sys=1.72%, ctx=1440, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149029: Fri Oct 24 11:11:07 2025 write: IOPS=147, BW=592KiB/s (606kB/s)(10.0MiB/17309msec); 0 zone resets slat (usec): min=20, max=5524, avg=145.76, stdev=318.95 clat (msec): min=16, max=176, avg=107.92, stdev=23.88 lat (msec): min=17, max=176, avg=108.06, stdev=23.87 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 68], 10.00th=[ 78], 20.00th=[ 86], | 30.00th=[ 93], 40.00th=[ 103], 50.00th=[ 109], 60.00th=[ 115], | 70.00th=[ 123], 80.00th=[ 130], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 163], 99.50th=[ 169], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 178] bw ( KiB/s): min= 429, max= 792, per=25.03%, avg=591.32, stdev=99.78, samples=34 iops : min= 107, max= 198, avg=147.47, stdev=25.05, samples=34 lat (msec) : 20=0.04%, 50=0.08%, 100=38.28%, 250=61.60% cpu : usr=0.24%, sys=1.57%, ctx=1210, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2362KiB/s (2419kB/s), 591KiB/s-592KiB/s (605kB/s-606kB/s), io=40.0MiB (41.9MB), run=17309-17339msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149041: Fri Oct 24 11:11:28 2025 read: IOPS=76, BW=306KiB/s (313kB/s)(5048KiB/16514msec) slat (usec): min=1825, max=23258, avg=6416.29, stdev=2860.55 clat (msec): min=6, max=183, avg=96.29, stdev=24.74 lat (msec): min=17, max=198, avg=102.71, stdev=25.69 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 60], 10.00th=[ 67], 20.00th=[ 74], | 30.00th=[ 82], 40.00th=[ 89], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 118], 90.00th=[ 129], 95.00th=[ 138], | 99.00th=[ 159], 99.50th=[ 167], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 211, max= 471, per=24.98%, avg=303.50, stdev=72.72, samples=32 iops : min= 52, max= 117, avg=75.41, stdev=18.10, samples=32 write: IOPS=78, BW=314KiB/s (322kB/s)(5192KiB/16514msec); 0 zone resets slat (usec): min=26, max=3108, avg=166.65, stdev=272.44 clat (msec): min=22, max=174, avg=102.13, stdev=24.52 lat (msec): min=23, max=174, avg=102.29, stdev=24.52 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 65], 10.00th=[ 72], 20.00th=[ 82], | 30.00th=[ 89], 40.00th=[ 95], 50.00th=[ 101], 60.00th=[ 107], | 70.00th=[ 114], 80.00th=[ 123], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 161], 99.50th=[ 165], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 191, max= 455, per=24.68%, avg=312.19, stdev=58.52, samples=32 iops : min= 47, max= 113, avg=77.59, stdev=14.67, samples=32 lat (msec) : 10=0.04%, 50=0.90%, 100=52.30%, 250=46.76% cpu : usr=0.38%, sys=3.01%, ctx=2596, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149042: Fri Oct 24 11:11:28 2025 read: IOPS=75, BW=302KiB/s (309kB/s)(4976KiB/16492msec) slat (usec): min=2035, max=27208, avg=6507.81, stdev=2941.82 clat (msec): min=8, max=182, avg=97.47, stdev=24.49 lat (msec): min=13, max=187, avg=103.99, stdev=25.37 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 83], 40.00th=[ 90], 50.00th=[ 97], 60.00th=[ 104], | 70.00th=[ 110], 80.00th=[ 118], 90.00th=[ 131], 95.00th=[ 140], | 99.00th=[ 161], 99.50th=[ 165], 99.90th=[ 180], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 213, max= 463, per=24.62%, avg=299.19, stdev=64.50, samples=32 iops : min= 53, max= 115, avg=74.34, stdev=16.02, samples=32 write: IOPS=79, BW=319KiB/s (327kB/s)(5264KiB/16492msec); 0 zone resets slat (usec): min=22, max=4748, avg=161.59, stdev=283.73 clat (msec): min=41, max=175, avg=100.94, stdev=24.04 lat (msec): min=41, max=175, avg=101.10, stdev=24.05 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 87], 40.00th=[ 93], 50.00th=[ 100], 60.00th=[ 106], | 70.00th=[ 113], 80.00th=[ 123], 90.00th=[ 134], 95.00th=[ 144], | 99.00th=[ 161], 99.50th=[ 169], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 207, max= 463, per=25.04%, avg=316.81, stdev=66.20, samples=32 iops : min= 51, max= 115, avg=78.75, stdev=16.58, samples=32 lat (msec) : 10=0.08%, 20=0.04%, 50=0.66%, 100=52.73%, 250=46.48% cpu : usr=0.44%, sys=3.06%, ctx=2628, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149043: Fri Oct 24 11:11:28 2025 read: IOPS=75, BW=303KiB/s (310kB/s)(4984KiB/16454msec) slat (usec): min=42, max=23294, avg=6440.42, stdev=2967.14 clat (msec): min=11, max=183, avg=96.34, stdev=24.77 lat (msec): min=11, max=192, avg=102.78, stdev=25.76 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 75], | 30.00th=[ 82], 40.00th=[ 89], 50.00th=[ 96], 60.00th=[ 103], | 70.00th=[ 110], 80.00th=[ 118], 90.00th=[ 128], 95.00th=[ 138], | 99.00th=[ 161], 99.50th=[ 169], 99.90th=[ 178], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 206, max= 495, per=24.84%, avg=301.75, stdev=64.40, samples=32 iops : min= 51, max= 123, avg=74.97, stdev=16.01, samples=32 write: IOPS=79, BW=319KiB/s (327kB/s)(5256KiB/16454msec); 0 zone resets slat (usec): min=26, max=4790, avg=172.87, stdev=313.30 clat (msec): min=46, max=192, avg=102.24, stdev=24.19 lat (msec): min=47, max=193, avg=102.42, stdev=24.21 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 124], 90.00th=[ 136], 95.00th=[ 146], | 99.00th=[ 163], 99.50th=[ 178], 99.90th=[ 188], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 190, max= 464, per=25.00%, avg=316.28, stdev=64.24, samples=32 iops : min= 47, max= 116, avg=78.66, stdev=16.07, samples=32 lat (msec) : 20=0.08%, 50=0.82%, 100=52.93%, 250=46.17% cpu : usr=0.41%, sys=3.01%, ctx=2570, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149044: Fri Oct 24 11:11:28 2025 read: IOPS=76, BW=306KiB/s (314kB/s)(5060KiB/16515msec) slat (usec): min=1734, max=21466, avg=6422.10, stdev=2992.69 clat (msec): min=14, max=174, avg=96.75, stdev=25.38 lat (msec): min=19, max=185, avg=103.17, stdev=26.33 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 60], 10.00th=[ 66], 20.00th=[ 74], | 30.00th=[ 82], 40.00th=[ 89], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 118], 90.00th=[ 130], 95.00th=[ 142], | 99.00th=[ 165], 99.50th=[ 169], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 190, max= 495, per=24.95%, avg=303.09, stdev=73.74, samples=32 iops : min= 47, max= 123, avg=75.31, stdev=18.44, samples=32 write: IOPS=78, BW=314KiB/s (321kB/s)(5180KiB/16515msec); 0 zone resets slat (usec): min=25, max=6482, avg=175.35, stdev=368.98 clat (msec): min=19, max=174, avg=102.18, stdev=24.89 lat (msec): min=19, max=174, avg=102.35, stdev=24.90 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 95], 50.00th=[ 102], 60.00th=[ 107], | 70.00th=[ 113], 80.00th=[ 125], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 163], 99.50th=[ 165], 99.90th=[ 171], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 230, max= 479, per=24.44%, avg=309.13, stdev=54.02, samples=32 iops : min= 57, max= 119, avg=76.84, stdev=13.41, samples=32 lat (msec) : 20=0.08%, 50=0.82%, 100=51.48%, 250=47.62% cpu : usr=0.45%, sys=2.94%, ctx=2540, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1215KiB/s (1244kB/s), 302KiB/s-306KiB/s (309kB/s-314kB/s), io=19.6MiB (20.5MB), run=16454-16515msec WRITE: bw=1265KiB/s (1295kB/s), 314KiB/s-319KiB/s (321kB/s-327kB/s), io=20.4MiB (21.4MB), run=16454-16515msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149051: Fri Oct 24 11:11:33 2025 read: IOPS=4, BW=167MiB/s (175MB/s)(40.0MiB/240msec) slat (nsec): min=16398k, max=16398k, avg=16397975.00, stdev= 0.00 clat (nsec): min=208491k, max=208491k, avg=208491205.00, stdev= 0.00 lat (nsec): min=224924k, max=224924k, avg=224924267.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.69%, ctx=3, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=40.0MiB (41.9MB), run=240-240msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck