== sanity test 398c: run fio to test AIO ================= 21:18:05 (1786756685) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.336228 s, 125 MB/s osc.lustre-OST0000-osc-ffff954fe00ce800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180774: Fri Aug 14 21:18:44 2026 write: IOPS=89, BW=358KiB/s (366kB/s)(10.0MiB/28636msec); 0 zone resets slat (usec): min=51, max=19377, avg=336.26, stdev=956.25 clat (msec): min=13, max=370, avg=178.09, stdev=44.88 lat (msec): min=14, max=370, avg=178.42, stdev=44.91 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 115], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 259], | 99.00th=[ 305], 99.50th=[ 326], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 372] bw ( KiB/s): min= 191, max= 496, per=24.89%, avg=355.71, stdev=69.94, samples=56 iops : min= 47, max= 124, avg=88.55, stdev=17.52, samples=56 lat (msec) : 20=0.08%, 100=1.09%, 250=91.45%, 500=7.38% cpu : usr=0.18%, sys=1.28%, ctx=2677, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180775: Fri Aug 14 21:18:44 2026 write: IOPS=89, BW=358KiB/s (367kB/s)(10.0MiB/28573msec); 0 zone resets slat (usec): min=50, max=16474, avg=335.92, stdev=890.42 clat (msec): min=14, max=373, avg=177.89, stdev=47.52 lat (msec): min=15, max=374, avg=178.23, stdev=47.57 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 117], 10.00th=[ 128], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 215], 90.00th=[ 245], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 338], 99.90th=[ 355], 99.95th=[ 368], | 99.99th=[ 376] bw ( KiB/s): min= 158, max= 595, per=25.10%, avg=358.66, stdev=78.47, samples=56 iops : min= 39, max= 148, avg=89.27, stdev=19.60, samples=56 lat (msec) : 20=0.04%, 50=0.08%, 100=1.91%, 250=89.49%, 500=8.48% cpu : usr=0.15%, sys=1.27%, ctx=2771, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180776: Fri Aug 14 21:18:44 2026 write: IOPS=89, BW=357KiB/s (366kB/s)(10.0MiB/28653msec); 0 zone resets slat (usec): min=49, max=20065, avg=302.26, stdev=953.31 clat (msec): min=25, max=344, avg=177.72, stdev=45.32 lat (msec): min=25, max=363, avg=178.03, stdev=45.36 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 120], 10.00th=[ 129], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 213], 90.00th=[ 243], 95.00th=[ 268], | 99.00th=[ 300], 99.50th=[ 321], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 143, max= 512, per=24.87%, avg=355.46, stdev=78.69, samples=57 iops : min= 35, max= 128, avg=88.47, stdev=19.68, samples=57 lat (msec) : 50=0.04%, 100=2.07%, 250=89.96%, 500=7.93% cpu : usr=0.15%, sys=1.20%, ctx=2649, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180777: Fri Aug 14 21:18:44 2026 write: IOPS=89, BW=357KiB/s (366kB/s)(10.0MiB/28649msec); 0 zone resets slat (usec): min=52, max=11607, avg=318.89, stdev=805.17 clat (msec): min=24, max=365, avg=177.71, stdev=46.62 lat (msec): min=24, max=367, avg=178.03, stdev=46.63 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 116], 10.00th=[ 130], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 243], 95.00th=[ 262], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 368] bw ( KiB/s): min= 230, max= 496, per=24.93%, avg=356.18, stdev=71.43, samples=56 iops : min= 57, max= 124, avg=88.66, stdev=17.86, samples=56 lat (msec) : 50=0.51%, 100=1.72%, 250=89.65%, 500=8.12% cpu : usr=0.18%, sys=1.27%, ctx=2832, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1430KiB/s (1464kB/s), 357KiB/s-358KiB/s (366kB/s-367kB/s), io=40.0MiB (41.9MB), run=28573-28653msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180791: Fri Aug 14 21:19:18 2026 read: IOPS=50, BW=201KiB/s (206kB/s)(5048KiB/25101msec) slat (usec): min=60, max=46280, avg=10512.50, stdev=5580.30 clat (msec): min=20, max=305, avg=148.83, stdev=35.24 lat (msec): min=31, max=311, avg=159.35, stdev=36.87 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 99], 10.00th=[ 108], 20.00th=[ 120], | 30.00th=[ 130], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 178], 90.00th=[ 197], 95.00th=[ 213], | 99.00th=[ 243], 99.50th=[ 251], 99.90th=[ 284], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 134, max= 278, per=25.24%, avg=200.43, stdev=35.05, samples=49 iops : min= 33, max= 69, avg=49.73, stdev= 8.84, samples=49 write: IOPS=51, BW=207KiB/s (212kB/s)(5192KiB/25101msec); 0 zone resets slat (usec): min=61, max=9159, avg=332.50, stdev=626.63 clat (msec): min=49, max=323, avg=153.71, stdev=36.15 lat (msec): min=49, max=324, avg=154.04, stdev=36.15 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 104], 10.00th=[ 113], 20.00th=[ 125], | 30.00th=[ 133], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 255], 99.50th=[ 271], 99.90th=[ 321], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 103, max= 303, per=25.02%, avg=206.88, stdev=45.79, samples=49 iops : min= 25, max= 75, avg=51.35, stdev=11.45, samples=49 lat (msec) : 50=0.16%, 100=4.92%, 250=93.98%, 500=0.94% cpu : usr=0.27%, sys=2.37%, ctx=4764, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180792: Fri Aug 14 21:19:18 2026 read: IOPS=49, BW=197KiB/s (202kB/s)(4976KiB/25258msec) slat (usec): min=2307, max=44409, avg=10532.40, stdev=5444.19 clat (msec): min=41, max=270, avg=150.47, stdev=35.57 lat (msec): min=59, max=291, avg=161.01, stdev=37.10 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 100], 10.00th=[ 109], 20.00th=[ 121], | 30.00th=[ 131], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 178], 90.00th=[ 199], 95.00th=[ 218], | 99.00th=[ 247], 99.50th=[ 262], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 94, max= 254, per=24.72%, avg=196.26, stdev=34.90, samples=50 iops : min= 23, max= 63, avg=48.70, stdev= 8.72, samples=50 write: IOPS=52, BW=208KiB/s (213kB/s)(5264KiB/25258msec); 0 zone resets slat (usec): min=63, max=18003, avg=347.35, stdev=832.69 clat (msec): min=76, max=266, avg=152.98, stdev=32.33 lat (msec): min=76, max=267, avg=153.33, stdev=32.36 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 105], 10.00th=[ 115], 20.00th=[ 126], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 180], 90.00th=[ 199], 95.00th=[ 213], | 99.00th=[ 241], 99.50th=[ 249], 99.90th=[ 264], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 128, max= 303, per=25.18%, avg=208.20, stdev=47.95, samples=50 iops : min= 32, max= 75, avg=51.68, stdev=12.08, samples=50 lat (msec) : 50=0.04%, 100=4.06%, 250=95.20%, 500=0.70% cpu : usr=0.24%, sys=2.31%, ctx=4950, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180793: Fri Aug 14 21:19:18 2026 read: IOPS=49, BW=198KiB/s (202kB/s)(4984KiB/25233msec) slat (usec): min=2937, max=46872, avg=10732.21, stdev=5347.87 clat (msec): min=39, max=277, avg=148.85, stdev=34.66 lat (msec): min=63, max=290, avg=159.58, stdev=36.24 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 97], 10.00th=[ 108], 20.00th=[ 120], | 30.00th=[ 129], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 178], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 243], 99.50th=[ 251], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 101, max= 303, per=24.88%, avg=197.52, stdev=41.58, samples=50 iops : min= 25, max= 75, avg=49.02, stdev=10.43, samples=50 write: IOPS=52, BW=208KiB/s (213kB/s)(5256KiB/25233msec); 0 zone resets slat (usec): min=57, max=8866, avg=320.20, stdev=604.59 clat (msec): min=82, max=279, avg=153.45, stdev=33.18 lat (msec): min=82, max=279, avg=153.78, stdev=33.19 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 106], 10.00th=[ 113], 20.00th=[ 126], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 201], 95.00th=[ 213], | 99.00th=[ 239], 99.50th=[ 253], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 119, max= 304, per=25.10%, avg=207.54, stdev=43.41, samples=50 iops : min= 29, max= 76, avg=51.52, stdev=10.94, samples=50 lat (msec) : 50=0.04%, 100=4.18%, 250=95.16%, 500=0.62% cpu : usr=0.27%, sys=2.17%, ctx=4967, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180794: Fri Aug 14 21:19:18 2026 read: IOPS=50, BW=200KiB/s (205kB/s)(5060KiB/25252msec) slat (usec): min=88, max=49235, avg=10605.35, stdev=5698.74 clat (msec): min=10, max=286, avg=149.28, stdev=36.06 lat (msec): min=25, max=299, avg=159.90, stdev=37.83 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 100], 10.00th=[ 108], 20.00th=[ 118], | 30.00th=[ 128], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 180], 90.00th=[ 194], 95.00th=[ 215], | 99.00th=[ 251], 99.50th=[ 262], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 119, max= 296, per=25.20%, avg=200.08, stdev=45.98, samples=50 iops : min= 29, max= 74, avg=49.66, stdev=11.51, samples=50 write: IOPS=51, BW=205KiB/s (210kB/s)(5180KiB/25252msec); 0 zone resets slat (usec): min=60, max=7987, avg=334.29, stdev=599.24 clat (msec): min=56, max=296, avg=154.16, stdev=33.91 lat (msec): min=56, max=297, avg=154.49, stdev=33.88 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 126], | 30.00th=[ 134], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 201], 95.00th=[ 215], | 99.00th=[ 249], 99.50th=[ 257], 99.90th=[ 288], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 103, max= 288, per=24.79%, avg=204.98, stdev=43.63, samples=50 iops : min= 25, max= 72, avg=50.88, stdev=11.00, samples=50 lat (msec) : 20=0.12%, 50=0.08%, 100=3.98%, 250=94.92%, 500=0.90% cpu : usr=0.22%, sys=2.29%, ctx=4883, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=795KiB/s (814kB/s), 197KiB/s-201KiB/s (202kB/s-206kB/s), io=19.6MiB (20.5MB), run=25101-25258msec WRITE: bw=827KiB/s (847kB/s), 205KiB/s-208KiB/s (210kB/s-213kB/s), io=20.4MiB (21.4MB), run=25101-25258msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180804: Fri Aug 14 21:19:26 2026 read: IOPS=2, BW=97.8MiB/s (103MB/s)(40.0MiB/409msec) slat (nsec): min=20471k, max=20471k, avg=20471148.00, stdev= 0.00 clat (nsec): min=371278k, max=371278k, avg=371278154.00, stdev= 0.00 lat (nsec): min=391754k, max=391754k, avg=391753810.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.70%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=97.8MiB/s (103MB/s), 97.8MiB/s-97.8MiB/s (103MB/s-103MB/s), io=40.0MiB (41.9MB), run=409-409msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout