== sanity test 398c: run fio to test AIO ================= 22:37:28 (1788921448) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.364921 s, 115 MB/s osc.lustre-OST0000-osc-ffff8a6e79321800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=188265: Tue Sep 8 22:37:55 2026 write: IOPS=121, BW=487KiB/s (498kB/s)(10.0MiB/21044msec); 0 zone resets slat (usec): min=37, max=41928, avg=336.10, stdev=1154.04 clat (msec): min=22, max=283, avg=130.35, stdev=32.12 lat (msec): min=23, max=284, avg=130.69, stdev=32.11 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 83], 10.00th=[ 91], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 134], | 70.00th=[ 142], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 188], | 99.00th=[ 230], 99.50th=[ 249], 99.90th=[ 259], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 272, max= 736, per=25.01%, avg=486.68, stdev=90.13, samples=41 iops : min= 68, max= 184, avg=121.29, stdev=22.56, samples=41 lat (msec) : 50=0.12%, 100=14.73%, 250=84.73%, 500=0.43% cpu : usr=0.26%, sys=1.88%, ctx=2752, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188266: Tue Sep 8 22:37:55 2026 write: IOPS=121, BW=487KiB/s (499kB/s)(10.0MiB/21023msec); 0 zone resets slat (usec): min=35, max=6892, avg=201.91, stdev=483.16 clat (msec): min=36, max=269, avg=130.68, stdev=31.79 lat (msec): min=37, max=269, avg=130.89, stdev=31.79 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 86], 10.00th=[ 92], 20.00th=[ 107], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 132], | 70.00th=[ 142], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 194], | 99.00th=[ 222], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 304, max= 720, per=25.06%, avg=487.71, stdev=88.01, samples=41 iops : min= 76, max= 180, avg=121.59, stdev=22.01, samples=41 lat (msec) : 50=0.04%, 100=14.84%, 250=85.04%, 500=0.08% cpu : usr=0.20%, sys=1.35%, ctx=1547, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188267: Tue Sep 8 22:37:55 2026 write: IOPS=122, BW=489KiB/s (500kB/s)(10.0MiB/20961msec); 0 zone resets slat (usec): min=35, max=10241, avg=299.56, stdev=649.52 clat (msec): min=17, max=281, avg=130.42, stdev=33.43 lat (msec): min=18, max=282, avg=130.72, stdev=33.42 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 91], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 134], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 188], | 99.00th=[ 230], 99.50th=[ 264], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 284] bw ( KiB/s): min= 288, max= 704, per=25.17%, avg=489.85, stdev=84.52, samples=41 iops : min= 72, max= 176, avg=122.10, stdev=21.14, samples=41 lat (msec) : 20=0.08%, 50=0.20%, 100=14.53%, 250=84.53%, 500=0.66% cpu : usr=0.27%, sys=1.90%, ctx=3112, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188268: Tue Sep 8 22:37:55 2026 write: IOPS=121, BW=487KiB/s (498kB/s)(10.0MiB/21042msec); 0 zone resets slat (usec): min=36, max=12978, avg=281.78, stdev=658.23 clat (msec): min=15, max=280, avg=130.25, stdev=32.49 lat (msec): min=16, max=280, avg=130.53, stdev=32.51 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 81], 10.00th=[ 91], 20.00th=[ 107], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 134], | 70.00th=[ 142], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 188], | 99.00th=[ 224], 99.50th=[ 259], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 312, max= 736, per=25.00%, avg=486.49, stdev=86.85, samples=41 iops : min= 78, max= 184, avg=121.24, stdev=21.74, samples=41 lat (msec) : 20=0.12%, 50=0.12%, 100=14.61%, 250=84.61%, 500=0.55% cpu : usr=0.20%, sys=1.82%, ctx=2757, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1946KiB/s (1993kB/s), 487KiB/s-489KiB/s (498kB/s-500kB/s), io=40.0MiB (41.9MB), run=20961-21044msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=188281: Tue Sep 8 22:38:25 2026 read: IOPS=56, BW=227KiB/s (232kB/s)(5048KiB/22247msec) slat (usec): min=2711, max=46184, avg=9709.05, stdev=5050.41 clat (msec): min=61, max=278, avg=130.41, stdev=31.78 lat (msec): min=73, max=297, avg=140.13, stdev=32.93 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 112], 40.00th=[ 118], 50.00th=[ 126], 60.00th=[ 133], | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 178], 95.00th=[ 190], | 99.00th=[ 220], 99.50th=[ 224], 99.90th=[ 245], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 103, max= 319, per=25.11%, avg=226.20, stdev=53.56, samples=44 iops : min= 25, max= 79, avg=56.11, stdev=13.50, samples=44 write: IOPS=58, BW=233KiB/s (239kB/s)(5192KiB/22247msec); 0 zone resets slat (usec): min=49, max=13605, avg=300.29, stdev=634.86 clat (msec): min=35, max=270, avg=135.90, stdev=31.94 lat (msec): min=36, max=270, avg=136.20, stdev=31.95 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 95], 10.00th=[ 101], 20.00th=[ 109], | 30.00th=[ 116], 40.00th=[ 123], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 150], 80.00th=[ 163], 90.00th=[ 180], 95.00th=[ 197], | 99.00th=[ 234], 99.50th=[ 253], 99.90th=[ 268], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 135, max= 335, per=24.82%, avg=232.80, stdev=47.12, samples=44 iops : min= 33, max= 83, avg=57.77, stdev=11.78, samples=44 lat (msec) : 50=0.04%, 100=12.62%, 250=86.99%, 500=0.35% cpu : usr=0.32%, sys=2.57%, ctx=4710, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188282: Tue Sep 8 22:38:25 2026 read: IOPS=55, BW=224KiB/s (229kB/s)(4976KiB/22255msec) slat (usec): min=2658, max=44019, avg=9814.29, stdev=5090.41 clat (msec): min=26, max=290, avg=131.18, stdev=32.04 lat (msec): min=34, max=297, avg=141.00, stdev=33.27 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 88], 10.00th=[ 96], 20.00th=[ 106], | 30.00th=[ 113], 40.00th=[ 120], 50.00th=[ 126], 60.00th=[ 134], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 176], 95.00th=[ 192], | 99.00th=[ 220], 99.50th=[ 228], 99.90th=[ 284], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 111, max= 327, per=24.69%, avg=222.45, stdev=49.26, samples=44 iops : min= 27, max= 81, avg=55.18, stdev=12.33, samples=44 write: IOPS=59, BW=237KiB/s (242kB/s)(5264KiB/22255msec); 0 zone resets slat (usec): min=49, max=8804, avg=316.17, stdev=657.98 clat (msec): min=51, max=281, avg=135.35, stdev=32.07 lat (msec): min=51, max=282, avg=135.67, stdev=32.07 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 94], 10.00th=[ 100], 20.00th=[ 109], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 180], 95.00th=[ 197], | 99.00th=[ 230], 99.50th=[ 249], 99.90th=[ 259], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 151, max= 367, per=25.14%, avg=235.82, stdev=49.75, samples=44 iops : min= 37, max= 91, avg=58.52, stdev=12.52, samples=44 lat (msec) : 50=0.04%, 100=12.11%, 250=87.54%, 500=0.31% cpu : usr=0.28%, sys=2.66%, ctx=4783, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188283: Tue Sep 8 22:38:25 2026 read: IOPS=56, BW=224KiB/s (229kB/s)(4984KiB/22245msec) slat (usec): min=2604, max=47693, avg=9837.44, stdev=5298.88 clat (msec): min=47, max=278, avg=132.16, stdev=33.34 lat (msec): min=61, max=293, avg=142.01, stdev=34.48 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 113], 40.00th=[ 121], 50.00th=[ 126], 60.00th=[ 136], | 70.00th=[ 146], 80.00th=[ 159], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 222], 99.50th=[ 234], 99.90th=[ 247], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 157, max= 352, per=24.85%, avg=223.91, stdev=45.59, samples=44 iops : min= 39, max= 88, avg=55.55, stdev=11.47, samples=44 write: IOPS=59, BW=236KiB/s (242kB/s)(5256KiB/22245msec); 0 zone resets slat (usec): min=38, max=5253, avg=272.75, stdev=440.22 clat (msec): min=21, max=257, avg=134.26, stdev=31.28 lat (msec): min=21, max=257, avg=134.53, stdev=31.30 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 100], 20.00th=[ 110], | 30.00th=[ 116], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 161], 90.00th=[ 180], 95.00th=[ 192], | 99.00th=[ 224], 99.50th=[ 232], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 119, max= 360, per=25.01%, avg=234.61, stdev=56.32, samples=44 iops : min= 29, max= 90, avg=58.20, stdev=14.13, samples=44 lat (msec) : 50=0.16%, 100=13.12%, 250=86.56%, 500=0.16% cpu : usr=0.32%, sys=2.61%, ctx=4765, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188284: Tue Sep 8 22:38:25 2026 read: IOPS=57, BW=228KiB/s (234kB/s)(5060KiB/22159msec) slat (usec): min=88, max=54237, avg=9691.35, stdev=5178.84 clat (msec): min=14, max=277, avg=131.55, stdev=33.29 lat (msec): min=15, max=294, avg=141.25, stdev=34.66 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 112], 40.00th=[ 121], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 178], 95.00th=[ 197], | 99.00th=[ 222], 99.50th=[ 230], 99.90th=[ 259], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 116, max= 351, per=25.20%, avg=227.07, stdev=52.31, samples=43 iops : min= 29, max= 87, avg=56.33, stdev=13.07, samples=43 write: IOPS=58, BW=234KiB/s (239kB/s)(5180KiB/22159msec); 0 zone resets slat (usec): min=44, max=11166, avg=307.09, stdev=643.12 clat (msec): min=47, max=291, avg=134.90, stdev=32.95 lat (msec): min=47, max=291, avg=135.21, stdev=32.97 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 94], 10.00th=[ 101], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 180], 95.00th=[ 194], | 99.00th=[ 245], 99.50th=[ 257], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 111, max= 367, per=24.69%, avg=231.60, stdev=47.93, samples=43 iops : min= 27, max= 91, avg=57.44, stdev=12.02, samples=43 lat (msec) : 20=0.04%, 50=0.16%, 100=12.73%, 250=86.52%, 500=0.55% cpu : usr=0.32%, sys=2.70%, ctx=4571, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=902KiB/s (923kB/s), 224KiB/s-228KiB/s (229kB/s-234kB/s), io=19.6MiB (20.5MB), run=22159-22255msec WRITE: bw=939KiB/s (961kB/s), 233KiB/s-237KiB/s (239kB/s-242kB/s), io=20.4MiB (21.4MB), run=22159-22255msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=188292: Tue Sep 8 22:38:32 2026 read: IOPS=2, BW=113MiB/s (118MB/s)(40.0MiB/354msec) slat (nsec): min=23643k, max=23643k, avg=23643330.00, stdev= 0.00 clat (nsec): min=308542k, max=308542k, avg=308542489.00, stdev= 0.00 lat (nsec): min=332191k, max=332191k, avg=332190952.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 500=100.00% cpu : usr=0.28%, sys=3.12%, ctx=12, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=113MiB/s (118MB/s), 113MiB/s-113MiB/s (118MB/s-118MB/s), io=40.0MiB (41.9MB), run=354-354msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout