== sanity test 398c: run fio to test AIO ================= 13:22:04 (1774891324) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.209436 s, 200 MB/s osc.lustre-OST0000-osc-ffff90e6332df800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148643: Mon Mar 30 13:22:55 2026 write: IOPS=53, BW=215KiB/s (220kB/s)(10.0MiB/47610msec); 0 zone resets slat (usec): min=36, max=8668, avg=221.89, stdev=252.37 clat (msec): min=120, max=634, avg=296.93, stdev=81.99 lat (msec): min=120, max=634, avg=297.16, stdev=82.00 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 184], 10.00th=[ 211], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 268], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 326], 80.00th=[ 359], 90.00th=[ 405], 95.00th=[ 451], | 99.00th=[ 535], 99.50th=[ 558], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 119, max= 410, per=24.97%, avg=214.49, stdev=52.77, samples=94 iops : min= 29, max= 102, avg=53.47, stdev=13.25, samples=94 lat (msec) : 250=31.21%, 500=66.02%, 750=2.77% cpu : usr=0.11%, sys=0.82%, ctx=2998, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148644: Mon Mar 30 13:22:55 2026 write: IOPS=53, BW=215KiB/s (220kB/s)(10.0MiB/47573msec); 0 zone resets slat (usec): min=34, max=3755, avg=225.77, stdev=228.64 clat (msec): min=46, max=589, avg=296.64, stdev=82.06 lat (msec): min=46, max=589, avg=296.87, stdev=82.07 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 182], 10.00th=[ 209], 20.00th=[ 232], | 30.00th=[ 251], 40.00th=[ 268], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 330], 80.00th=[ 359], 90.00th=[ 405], 95.00th=[ 451], | 99.00th=[ 535], 99.50th=[ 558], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 103, max= 405, per=24.99%, avg=214.69, stdev=52.37, samples=94 iops : min= 25, max= 101, avg=53.52, stdev=13.16, samples=94 lat (msec) : 50=0.04%, 100=0.04%, 250=29.73%, 500=67.89%, 750=2.30% cpu : usr=0.12%, sys=0.83%, ctx=3032, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148645: Mon Mar 30 13:22:55 2026 write: IOPS=53, BW=215KiB/s (220kB/s)(10.0MiB/47630msec); 0 zone resets slat (usec): min=35, max=3275, avg=211.94, stdev=175.72 clat (msec): min=122, max=589, avg=297.17, stdev=80.90 lat (msec): min=123, max=589, avg=297.39, stdev=80.90 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 188], 10.00th=[ 209], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 268], 50.00th=[ 288], 60.00th=[ 305], | 70.00th=[ 326], 80.00th=[ 355], 90.00th=[ 414], 95.00th=[ 456], | 99.00th=[ 535], 99.50th=[ 550], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 592] bw ( KiB/s): min= 103, max= 384, per=24.92%, avg=214.07, stdev=53.34, samples=94 iops : min= 25, max= 96, avg=53.37, stdev=13.40, samples=94 lat (msec) : 250=30.66%, 500=67.15%, 750=2.19% cpu : usr=0.09%, sys=0.83%, ctx=2921, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148646: Mon Mar 30 13:22:55 2026 write: IOPS=53, BW=215KiB/s (220kB/s)(10.0MiB/47631msec); 0 zone resets slat (usec): min=42, max=4822, avg=228.63, stdev=240.50 clat (msec): min=58, max=589, avg=297.29, stdev=82.66 lat (msec): min=59, max=589, avg=297.52, stdev=82.66 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 188], 10.00th=[ 211], 20.00th=[ 230], | 30.00th=[ 249], 40.00th=[ 268], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 326], 80.00th=[ 359], 90.00th=[ 414], 95.00th=[ 456], | 99.00th=[ 535], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 592] bw ( KiB/s): min= 119, max= 378, per=24.98%, avg=214.56, stdev=53.90, samples=94 iops : min= 29, max= 94, avg=53.49, stdev=13.53, samples=94 lat (msec) : 100=0.35%, 250=30.47%, 500=66.72%, 750=2.46% cpu : usr=0.09%, sys=0.84%, ctx=3070, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=860KiB/s (881kB/s), 215KiB/s-215KiB/s (220kB/s-220kB/s), io=40.0MiB (41.9MB), run=47573-47631msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148663: Mon Mar 30 13:23:34 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(5048KiB/35124msec) slat (usec): min=42, max=17160, avg=4620.05, stdev=2420.48 clat (msec): min=6, max=356, avg=138.76, stdev=44.37 lat (msec): min=9, max=365, avg=143.38, stdev=45.23 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 106], | 30.00th=[ 114], 40.00th=[ 123], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 199], 95.00th=[ 226], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 39, max= 280, per=25.36%, avg=144.04, stdev=46.66, samples=69 iops : min= 9, max= 70, avg=35.78, stdev=11.72, samples=69 write: IOPS=36, BW=148KiB/s (151kB/s)(5192KiB/35124msec); 0 zone resets slat (usec): min=42, max=5174, avg=229.86, stdev=298.33 clat (msec): min=34, max=637, avg=293.01, stdev=92.40 lat (msec): min=34, max=637, avg=293.25, stdev=92.40 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 174], 10.00th=[ 190], 20.00th=[ 215], | 30.00th=[ 239], 40.00th=[ 259], 50.00th=[ 279], 60.00th=[ 300], | 70.00th=[ 326], 80.00th=[ 368], 90.00th=[ 414], 95.00th=[ 456], | 99.00th=[ 584], 99.50th=[ 617], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 63, max= 226, per=24.98%, avg=147.64, stdev=38.19, samples=69 iops : min= 15, max= 56, avg=36.68, stdev= 9.69, samples=69 lat (msec) : 10=0.04%, 50=0.82%, 100=7.07%, 250=57.66%, 500=32.93% lat (msec) : 750=1.48% cpu : usr=0.18%, sys=1.48%, ctx=5062, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148664: Mon Mar 30 13:23:34 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(4976KiB/35318msec) slat (usec): min=1340, max=15563, avg=4730.93, stdev=2335.82 clat (msec): min=5, max=316, avg=138.70, stdev=41.88 lat (msec): min=15, max=321, avg=143.43, stdev=42.66 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 84], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 134], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 171], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 264], 99.50th=[ 288], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 39, max= 248, per=24.64%, avg=139.94, stdev=45.94, samples=70 iops : min= 9, max= 62, avg=34.76, stdev=11.56, samples=70 write: IOPS=37, BW=149KiB/s (153kB/s)(5264KiB/35318msec); 0 zone resets slat (usec): min=40, max=4130, avg=228.66, stdev=257.11 clat (msec): min=78, max=677, avg=292.95, stdev=93.76 lat (msec): min=78, max=678, avg=293.18, stdev=93.76 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 169], 10.00th=[ 190], 20.00th=[ 213], | 30.00th=[ 236], 40.00th=[ 259], 50.00th=[ 279], 60.00th=[ 305], | 70.00th=[ 330], 80.00th=[ 359], 90.00th=[ 418], 95.00th=[ 456], | 99.00th=[ 592], 99.50th=[ 625], 99.90th=[ 659], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 63, max= 232, per=25.15%, avg=148.64, stdev=38.73, samples=70 iops : min= 15, max= 58, avg=36.94, stdev= 9.82, samples=70 lat (msec) : 10=0.04%, 20=0.08%, 50=0.31%, 100=7.03%, 250=59.34% lat (msec) : 500=31.68%, 750=1.52% cpu : usr=0.16%, sys=1.48%, ctx=5190, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148665: Mon Mar 30 13:23:34 2026 read: IOPS=35, BW=141KiB/s (145kB/s)(4984KiB/35317msec) slat (usec): min=1335, max=20960, avg=4618.39, stdev=2372.61 clat (msec): min=5, max=333, avg=138.91, stdev=42.15 lat (msec): min=7, max=343, avg=143.54, stdev=42.95 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 134], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 194], 95.00th=[ 215], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 31, max= 272, per=24.79%, avg=140.83, stdev=51.84, samples=70 iops : min= 7, max= 68, avg=34.99, stdev=13.06, samples=70 write: IOPS=37, BW=149KiB/s (152kB/s)(5256KiB/35317msec); 0 zone resets slat (usec): min=46, max=4217, avg=221.64, stdev=215.73 clat (msec): min=32, max=655, avg=293.13, stdev=93.65 lat (msec): min=32, max=656, avg=293.35, stdev=93.66 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 167], 10.00th=[ 186], 20.00th=[ 215], | 30.00th=[ 236], 40.00th=[ 259], 50.00th=[ 279], 60.00th=[ 300], | 70.00th=[ 330], 80.00th=[ 368], 90.00th=[ 418], 95.00th=[ 456], | 99.00th=[ 592], 99.50th=[ 609], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 71, max= 240, per=25.00%, avg=147.77, stdev=36.90, samples=70 iops : min= 17, max= 60, avg=36.73, stdev= 9.35, samples=70 lat (msec) : 10=0.08%, 20=0.08%, 50=0.35%, 100=6.99%, 250=58.63% lat (msec) : 500=32.30%, 750=1.56% cpu : usr=0.15%, sys=1.48%, ctx=5136, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148666: Mon Mar 30 13:23:34 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(5060KiB/35200msec) slat (usec): min=1304, max=19318, avg=4732.56, stdev=2398.78 clat (msec): min=5, max=300, avg=139.64, stdev=43.91 lat (msec): min=11, max=308, avg=144.38, stdev=44.68 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 85], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 123], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 203], 95.00th=[ 226], | 99.00th=[ 271], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 63, max= 256, per=25.39%, avg=144.19, stdev=45.53, samples=69 iops : min= 15, max= 64, avg=35.83, stdev=11.49, samples=69 write: IOPS=36, BW=147KiB/s (151kB/s)(5180KiB/35200msec); 0 zone resets slat (usec): min=55, max=3051, avg=223.98, stdev=196.41 clat (msec): min=47, max=671, avg=293.04, stdev=90.96 lat (msec): min=47, max=672, avg=293.27, stdev=90.96 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 171], 10.00th=[ 190], 20.00th=[ 215], | 30.00th=[ 241], 40.00th=[ 264], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 326], 80.00th=[ 363], 90.00th=[ 409], 95.00th=[ 447], | 99.00th=[ 584], 99.50th=[ 609], 99.90th=[ 667], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 63, max= 232, per=24.75%, avg=146.25, stdev=38.08, samples=69 iops : min= 15, max= 58, avg=36.33, stdev= 9.67, samples=69 lat (msec) : 10=0.12%, 20=0.04%, 50=0.39%, 100=7.07%, 250=57.93% lat (msec) : 500=33.09%, 750=1.37% cpu : usr=0.17%, sys=1.50%, ctx=5238, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=568KiB/s (582kB/s), 141KiB/s-144KiB/s (144kB/s-147kB/s), io=19.6MiB (20.5MB), run=35124-35318msec WRITE: bw=592KiB/s (606kB/s), 147KiB/s-149KiB/s (151kB/s-153kB/s), io=20.4MiB (21.4MB), run=35124-35318msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148675: Mon Mar 30 13:23:38 2026 read: IOPS=6, BW=260MiB/s (272MB/s)(40.0MiB/154msec) slat (nsec): min=20158k, max=20158k, avg=20158258.00, stdev= 0.00 clat (nsec): min=124491k, max=124491k, avg=124491147.00, stdev= 0.00 lat (nsec): min=144667k, max=144667k, avg=144667423.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 125], | 70.00th=[ 125], 80.00th=[ 125], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.84%, ctx=34, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=260MiB/s (272MB/s), 260MiB/s-260MiB/s (272MB/s-272MB/s), io=40.0MiB (41.9MB), run=154-154msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout