== sanity test 398c: run fio to test AIO ================= 08:08:30 (1787832510) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.723887 s, 57.9 MB/s osc.lustre-OST0000-osc-ffff90c1c6458800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179817: Thu Aug 27 08:09:16 2026 write: IOPS=77, BW=312KiB/s (319kB/s)(10.0MiB/32869msec); 0 zone resets slat (usec): min=29, max=17427, avg=283.27, stdev=967.95 clat (msec): min=48, max=366, avg=204.79, stdev=52.43 lat (msec): min=50, max=366, avg=205.08, stdev=52.45 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 132], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 368] bw ( KiB/s): min= 143, max= 546, per=24.97%, avg=311.14, stdev=76.59, samples=64 iops : min= 35, max= 136, avg=77.34, stdev=19.22, samples=64 lat (msec) : 50=0.04%, 100=0.35%, 250=82.07%, 500=17.54% cpu : usr=0.20%, sys=1.01%, ctx=2222, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179818: Thu Aug 27 08:09:16 2026 write: IOPS=77, BW=312KiB/s (319kB/s)(10.0MiB/32864msec); 0 zone resets slat (usec): min=28, max=16662, avg=381.37, stdev=1136.64 clat (msec): min=74, max=376, avg=204.88, stdev=50.81 lat (msec): min=74, max=376, avg=205.26, stdev=50.84 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 199, max= 530, per=24.89%, avg=310.08, stdev=64.85, samples=64 iops : min= 49, max= 132, avg=77.06, stdev=16.27, samples=64 lat (msec) : 100=0.59%, 250=81.37%, 500=18.05% cpu : usr=0.22%, sys=1.19%, ctx=2871, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179819: Thu Aug 27 08:09:16 2026 write: IOPS=77, BW=312KiB/s (319kB/s)(10.0MiB/32841msec); 0 zone resets slat (usec): min=31, max=24188, avg=427.38, stdev=1457.29 clat (msec): min=31, max=400, avg=204.59, stdev=52.46 lat (msec): min=32, max=400, avg=205.02, stdev=52.44 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 368], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 159, max= 546, per=25.01%, avg=311.59, stdev=71.33, samples=64 iops : min= 39, max= 136, avg=77.42, stdev=17.87, samples=64 lat (msec) : 50=0.08%, 100=0.43%, 250=79.69%, 500=19.80% cpu : usr=0.23%, sys=1.17%, ctx=2724, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179820: Thu Aug 27 08:09:16 2026 write: IOPS=77, BW=312KiB/s (319kB/s)(10.0MiB/32857msec); 0 zone resets slat (usec): min=25, max=23647, avg=437.01, stdev=1393.12 clat (msec): min=23, max=392, avg=204.68, stdev=53.86 lat (msec): min=23, max=393, avg=205.12, stdev=53.85 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 368], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 183, max= 530, per=24.96%, avg=311.02, stdev=69.58, samples=64 iops : min= 45, max= 132, avg=77.30, stdev=17.45, samples=64 lat (msec) : 50=0.16%, 100=1.02%, 250=80.00%, 500=18.83% cpu : usr=0.26%, sys=1.56%, ctx=3043, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1246KiB/s (1276kB/s), 312KiB/s-312KiB/s (319kB/s-319kB/s), io=40.0MiB (41.9MB), run=32841-32869msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179835: Thu Aug 27 08:10:09 2026 read: IOPS=29, BW=118KiB/s (121kB/s)(5048KiB/42776msec) slat (usec): min=33, max=83177, avg=18971.52, stdev=11190.05 clat (msec): min=19, max=541, avg=257.26, stdev=70.56 lat (msec): min=48, max=570, avg=276.24, stdev=73.05 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 155], 10.00th=[ 171], 20.00th=[ 199], | 30.00th=[ 218], 40.00th=[ 234], 50.00th=[ 251], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 317], 90.00th=[ 355], 95.00th=[ 384], | 99.00th=[ 447], 99.50th=[ 468], 99.90th=[ 531], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 62, max= 214, per=25.10%, avg=117.20, stdev=32.98, samples=84 iops : min= 15, max= 53, avg=28.70, stdev= 8.28, samples=84 write: IOPS=30, BW=121KiB/s (124kB/s)(5192KiB/42776msec); 0 zone resets slat (usec): min=29, max=28002, avg=424.89, stdev=1642.52 clat (msec): min=84, max=524, avg=257.59, stdev=67.75 lat (msec): min=84, max=524, avg=258.02, stdev=67.72 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 165], 10.00th=[ 180], 20.00th=[ 203], | 30.00th=[ 218], 40.00th=[ 234], 50.00th=[ 247], 60.00th=[ 266], | 70.00th=[ 284], 80.00th=[ 313], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 439], 99.50th=[ 493], 99.90th=[ 514], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 55, max= 190, per=24.86%, avg=120.83, stdev=38.90, samples=84 iops : min= 13, max= 47, avg=29.62, stdev= 9.79, samples=84 lat (msec) : 20=0.04%, 100=0.08%, 250=50.31%, 500=49.38%, 750=0.20% cpu : usr=0.29%, sys=1.87%, ctx=4310, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179836: Thu Aug 27 08:10:09 2026 read: IOPS=28, BW=116KiB/s (119kB/s)(4976KiB/42942msec) slat (usec): min=2778, max=93352, avg=19318.72, stdev=11611.71 clat (msec): min=101, max=605, avg=255.34, stdev=72.07 lat (msec): min=115, max=638, avg=274.67, stdev=75.23 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 192], | 30.00th=[ 211], 40.00th=[ 230], 50.00th=[ 247], 60.00th=[ 266], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 451], 99.50th=[ 489], 99.90th=[ 592], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 49, max= 190, per=24.65%, avg=115.12, stdev=31.98, samples=84 iops : min= 12, max= 47, avg=28.20, stdev= 8.02, samples=84 write: IOPS=30, BW=123KiB/s (126kB/s)(5264KiB/42942msec); 0 zone resets slat (usec): min=32, max=31651, avg=356.04, stdev=1243.00 clat (msec): min=55, max=560, avg=259.22, stdev=67.14 lat (msec): min=55, max=560, avg=259.58, stdev=67.16 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 167], 10.00th=[ 182], 20.00th=[ 203], | 30.00th=[ 220], 40.00th=[ 234], 50.00th=[ 251], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 355], 95.00th=[ 384], | 99.00th=[ 439], 99.50th=[ 485], 99.90th=[ 531], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 39, max= 208, per=25.18%, avg=122.36, stdev=37.40, samples=84 iops : min= 9, max= 52, avg=30.01, stdev= 9.40, samples=84 lat (msec) : 100=0.08%, 250=50.43%, 500=49.06%, 750=0.43% cpu : usr=0.27%, sys=1.70%, ctx=4340, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179837: Thu Aug 27 08:10:09 2026 read: IOPS=29, BW=117KiB/s (119kB/s)(4984KiB/42737msec) slat (usec): min=3261, max=89986, avg=19141.03, stdev=11208.65 clat (msec): min=24, max=595, avg=253.00, stdev=68.73 lat (msec): min=40, max=618, avg=272.16, stdev=71.27 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 213], 40.00th=[ 230], 50.00th=[ 247], 60.00th=[ 266], | 70.00th=[ 284], 80.00th=[ 305], 90.00th=[ 347], 95.00th=[ 376], | 99.00th=[ 443], 99.50th=[ 468], 99.90th=[ 575], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 47, max= 197, per=24.96%, avg=116.56, stdev=35.30, samples=84 iops : min= 11, max= 49, avg=28.56, stdev= 8.87, samples=84 write: IOPS=30, BW=123KiB/s (126kB/s)(5256KiB/42737msec); 0 zone resets slat (usec): min=34, max=18216, avg=352.25, stdev=1131.37 clat (msec): min=90, max=593, avg=258.31, stdev=68.27 lat (msec): min=91, max=593, avg=258.66, stdev=68.26 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 165], 10.00th=[ 180], 20.00th=[ 199], | 30.00th=[ 215], 40.00th=[ 232], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 380], | 99.00th=[ 426], 99.50th=[ 510], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 23, max= 198, per=25.18%, avg=122.39, stdev=35.17, samples=84 iops : min= 5, max= 49, avg=30.01, stdev= 8.81, samples=84 lat (msec) : 50=0.08%, 100=0.04%, 250=51.33%, 500=48.09%, 750=0.47% cpu : usr=0.26%, sys=1.64%, ctx=4343, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179838: Thu Aug 27 08:10:09 2026 read: IOPS=29, BW=118KiB/s (121kB/s)(5060KiB/42916msec) slat (usec): min=3883, max=93867, avg=19435.60, stdev=11766.41 clat (msec): min=109, max=576, avg=254.97, stdev=68.48 lat (msec): min=119, max=612, avg=274.41, stdev=71.84 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 159], 10.00th=[ 176], 20.00th=[ 197], | 30.00th=[ 215], 40.00th=[ 232], 50.00th=[ 249], 60.00th=[ 264], | 70.00th=[ 284], 80.00th=[ 309], 90.00th=[ 347], 95.00th=[ 376], | 99.00th=[ 451], 99.50th=[ 493], 99.90th=[ 558], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 55, max= 191, per=25.26%, avg=117.98, stdev=32.11, samples=84 iops : min= 13, max= 47, avg=28.90, stdev= 8.08, samples=84 write: IOPS=30, BW=121KiB/s (124kB/s)(5180KiB/42916msec); 0 zone resets slat (usec): min=31, max=13974, avg=314.41, stdev=940.15 clat (msec): min=123, max=594, avg=258.51, stdev=68.19 lat (msec): min=123, max=594, avg=258.83, stdev=68.22 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 165], 10.00th=[ 180], 20.00th=[ 201], | 30.00th=[ 215], 40.00th=[ 232], 50.00th=[ 247], 60.00th=[ 266], | 70.00th=[ 292], 80.00th=[ 313], 90.00th=[ 355], 95.00th=[ 380], | 99.00th=[ 451], 99.50th=[ 489], 99.90th=[ 550], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 47, max= 221, per=24.70%, avg=120.02, stdev=36.30, samples=84 iops : min= 11, max= 55, avg=29.44, stdev= 9.13, samples=84 lat (msec) : 250=51.56%, 500=48.01%, 750=0.43% cpu : usr=0.30%, sys=1.78%, ctx=4214, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=467KiB/s (479kB/s), 116KiB/s-118KiB/s (119kB/s-121kB/s), io=19.6MiB (20.5MB), run=42737-42942msec WRITE: bw=487KiB/s (498kB/s), 121KiB/s-123KiB/s (124kB/s-126kB/s), io=20.4MiB (21.4MB), run=42737-42942msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179848: Thu Aug 27 08:10:22 2026 read: IOPS=0, BW=37.5MiB/s (39.3MB/s)(40.0MiB/1068msec) slat (nsec): min=60616k, max=60616k, avg=60616085.00, stdev= 0.00 clat (nsec): min=956499k, max=956499k, avg=956498774.00, stdev= 0.00 lat (nsec): min=1017.1M, max=1017.1M, avg=1017121021.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 961], 5.00th=[ 961], 10.00th=[ 961], 20.00th=[ 961], | 30.00th=[ 961], 40.00th=[ 961], 50.00th=[ 961], 60.00th=[ 961], | 70.00th=[ 961], 80.00th=[ 961], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.31%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=37.5MiB/s (39.3MB/s), 37.5MiB/s-37.5MiB/s (39.3MB/s-39.3MB/s), io=40.0MiB (41.9MB), run=1068-1068msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout