== sanity test 398c: run fio to test AIO ================= 06:22:30 (1779704550) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.807784 s, 51.9 MB/s osc.lustre-OST0000-osc-ffff9a4fc3951800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180115: Mon May 25 06:23:15 2026 write: IOPS=86, BW=348KiB/s (356kB/s)(10.0MiB/29445msec); 0 zone resets slat (usec): min=37, max=20027, avg=293.79, stdev=995.70 clat (msec): min=33, max=404, avg=183.47, stdev=60.25 lat (msec): min=34, max=404, avg=183.76, stdev=60.26 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 83], 10.00th=[ 104], 20.00th=[ 131], | 30.00th=[ 146], 40.00th=[ 167], 50.00th=[ 182], 60.00th=[ 197], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 355], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 205, max= 766, per=25.34%, avg=347.72, stdev=102.75, samples=58 iops : min= 51, max= 191, avg=86.55, stdev=25.71, samples=58 lat (msec) : 50=0.08%, 100=8.12%, 250=77.54%, 500=14.26% cpu : usr=0.19%, sys=1.08%, ctx=2845, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180116: Mon May 25 06:23:15 2026 write: IOPS=85, BW=343KiB/s (351kB/s)(10.0MiB/29847msec); 0 zone resets slat (usec): min=36, max=24034, avg=284.12, stdev=981.85 clat (msec): min=23, max=374, avg=184.50, stdev=59.52 lat (msec): min=23, max=399, avg=184.79, stdev=59.57 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 89], 10.00th=[ 106], 20.00th=[ 132], | 30.00th=[ 150], 40.00th=[ 167], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 376] bw ( KiB/s): min= 159, max= 763, per=25.07%, avg=343.98, stdev=116.00, samples=58 iops : min= 39, max= 190, avg=85.62, stdev=28.98, samples=58 lat (msec) : 50=0.08%, 100=7.89%, 250=78.24%, 500=13.79% cpu : usr=0.14%, sys=1.11%, ctx=2748, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180117: Mon May 25 06:23:15 2026 write: IOPS=85, BW=343KiB/s (352kB/s)(10.0MiB/29827msec); 0 zone resets slat (usec): min=37, max=31479, avg=263.94, stdev=956.72 clat (msec): min=11, max=355, avg=184.53, stdev=59.47 lat (msec): min=12, max=355, avg=184.79, stdev=59.50 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 85], 10.00th=[ 108], 20.00th=[ 130], | 30.00th=[ 150], 40.00th=[ 167], 50.00th=[ 184], 60.00th=[ 199], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 355] bw ( KiB/s): min= 158, max= 691, per=25.00%, avg=343.05, stdev=112.01, samples=58 iops : min= 39, max= 172, avg=85.41, stdev=28.03, samples=58 lat (msec) : 20=0.04%, 50=0.08%, 100=7.66%, 250=77.46%, 500=14.77% cpu : usr=0.16%, sys=1.01%, ctx=2452, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180118: Mon May 25 06:23:15 2026 write: IOPS=86, BW=344KiB/s (352kB/s)(10.0MiB/29749msec); 0 zone resets slat (usec): min=37, max=14660, avg=236.59, stdev=703.08 clat (msec): min=30, max=365, avg=184.52, stdev=59.55 lat (msec): min=31, max=365, avg=184.76, stdev=59.56 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 93], 10.00th=[ 108], 20.00th=[ 129], | 30.00th=[ 148], 40.00th=[ 169], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 182, max= 763, per=25.12%, avg=344.62, stdev=117.31, samples=58 iops : min= 45, max= 190, avg=85.79, stdev=29.27, samples=58 lat (msec) : 50=0.04%, 100=7.15%, 250=79.73%, 500=13.09% cpu : usr=0.16%, sys=0.99%, ctx=2113, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1372KiB/s (1405kB/s), 343KiB/s-348KiB/s (351kB/s-356kB/s), io=40.0MiB (41.9MB), run=29445-29847msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180133: Mon May 25 06:24:02 2026 read: IOPS=34, BW=140KiB/s (143kB/s)(5048KiB/36085msec) slat (usec): min=2842, max=77944, avg=16178.89, stdev=9478.46 clat (msec): min=62, max=412, avg=212.85, stdev=51.40 lat (msec): min=74, max=438, avg=229.04, stdev=53.96 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 70, max= 220, per=25.04%, avg=139.21, stdev=34.10, samples=71 iops : min= 17, max= 55, avg=34.31, stdev= 8.56, samples=71 write: IOPS=35, BW=144KiB/s (147kB/s)(5192KiB/36085msec); 0 zone resets slat (usec): min=38, max=11262, avg=300.71, stdev=799.53 clat (msec): min=62, max=398, avg=218.96, stdev=47.59 lat (msec): min=62, max=399, avg=219.27, stdev=47.61 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 153], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 77, max= 223, per=24.83%, avg=143.51, stdev=34.39, samples=71 iops : min= 19, max= 55, avg=35.41, stdev= 8.59, samples=71 lat (msec) : 100=0.27%, 250=78.09%, 500=21.64% cpu : usr=0.23%, sys=1.72%, ctx=4309, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180134: Mon May 25 06:24:02 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(4976KiB/36080msec) slat (usec): min=92, max=92418, avg=16369.27, stdev=9442.72 clat (msec): min=37, max=463, avg=215.84, stdev=53.81 lat (msec): min=50, max=473, avg=232.21, stdev=55.65 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 376], 99.50th=[ 388], 99.90th=[ 426], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 78, max= 200, per=24.66%, avg=137.13, stdev=29.64, samples=71 iops : min= 19, max= 50, avg=33.80, stdev= 7.43, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5264KiB/36080msec); 0 zone resets slat (usec): min=38, max=22765, avg=302.70, stdev=1008.70 clat (msec): min=24, max=473, avg=218.34, stdev=53.60 lat (msec): min=25, max=473, avg=218.65, stdev=53.65 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 372], 99.50th=[ 405], 99.90th=[ 464], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 80, max= 262, per=25.14%, avg=145.31, stdev=40.27, samples=71 iops : min= 20, max= 65, avg=35.87, stdev=10.12, samples=71 lat (msec) : 50=0.12%, 100=0.43%, 250=76.09%, 500=23.36% cpu : usr=0.24%, sys=1.78%, ctx=4138, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180135: Mon May 25 06:24:02 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(4984KiB/36068msec) slat (usec): min=2504, max=92491, avg=16006.59, stdev=9473.51 clat (msec): min=11, max=399, avg=214.25, stdev=52.37 lat (msec): min=20, max=413, avg=230.27, stdev=54.72 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 372], 99.90th=[ 388], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 66, max= 236, per=24.86%, avg=138.24, stdev=35.05, samples=71 iops : min= 16, max= 59, avg=34.08, stdev= 8.80, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5256KiB/36068msec); 0 zone resets slat (usec): min=54, max=13898, avg=293.88, stdev=854.05 clat (msec): min=63, max=445, avg=217.50, stdev=49.43 lat (msec): min=63, max=445, avg=217.80, stdev=49.44 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 150], 10.00th=[ 161], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 55, max= 264, per=25.04%, avg=144.75, stdev=43.35, samples=71 iops : min= 13, max= 66, avg=35.70, stdev=10.88, samples=71 lat (msec) : 20=0.04%, 100=0.16%, 250=77.19%, 500=22.62% cpu : usr=0.29%, sys=1.66%, ctx=4250, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180136: Mon May 25 06:24:02 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5060KiB/35960msec) slat (usec): min=3281, max=71600, avg=15644.49, stdev=8857.44 clat (msec): min=34, max=401, avg=211.68, stdev=51.95 lat (msec): min=42, max=454, avg=227.33, stdev=54.60 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 368], 99.90th=[ 376], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 56, max= 225, per=25.36%, avg=141.03, stdev=36.84, samples=70 iops : min= 14, max= 56, avg=34.79, stdev= 9.25, samples=70 write: IOPS=36, BW=144KiB/s (148kB/s)(5180KiB/35960msec); 0 zone resets slat (usec): min=55, max=17432, avg=289.36, stdev=769.70 clat (msec): min=61, max=443, avg=219.03, stdev=49.98 lat (msec): min=61, max=444, avg=219.32, stdev=49.99 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 71, max= 212, per=24.95%, avg=144.20, stdev=31.16, samples=70 iops : min= 17, max= 53, avg=35.57, stdev= 7.83, samples=70 lat (msec) : 50=0.08%, 100=0.20%, 250=77.27%, 500=22.46% cpu : usr=0.28%, sys=1.79%, ctx=4398, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=556KiB/s (569kB/s), 138KiB/s-141KiB/s (141kB/s-144kB/s), io=19.6MiB (20.5MB), run=35960-36085msec WRITE: bw=579KiB/s (593kB/s), 144KiB/s-146KiB/s (147kB/s-149kB/s), io=20.4MiB (21.4MB), run=35960-36085msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180145: Mon May 25 06:24:15 2026 read: IOPS=2, BW=80.3MiB/s (84.2MB/s)(40.0MiB/498msec) slat (nsec): min=32049k, max=32049k, avg=32049160.00, stdev= 0.00 clat (nsec): min=451551k, max=451551k, avg=451550860.00, stdev= 0.00 lat (nsec): min=483628k, max=483628k, avg=483627725.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.41%, ctx=10, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.3MiB/s (84.2MB/s), 80.3MiB/s-80.3MiB/s (84.2MB/s-84.2MB/s), io=40.0MiB (41.9MB), run=498-498msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout