== sanity test 398c: run fio to test AIO ================= 20:23:33 (1788567813) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.765093 s, 54.8 MB/s osc.lustre-OST0000-osc-ffff9a6584ef6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179665: Fri Sep 4 20:24:23 2026 write: IOPS=72, BW=290KiB/s (297kB/s)(10.0MiB/35254msec); 0 zone resets slat (usec): min=24, max=19085, avg=191.50, stdev=928.72 clat (msec): min=41, max=430, avg=219.07, stdev=57.53 lat (msec): min=42, max=430, avg=219.27, stdev=57.56 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 131], 10.00th=[ 140], 20.00th=[ 171], | 30.00th=[ 188], 40.00th=[ 203], 50.00th=[ 215], 60.00th=[ 230], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 321], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 167, max= 510, per=25.03%, avg=290.61, stdev=67.89, samples=69 iops : min= 41, max= 127, avg=72.22, stdev=17.00, samples=69 lat (msec) : 50=0.08%, 250=70.70%, 500=29.22% cpu : usr=0.19%, sys=0.81%, ctx=2259, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179666: Fri Sep 4 20:24:23 2026 write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35238msec); 0 zone resets slat (usec): min=21, max=19700, avg=167.68, stdev=824.29 clat (msec): min=36, max=526, avg=219.11, stdev=57.53 lat (msec): min=36, max=526, avg=219.28, stdev=57.51 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 134], 10.00th=[ 142], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 317], | 99.00th=[ 347], 99.50th=[ 351], 99.90th=[ 460], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 135, max= 510, per=25.06%, avg=290.90, stdev=73.69, samples=69 iops : min= 33, max= 127, avg=72.30, stdev=18.48, samples=69 lat (msec) : 50=0.04%, 100=0.12%, 250=69.26%, 500=30.51%, 750=0.08% cpu : usr=0.15%, sys=0.58%, ctx=1228, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179667: Fri Sep 4 20:24:23 2026 write: IOPS=73, BW=293KiB/s (300kB/s)(10.0MiB/34917msec); 0 zone resets slat (usec): min=29, max=22851, avg=195.07, stdev=840.39 clat (msec): min=19, max=586, avg=217.84, stdev=62.56 lat (msec): min=19, max=586, avg=218.03, stdev=62.54 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 230], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 368], 99.50th=[ 498], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 150, max= 542, per=25.17%, avg=292.23, stdev=73.18, samples=69 iops : min= 37, max= 135, avg=72.62, stdev=18.35, samples=69 lat (msec) : 20=0.04%, 50=0.12%, 100=0.86%, 250=70.00%, 500=28.79% lat (msec) : 750=0.20% cpu : usr=0.19%, sys=0.86%, ctx=2624, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179668: Fri Sep 4 20:24:23 2026 write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35234msec); 0 zone resets slat (usec): min=21, max=35264, avg=158.60, stdev=928.00 clat (msec): min=32, max=378, avg=218.96, stdev=57.72 lat (msec): min=32, max=379, avg=219.12, stdev=57.75 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 169], | 30.00th=[ 190], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 127, max= 510, per=24.92%, avg=289.30, stdev=83.10, samples=69 iops : min= 31, max= 127, avg=71.91, stdev=20.85, samples=69 lat (msec) : 50=0.08%, 100=0.35%, 250=70.78%, 500=28.79% cpu : usr=0.11%, sys=0.63%, ctx=1388, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1162KiB/s (1190kB/s), 290KiB/s-293KiB/s (297kB/s-300kB/s), io=40.0MiB (41.9MB), run=34917-35254msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179684: Fri Sep 4 20:25:16 2026 read: IOPS=31, BW=125KiB/s (128kB/s)(5048KiB/40292msec) slat (usec): min=3612, max=85684, avg=18226.33, stdev=10364.64 clat (msec): min=25, max=521, avg=238.84, stdev=62.60 lat (msec): min=49, max=556, avg=257.07, stdev=65.77 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 165], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 224], 60.00th=[ 239], | 70.00th=[ 259], 80.00th=[ 284], 90.00th=[ 330], 95.00th=[ 368], | 99.00th=[ 426], 99.50th=[ 447], 99.90th=[ 510], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 21, max= 190, per=25.11%, avg=125.04, stdev=31.29, samples=79 iops : min= 5, max= 47, avg=30.73, stdev= 7.88, samples=79 write: IOPS=32, BW=129KiB/s (132kB/s)(5192KiB/40292msec); 0 zone resets slat (usec): min=31, max=53768, avg=238.38, stdev=1639.66 clat (msec): min=72, max=530, avg=240.82, stdev=55.94 lat (msec): min=73, max=530, avg=241.06, stdev=56.07 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 174], 10.00th=[ 184], 20.00th=[ 197], | 30.00th=[ 207], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 257], 80.00th=[ 284], 90.00th=[ 326], 95.00th=[ 355], | 99.00th=[ 393], 99.50th=[ 460], 99.90th=[ 523], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 26, max= 215, per=24.85%, avg=128.71, stdev=40.47, samples=79 iops : min= 6, max= 53, avg=31.65, stdev=10.15, samples=79 lat (msec) : 50=0.08%, 100=0.23%, 250=65.66%, 500=33.83%, 750=0.20% cpu : usr=0.21%, sys=1.41%, ctx=3707, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179685: Fri Sep 4 20:25:16 2026 read: IOPS=30, BW=124KiB/s (127kB/s)(4976KiB/40247msec) slat (usec): min=3335, max=89737, avg=17967.33, stdev=10355.79 clat (msec): min=24, max=573, avg=237.93, stdev=66.62 lat (msec): min=32, max=601, avg=255.91, stdev=69.59 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 188], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 222], 60.00th=[ 234], | 70.00th=[ 253], 80.00th=[ 284], 90.00th=[ 330], 95.00th=[ 376], | 99.00th=[ 464], 99.50th=[ 493], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 16, max= 216, per=24.87%, avg=123.85, stdev=38.59, samples=79 iops : min= 4, max= 54, avg=30.44, stdev= 9.71, samples=79 write: IOPS=32, BW=131KiB/s (134kB/s)(5264KiB/40247msec); 0 zone resets slat (usec): min=23, max=23362, avg=185.07, stdev=785.81 clat (msec): min=93, max=573, avg=242.15, stdev=60.72 lat (msec): min=93, max=573, avg=242.34, stdev=60.70 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 174], 10.00th=[ 184], 20.00th=[ 197], | 30.00th=[ 207], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 284], 90.00th=[ 326], 95.00th=[ 363], | 99.00th=[ 439], 99.50th=[ 498], 99.90th=[ 542], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 55, max= 247, per=25.35%, avg=131.32, stdev=38.63, samples=79 iops : min= 13, max= 61, avg=32.32, stdev= 9.69, samples=79 lat (msec) : 50=0.08%, 100=0.08%, 250=67.34%, 500=32.07%, 750=0.43% cpu : usr=0.22%, sys=1.34%, ctx=3370, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179686: Fri Sep 4 20:25:16 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(4984KiB/39693msec) slat (usec): min=56, max=86208, avg=17471.98, stdev=10186.43 clat (msec): min=28, max=503, avg=235.41, stdev=60.14 lat (msec): min=56, max=534, avg=252.90, stdev=62.88 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 161], 10.00th=[ 174], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 224], 60.00th=[ 236], | 70.00th=[ 253], 80.00th=[ 279], 90.00th=[ 313], 95.00th=[ 359], | 99.00th=[ 409], 99.50th=[ 439], 99.90th=[ 489], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 47, max= 200, per=25.27%, avg=125.83, stdev=35.10, samples=78 iops : min= 11, max= 50, avg=30.92, stdev= 8.87, samples=78 write: IOPS=33, BW=132KiB/s (136kB/s)(5256KiB/39693msec); 0 zone resets slat (usec): min=32, max=29096, avg=207.07, stdev=978.62 clat (msec): min=111, max=609, avg=242.44, stdev=61.02 lat (msec): min=111, max=609, avg=242.65, stdev=61.05 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 174], 10.00th=[ 184], 20.00th=[ 199], | 30.00th=[ 207], 40.00th=[ 218], 50.00th=[ 226], 60.00th=[ 241], | 70.00th=[ 255], 80.00th=[ 284], 90.00th=[ 330], 95.00th=[ 359], | 99.00th=[ 435], 99.50th=[ 481], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 55, max= 223, per=25.46%, avg=131.88, stdev=37.54, samples=78 iops : min= 13, max= 55, avg=32.44, stdev= 9.40, samples=78 lat (msec) : 50=0.08%, 100=0.08%, 250=67.30%, 500=32.27%, 750=0.27% cpu : usr=0.24%, sys=1.30%, ctx=3766, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179687: Fri Sep 4 20:25:16 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(5060KiB/40242msec) slat (usec): min=3790, max=84674, avg=18143.96, stdev=10115.50 clat (msec): min=26, max=494, avg=236.93, stdev=61.69 lat (msec): min=54, max=533, avg=255.08, stdev=64.94 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 163], 10.00th=[ 174], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 222], 60.00th=[ 234], | 70.00th=[ 253], 80.00th=[ 279], 90.00th=[ 326], 95.00th=[ 363], | 99.00th=[ 422], 99.50th=[ 451], 99.90th=[ 485], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 37, max= 184, per=25.32%, avg=126.11, stdev=33.74, samples=79 iops : min= 9, max= 46, avg=30.99, stdev= 8.46, samples=79 write: IOPS=32, BW=129KiB/s (132kB/s)(5180KiB/40242msec); 0 zone resets slat (usec): min=30, max=30025, avg=269.57, stdev=1447.82 clat (msec): min=121, max=603, avg=243.37, stdev=58.36 lat (msec): min=122, max=603, avg=243.64, stdev=58.40 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 176], 10.00th=[ 186], 20.00th=[ 199], | 30.00th=[ 207], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 259], 80.00th=[ 288], 90.00th=[ 326], 95.00th=[ 355], | 99.00th=[ 430], 99.50th=[ 481], 99.90th=[ 542], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 55, max= 236, per=24.91%, avg=129.04, stdev=34.80, samples=79 iops : min= 13, max= 59, avg=31.72, stdev= 8.79, samples=79 lat (msec) : 50=0.04%, 100=0.04%, 250=66.88%, 500=32.89%, 750=0.16% cpu : usr=0.30%, sys=1.53%, ctx=3761, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=498KiB/s (510kB/s), 124KiB/s-126KiB/s (127kB/s-129kB/s), io=19.6MiB (20.5MB), run=39693-40292msec WRITE: bw=519KiB/s (531kB/s), 129KiB/s-132KiB/s (132kB/s-136kB/s), io=20.4MiB (21.4MB), run=39693-40292msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179704: Fri Sep 4 20:25:37 2026 read: IOPS=1, BW=65.1MiB/s (68.3MB/s)(40.0MiB/614msec) slat (nsec): min=44029k, max=44029k, avg=44028732.00, stdev= 0.00 clat (nsec): min=534597k, max=534597k, avg=534596561.00, stdev= 0.00 lat (nsec): min=578629k, max=578629k, avg=578629010.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 535], 10.00th=[ 535], 20.00th=[ 535], | 30.00th=[ 535], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.96%, ctx=9, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.1MiB/s (68.3MB/s), 65.1MiB/s-65.1MiB/s (68.3MB/s-68.3MB/s), io=40.0MiB (41.9MB), run=614-614msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout