== sanity test 398c: run fio to test AIO ================= 12:09:40 (1790957380) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.240951 s, 174 MB/s osc.lustre-OST0000-osc-ffff9758f9062800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145442: Fri Oct 2 12:10:57 2026 write: IOPS=35, BW=141KiB/s (144kB/s)(10.0MiB/72584msec); 0 zone resets slat (usec): min=23, max=6538, avg=176.75, stdev=295.62 clat (msec): min=125, max=894, avg=453.22, stdev=101.09 lat (msec): min=126, max=895, avg=453.40, stdev=101.09 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 300], 10.00th=[ 347], 20.00th=[ 384], | 30.00th=[ 405], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 472], | 70.00th=[ 498], 80.00th=[ 527], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 709], 99.50th=[ 785], 99.90th=[ 877], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 47, max= 316, per=24.89%, avg=140.36, stdev=36.30, samples=144 iops : min= 11, max= 79, avg=34.69, stdev= 9.13, samples=144 lat (msec) : 250=2.62%, 500=67.97%, 750=28.55%, 1000=0.86% cpu : usr=0.07%, sys=0.45%, ctx=2460, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145443: Fri Oct 2 12:10:57 2026 write: IOPS=35, BW=141KiB/s (145kB/s)(10.0MiB/72561msec); 0 zone resets slat (usec): min=23, max=4278, avg=189.29, stdev=321.80 clat (msec): min=68, max=838, avg=452.90, stdev=99.47 lat (msec): min=68, max=838, avg=453.09, stdev=99.46 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 384], | 30.00th=[ 405], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 472], | 70.00th=[ 498], 80.00th=[ 527], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 693], 99.50th=[ 743], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 835] bw ( KiB/s): min= 39, max= 316, per=24.89%, avg=140.36, stdev=33.54, samples=144 iops : min= 9, max= 79, avg=34.69, stdev= 8.47, samples=144 lat (msec) : 100=0.04%, 250=2.46%, 500=68.36%, 750=28.75%, 1000=0.39% cpu : usr=0.07%, sys=0.46%, ctx=2558, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145444: Fri Oct 2 12:10:57 2026 write: IOPS=35, BW=141KiB/s (145kB/s)(10.0MiB/72560msec); 0 zone resets slat (usec): min=24, max=3721, avg=175.39, stdev=294.97 clat (msec): min=111, max=957, avg=452.78, stdev=100.27 lat (msec): min=112, max=958, avg=452.96, stdev=100.27 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 380], | 30.00th=[ 405], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 472], | 70.00th=[ 498], 80.00th=[ 531], 90.00th=[ 575], 95.00th=[ 625], | 99.00th=[ 709], 99.50th=[ 735], 99.90th=[ 802], 99.95th=[ 844], | 99.99th=[ 961] bw ( KiB/s): min= 40, max= 316, per=24.89%, avg=140.37, stdev=34.04, samples=144 iops : min= 10, max= 79, avg=34.69, stdev= 8.56, samples=144 lat (msec) : 250=2.27%, 500=69.18%, 750=28.32%, 1000=0.23% cpu : usr=0.07%, sys=0.43%, ctx=2517, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145445: Fri Oct 2 12:10:57 2026 write: IOPS=35, BW=141KiB/s (144kB/s)(10.0MiB/72610msec); 0 zone resets slat (usec): min=22, max=5392, avg=177.21, stdev=277.00 clat (msec): min=59, max=880, avg=453.18, stdev=97.74 lat (msec): min=60, max=880, avg=453.36, stdev=97.73 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 384], | 30.00th=[ 405], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 472], | 70.00th=[ 498], 80.00th=[ 527], 90.00th=[ 575], 95.00th=[ 617], | 99.00th=[ 709], 99.50th=[ 743], 99.90th=[ 844], 99.95th=[ 852], | 99.99th=[ 877] bw ( KiB/s): min= 31, max= 264, per=24.83%, avg=140.06, stdev=30.51, samples=144 iops : min= 7, max= 66, avg=34.62, stdev= 7.71, samples=144 lat (msec) : 100=0.04%, 250=2.34%, 500=67.81%, 750=29.38%, 1000=0.43% cpu : usr=0.08%, sys=0.44%, ctx=2571, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=564KiB/s (578kB/s), 141KiB/s-141KiB/s (144kB/s-145kB/s), io=40.0MiB (41.9MB), run=72560-72610msec 5% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145466: Fri Oct 2 12:11:55 2026 read: IOPS=23, BW=94.4KiB/s (96.7kB/s)(5048KiB/53448msec) slat (usec): min=2054, max=23521, avg=6537.69, stdev=2611.86 clat (msec): min=13, max=389, avg=209.33, stdev=50.82 lat (msec): min=19, max=398, avg=215.88, stdev=51.35 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 234], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8, max= 199, per=24.94%, avg=93.52, stdev=32.89, samples=106 iops : min= 2, max= 49, avg=23.06, stdev= 8.23, samples=106 write: IOPS=24, BW=97.1KiB/s (99.5kB/s)(5192KiB/53448msec); 0 zone resets slat (usec): min=26, max=13090, avg=226.09, stdev=487.40 clat (msec): min=108, max=796, avg=447.54, stdev=126.56 lat (msec): min=108, max=796, avg=447.77, stdev=126.58 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 368], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 684], | 99.00th=[ 751], 99.50th=[ 768], 99.90th=[ 785], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 15, max= 160, per=24.75%, avg=96.53, stdev=28.75, samples=106 iops : min= 3, max= 40, avg=23.81, stdev= 7.26, samples=106 lat (msec) : 20=0.12%, 50=0.23%, 100=0.20%, 250=41.37%, 500=41.95% lat (msec) : 750=15.55%, 1000=0.59% cpu : usr=0.13%, sys=1.06%, ctx=5233, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145467: Fri Oct 2 12:11:55 2026 read: IOPS=23, BW=93.3KiB/s (95.5kB/s)(4976KiB/53341msec) slat (usec): min=1912, max=37425, avg=6421.86, stdev=2758.58 clat (msec): min=5, max=417, avg=206.90, stdev=51.53 lat (msec): min=12, max=422, avg=213.33, stdev=52.25 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 351], 99.90th=[ 401], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 15, max= 175, per=24.67%, avg=92.53, stdev=34.95, samples=106 iops : min= 3, max= 43, avg=22.81, stdev= 8.80, samples=106 write: IOPS=24, BW=98.7KiB/s (101kB/s)(5264KiB/53341msec); 0 zone resets slat (usec): min=26, max=4944, avg=220.29, stdev=325.04 clat (msec): min=95, max=789, avg=445.74, stdev=130.79 lat (msec): min=95, max=789, avg=445.97, stdev=130.82 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 247], 10.00th=[ 279], 20.00th=[ 326], | 30.00th=[ 372], 40.00th=[ 414], 50.00th=[ 451], 60.00th=[ 477], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 760], 99.50th=[ 776], 99.90th=[ 785], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 48, max= 184, per=25.17%, avg=98.16, stdev=27.12, samples=106 iops : min= 12, max= 46, avg=24.22, stdev= 6.81, samples=106 lat (msec) : 10=0.08%, 50=0.16%, 100=0.35%, 250=42.15%, 500=40.27% lat (msec) : 750=16.33%, 1000=0.66% cpu : usr=0.14%, sys=1.01%, ctx=5292, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145468: Fri Oct 2 12:11:55 2026 read: IOPS=23, BW=93.7KiB/s (95.9kB/s)(4984KiB/53194msec) slat (usec): min=36, max=20358, avg=6372.61, stdev=2718.42 clat (msec): min=16, max=401, avg=206.51, stdev=49.20 lat (msec): min=21, max=407, avg=212.89, stdev=49.92 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 23, max= 200, per=25.04%, avg=93.89, stdev=35.40, samples=105 iops : min= 5, max= 50, avg=23.15, stdev= 8.84, samples=105 write: IOPS=24, BW=98.8KiB/s (101kB/s)(5256KiB/53194msec); 0 zone resets slat (usec): min=31, max=4466, avg=225.48, stdev=339.03 clat (msec): min=61, max=795, avg=445.27, stdev=129.10 lat (msec): min=61, max=796, avg=445.49, stdev=129.12 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 251], 10.00th=[ 284], 20.00th=[ 330], | 30.00th=[ 372], 40.00th=[ 405], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 751], 99.50th=[ 776], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 32, max= 176, per=25.07%, avg=97.78, stdev=29.73, samples=105 iops : min= 8, max= 44, avg=24.12, stdev= 7.51, samples=105 lat (msec) : 20=0.04%, 50=0.23%, 100=0.62%, 250=41.95%, 500=40.00% lat (msec) : 750=16.52%, 1000=0.62% cpu : usr=0.13%, sys=1.07%, ctx=5134, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145469: Fri Oct 2 12:11:55 2026 read: IOPS=23, BW=95.1KiB/s (97.4kB/s)(5060KiB/53185msec) slat (usec): min=1932, max=24845, avg=6422.01, stdev=2659.27 clat (msec): min=13, max=623, avg=209.29, stdev=52.08 lat (msec): min=23, max=638, avg=215.72, stdev=52.77 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 140], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 220], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 338], 99.50th=[ 384], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8, max= 200, per=25.43%, avg=95.36, stdev=30.96, samples=105 iops : min= 2, max= 50, avg=23.51, stdev= 7.78, samples=105 write: IOPS=24, BW=97.4KiB/s (99.7kB/s)(5180KiB/53185msec); 0 zone resets slat (usec): min=31, max=5741, avg=229.25, stdev=363.94 clat (msec): min=72, max=1057, avg=445.06, stdev=129.19 lat (msec): min=73, max=1057, avg=445.29, stdev=129.23 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 372], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 468], | 70.00th=[ 506], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 760], 99.50th=[ 885], 99.90th=[ 1045], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 39, max= 184, per=24.90%, avg=97.10, stdev=28.25, samples=105 iops : min= 9, max= 46, avg=23.95, stdev= 7.13, samples=105 lat (msec) : 20=0.04%, 50=0.23%, 100=0.39%, 250=41.68%, 500=41.37% lat (msec) : 750=15.62%, 1000=0.47%, 2000=0.20% cpu : usr=0.13%, sys=1.02%, ctx=5169, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=375KiB/s (384kB/s), 93.3KiB/s-95.1KiB/s (95.5kB/s-97.4kB/s), io=19.6MiB (20.5MB), run=53185-53448msec WRITE: bw=391KiB/s (400kB/s), 97.1KiB/s-98.8KiB/s (99.5kB/s-101kB/s), io=20.4MiB (21.4MB), run=53185-53448msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145480: Fri Oct 2 12:11:59 2026 read: IOPS=5, BW=217MiB/s (228MB/s)(40.0MiB/184msec) slat (nsec): min=13987k, max=13987k, avg=13986586.00, stdev= 0.00 clat (nsec): min=161213k, max=161213k, avg=161213240.00, stdev= 0.00 lat (nsec): min=175204k, max=175204k, avg=175204421.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.01%, ctx=27, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=40.0MiB (41.9MB), run=184-184msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout