== sanity test 398c: run fio to test AIO ================= 10:59:26 (1788361166) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.245125 s, 171 MB/s osc.lustre-OST0000-osc-ffff9af30571b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=158931: Wed Sep 2 11:00:48 2026 write: IOPS=33, BW=133KiB/s (136kB/s)(10.0MiB/76821msec); 0 zone resets slat (usec): min=36, max=7834, avg=235.83, stdev=474.51 clat (msec): min=127, max=1026, avg=479.42, stdev=118.42 lat (msec): min=127, max=1026, avg=479.66, stdev=118.43 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 305], 10.00th=[ 355], 20.00th=[ 397], | 30.00th=[ 422], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 531], 80.00th=[ 567], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 802], 99.50th=[ 818], 99.90th=[ 927], 99.95th=[ 961], | 99.99th=[ 1028] bw ( KiB/s): min= 40, max= 325, per=24.91%, avg=132.80, stdev=37.62, samples=152 iops : min= 10, max= 81, avg=32.89, stdev= 9.46, samples=152 lat (msec) : 250=3.20%, 500=55.86%, 750=38.59%, 1000=2.30%, 2000=0.04% cpu : usr=0.07%, sys=0.49%, ctx=3095, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158932: Wed Sep 2 11:00:48 2026 write: IOPS=33, BW=134KiB/s (137kB/s)(10.0MiB/76562msec); 0 zone resets slat (usec): min=38, max=5795, avg=229.51, stdev=385.31 clat (msec): min=86, max=1213, avg=478.04, stdev=126.85 lat (msec): min=86, max=1213, avg=478.27, stdev=126.87 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 296], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 422], 40.00th=[ 443], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 531], 80.00th=[ 575], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 818], 99.50th=[ 911], 99.90th=[ 1116], 99.95th=[ 1150], | 99.99th=[ 1217] bw ( KiB/s): min= 16, max= 414, per=24.92%, avg=132.80, stdev=42.38, samples=152 iops : min= 4, max= 103, avg=32.89, stdev=10.64, samples=152 lat (msec) : 100=0.62%, 250=2.58%, 500=57.66%, 750=37.15%, 1000=1.68% lat (msec) : 2000=0.31% cpu : usr=0.07%, sys=0.50%, ctx=3261, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158933: Wed Sep 2 11:00:48 2026 write: IOPS=33, BW=134KiB/s (137kB/s)(10.0MiB/76655msec); 0 zone resets slat (usec): min=34, max=15429, avg=248.13, stdev=547.53 clat (msec): min=112, max=1140, avg=478.17, stdev=124.30 lat (msec): min=112, max=1141, avg=478.42, stdev=124.31 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 292], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 418], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 502], | 70.00th=[ 531], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 802], 99.50th=[ 902], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1150] bw ( KiB/s): min= 31, max= 414, per=24.91%, avg=132.79, stdev=40.93, samples=152 iops : min= 7, max= 103, avg=32.89, stdev=10.26, samples=152 lat (msec) : 250=3.12%, 500=56.17%, 750=38.67%, 1000=1.80%, 2000=0.23% cpu : usr=0.07%, sys=0.49%, ctx=2922, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158934: Wed Sep 2 11:00:48 2026 write: IOPS=33, BW=133KiB/s (137kB/s)(10.0MiB/76734msec); 0 zone resets slat (usec): min=38, max=10681, avg=245.79, stdev=530.81 clat (msec): min=94, max=1061, avg=478.84, stdev=122.54 lat (msec): min=94, max=1061, avg=479.09, stdev=122.54 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 300], 10.00th=[ 351], 20.00th=[ 393], | 30.00th=[ 422], 40.00th=[ 447], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 802], 99.50th=[ 885], 99.90th=[ 1020], 99.95th=[ 1028], | 99.99th=[ 1062] bw ( KiB/s): min= 55, max= 363, per=24.93%, avg=132.88, stdev=35.74, samples=152 iops : min= 13, max= 90, avg=32.91, stdev= 8.96, samples=152 lat (msec) : 100=0.12%, 250=3.32%, 500=56.13%, 750=38.55%, 1000=1.76% lat (msec) : 2000=0.12% cpu : usr=0.06%, sys=0.56%, ctx=3086, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=533KiB/s (546kB/s), 133KiB/s-134KiB/s (136kB/s-137kB/s), io=40.0MiB (41.9MB), run=76562-76821msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=158954: Wed Sep 2 11:01:49 2026 read: IOPS=22, BW=89.8KiB/s (92.0kB/s)(5048KiB/56186msec) slat (usec): min=41, max=26504, avg=7625.92, stdev=3805.99 clat (msec): min=11, max=699, avg=222.58, stdev=67.88 lat (msec): min=11, max=709, avg=230.21, stdev=68.70 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 176], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 266], 90.00th=[ 300], 95.00th=[ 334], | 99.00th=[ 451], 99.50th=[ 550], 99.90th=[ 659], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 24, max= 168, per=25.27%, avg=89.69, stdev=31.59, samples=111 iops : min= 6, max= 42, avg=22.03, stdev= 7.99, samples=111 write: IOPS=23, BW=92.4KiB/s (94.6kB/s)(5192KiB/56186msec); 0 zone resets slat (usec): min=38, max=13345, avg=276.94, stdev=547.67 clat (msec): min=71, max=955, avg=467.80, stdev=139.00 lat (msec): min=72, max=955, avg=468.08, stdev=139.02 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 642], 95.00th=[ 726], | 99.00th=[ 869], 99.50th=[ 885], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 23, max= 160, per=24.88%, avg=92.05, stdev=28.08, samples=111 iops : min= 5, max= 40, avg=22.63, stdev= 7.04, samples=111 lat (msec) : 20=0.12%, 50=0.23%, 100=0.08%, 250=37.97%, 500=43.09% lat (msec) : 750=16.37%, 1000=2.15% cpu : usr=0.13%, sys=1.03%, ctx=5331, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158955: Wed Sep 2 11:01:49 2026 read: IOPS=22, BW=88.2KiB/s (90.3kB/s)(4976KiB/56427msec) slat (usec): min=1862, max=24706, avg=7661.72, stdev=3721.38 clat (msec): min=11, max=554, avg=219.65, stdev=60.99 lat (msec): min=15, max=567, avg=227.32, stdev=61.82 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 264], 90.00th=[ 300], 95.00th=[ 326], | 99.00th=[ 414], 99.50th=[ 468], 99.90th=[ 542], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 31, max= 224, per=24.64%, avg=87.48, stdev=33.21, samples=112 iops : min= 7, max= 56, avg=21.47, stdev= 8.32, samples=112 write: IOPS=23, BW=93.3KiB/s (95.5kB/s)(5264KiB/56427msec); 0 zone resets slat (usec): min=47, max=5085, avg=277.27, stdev=465.71 clat (msec): min=136, max=929, avg=469.72, stdev=135.10 lat (msec): min=136, max=929, avg=470.00, stdev=135.10 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 268], 10.00th=[ 309], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 651], 95.00th=[ 735], | 99.00th=[ 869], 99.50th=[ 894], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 23, max= 159, per=25.07%, avg=92.74, stdev=27.76, samples=112 iops : min= 5, max= 39, avg=22.79, stdev= 7.02, samples=112 lat (msec) : 20=0.12%, 50=0.12%, 100=0.08%, 250=37.42%, 500=44.02% lat (msec) : 750=16.05%, 1000=2.19% cpu : usr=0.15%, sys=1.06%, ctx=5217, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158956: Wed Sep 2 11:01:49 2026 read: IOPS=22, BW=88.3KiB/s (90.4kB/s)(4984KiB/56429msec) slat (usec): min=2137, max=35147, avg=7944.34, stdev=3927.78 clat (msec): min=7, max=560, avg=218.81, stdev=61.36 lat (msec): min=11, max=574, avg=226.76, stdev=62.36 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 422], 99.50th=[ 468], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 23, max= 174, per=24.76%, avg=87.88, stdev=34.47, samples=112 iops : min= 5, max= 43, avg=21.58, stdev= 8.66, samples=112 write: IOPS=23, BW=93.1KiB/s (95.4kB/s)(5256KiB/56429msec); 0 zone resets slat (usec): min=40, max=11302, avg=263.87, stdev=523.43 clat (msec): min=112, max=976, avg=470.65, stdev=139.88 lat (msec): min=112, max=976, avg=470.92, stdev=139.87 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 264], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 659], 95.00th=[ 743], | 99.00th=[ 869], 99.50th=[ 902], 99.90th=[ 969], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 31, max= 176, per=24.92%, avg=92.19, stdev=28.58, samples=112 iops : min= 7, max= 44, avg=22.65, stdev= 7.21, samples=112 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.04%, 250=38.05% lat (msec) : 500=42.77%, 750=16.56%, 1000=2.30% cpu : usr=0.13%, sys=1.03%, ctx=5182, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158957: Wed Sep 2 11:01:49 2026 read: IOPS=22, BW=89.9KiB/s (92.0kB/s)(5060KiB/56290msec) slat (usec): min=2332, max=32963, avg=7515.97, stdev=3681.00 clat (msec): min=3, max=543, avg=218.11, stdev=62.06 lat (msec): min=10, max=558, avg=225.63, stdev=62.99 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 138], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 264], 90.00th=[ 296], 95.00th=[ 330], | 99.00th=[ 409], 99.50th=[ 468], 99.90th=[ 477], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 16, max= 184, per=25.30%, avg=89.83, stdev=32.67, samples=111 iops : min= 4, max= 46, avg=22.06, stdev= 8.19, samples=111 write: IOPS=23, BW=92.0KiB/s (94.2kB/s)(5180KiB/56290msec); 0 zone resets slat (usec): min=38, max=17520, avg=304.84, stdev=781.61 clat (msec): min=193, max=958, avg=473.47, stdev=136.40 lat (msec): min=193, max=959, avg=473.78, stdev=136.44 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 751], | 99.00th=[ 877], 99.50th=[ 894], 99.90th=[ 953], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 31, max= 160, per=24.70%, avg=91.40, stdev=25.89, samples=111 iops : min= 7, max= 40, avg=22.46, stdev= 6.53, samples=111 lat (msec) : 4=0.04%, 20=0.04%, 50=0.47%, 250=37.66%, 500=43.36% lat (msec) : 750=15.74%, 1000=2.70% cpu : usr=0.13%, sys=1.04%, ctx=5267, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=356KiB/s (364kB/s), 88.2KiB/s-89.9KiB/s (90.3kB/s-92.0kB/s), io=19.6MiB (20.5MB), run=56186-56429msec WRITE: bw=370KiB/s (379kB/s), 92.0KiB/s-93.3KiB/s (94.2kB/s-95.5kB/s), io=20.4MiB (21.4MB), run=56186-56429msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=158982: Wed Sep 2 11:01:55 2026 read: IOPS=3, BW=129MiB/s (135MB/s)(40.0MiB/310msec) slat (nsec): min=17041k, max=17041k, avg=17040507.00, stdev= 0.00 clat (nsec): min=278726k, max=278726k, avg=278725771.00, stdev= 0.00 lat (nsec): min=295771k, max=295771k, avg=295771330.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.53%, ctx=3, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=40.0MiB (41.9MB), run=310-310msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout