== sanity test 398c: run fio to test AIO ================= 15:08:36 (1790881716) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.174538 s, 240 MB/s osc.lustre-OST0000-osc-ffff955f4a64b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=193652: Thu Oct 1 15:08:52 2026 write: IOPS=217, BW=870KiB/s (891kB/s)(10.0MiB/11774msec); 0 zone resets slat (usec): min=21, max=6018, avg=142.21, stdev=250.13 clat (msec): min=9, max=148, avg=73.36, stdev=14.66 lat (msec): min=11, max=148, avg=73.50, stdev=14.66 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 91], 95.00th=[ 97], | 99.00th=[ 105], 99.50th=[ 112], 99.90th=[ 142], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 741, max= 1088, per=25.14%, avg=869.43, stdev=111.42, samples=23 iops : min= 185, max= 272, avg=217.22, stdev=27.86, samples=23 lat (msec) : 10=0.04%, 50=2.93%, 100=94.26%, 250=2.77% cpu : usr=0.24%, sys=2.04%, ctx=2930, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=193653: Thu Oct 1 15:08:52 2026 write: IOPS=216, BW=865KiB/s (886kB/s)(10.0MiB/11840msec); 0 zone resets slat (usec): min=23, max=4450, avg=123.49, stdev=207.62 clat (msec): min=8, max=130, avg=73.65, stdev=14.19 lat (msec): min=8, max=130, avg=73.77, stdev=14.19 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 92], 95.00th=[ 97], | 99.00th=[ 107], 99.50th=[ 111], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 131] bw ( KiB/s): min= 720, max= 1048, per=24.97%, avg=863.78, stdev=104.86, samples=23 iops : min= 180, max= 262, avg=215.83, stdev=26.27, samples=23 lat (msec) : 10=0.04%, 50=2.54%, 100=94.34%, 250=3.09% cpu : usr=0.22%, sys=1.78%, ctx=2825, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=193654: Thu Oct 1 15:08:52 2026 write: IOPS=216, BW=867KiB/s (888kB/s)(10.0MiB/11811msec); 0 zone resets slat (usec): min=26, max=2653, avg=118.23, stdev=183.58 clat (msec): min=15, max=134, avg=73.50, stdev=14.02 lat (msec): min=15, max=134, avg=73.62, stdev=14.02 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 82], 80.00th=[ 85], 90.00th=[ 90], 95.00th=[ 97], | 99.00th=[ 107], 99.50th=[ 109], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 136] bw ( KiB/s): min= 693, max= 1042, per=25.00%, avg=864.91, stdev=105.17, samples=23 iops : min= 173, max= 260, avg=216.09, stdev=26.31, samples=23 lat (msec) : 20=0.66%, 50=2.11%, 100=94.18%, 250=3.05% cpu : usr=0.20%, sys=1.63%, ctx=2812, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=193655: Thu Oct 1 15:08:52 2026 write: IOPS=216, BW=865KiB/s (886kB/s)(10.0MiB/11840msec); 0 zone resets slat (usec): min=21, max=3781, avg=132.56, stdev=210.35 clat (msec): min=25, max=142, avg=73.60, stdev=13.45 lat (msec): min=26, max=142, avg=73.74, stdev=13.46 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 54], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 92], 95.00th=[ 96], | 99.00th=[ 104], 99.50th=[ 106], 99.90th=[ 138], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 749, max= 1128, per=25.02%, avg=865.30, stdev=107.99, samples=23 iops : min= 187, max= 282, avg=216.17, stdev=26.99, samples=23 lat (msec) : 50=1.76%, 100=96.02%, 250=2.23% cpu : usr=0.31%, sys=1.78%, ctx=2840, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3459KiB/s (3542kB/s), 865KiB/s-870KiB/s (886kB/s-891kB/s), io=40.0MiB (41.9MB), run=11774-11840msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=193667: Thu Oct 1 15:09:08 2026 read: IOPS=106, BW=428KiB/s (438kB/s)(5048KiB/11802msec) slat (usec): min=26, max=18132, avg=4630.80, stdev=2049.56 clat (msec): min=9, max=127, avg=69.93, stdev=16.05 lat (msec): min=10, max=129, avg=74.56, stdev=16.64 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 58], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 77], 80.00th=[ 84], 90.00th=[ 92], 95.00th=[ 100], | 99.00th=[ 116], 99.50th=[ 120], 99.90th=[ 126], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 327, max= 528, per=25.17%, avg=424.04, stdev=60.63, samples=23 iops : min= 81, max= 132, avg=105.70, stdev=15.30, samples=23 write: IOPS=109, BW=440KiB/s (450kB/s)(5192KiB/11802msec); 0 zone resets slat (usec): min=28, max=4098, avg=138.29, stdev=227.66 clat (msec): min=11, max=120, avg=72.54, stdev=15.55 lat (msec): min=11, max=121, avg=72.68, stdev=15.56 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 51], 10.00th=[ 54], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 112], 99.50th=[ 115], 99.90th=[ 118], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 339, max= 558, per=25.02%, avg=438.83, stdev=62.50, samples=23 iops : min= 84, max= 139, avg=109.39, stdev=15.62, samples=23 lat (msec) : 10=0.04%, 20=0.16%, 50=6.02%, 100=88.67%, 250=5.12% cpu : usr=0.32%, sys=2.73%, ctx=4798, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193668: Thu Oct 1 15:09:08 2026 read: IOPS=104, BW=418KiB/s (428kB/s)(4976KiB/11893msec) slat (usec): min=1340, max=57215, avg=4759.09, stdev=2519.07 clat (msec): min=23, max=122, avg=69.88, stdev=15.12 lat (msec): min=34, max=127, avg=74.64, stdev=15.60 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 78], 80.00th=[ 83], 90.00th=[ 91], 95.00th=[ 99], | 99.00th=[ 110], 99.50th=[ 114], 99.90th=[ 120], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 292, max= 504, per=24.61%, avg=414.65, stdev=66.43, samples=23 iops : min= 73, max= 126, avg=103.35, stdev=16.59, samples=23 write: IOPS=110, BW=443KiB/s (453kB/s)(5264KiB/11893msec); 0 zone resets slat (usec): min=24, max=1834, avg=124.22, stdev=144.21 clat (msec): min=22, max=123, avg=73.00, stdev=15.66 lat (msec): min=22, max=123, avg=73.12, stdev=15.67 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 103], | 99.00th=[ 113], 99.50th=[ 116], 99.90th=[ 122], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 335, max= 544, per=25.04%, avg=439.13, stdev=54.88, samples=23 iops : min= 83, max= 136, avg=109.48, stdev=13.81, samples=23 lat (msec) : 50=5.27%, 100=89.73%, 250=5.00% cpu : usr=0.31%, sys=2.74%, ctx=4826, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193669: Thu Oct 1 15:09:08 2026 read: IOPS=104, BW=419KiB/s (429kB/s)(4984KiB/11908msec) slat (usec): min=1409, max=63661, avg=4781.93, stdev=2658.84 clat (msec): min=2, max=124, avg=70.74, stdev=16.08 lat (msec): min=6, max=130, avg=75.53, stdev=16.55 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 85], 90.00th=[ 93], 95.00th=[ 101], | 99.00th=[ 112], 99.50th=[ 117], 99.90th=[ 123], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 308, max= 496, per=24.65%, avg=415.30, stdev=49.84, samples=23 iops : min= 77, max= 124, avg=103.52, stdev=12.41, samples=23 write: IOPS=110, BW=441KiB/s (452kB/s)(5256KiB/11908msec); 0 zone resets slat (usec): min=28, max=2316, avg=136.94, stdev=173.01 clat (msec): min=8, max=125, avg=72.38, stdev=15.72 lat (msec): min=8, max=125, avg=72.52, stdev=15.72 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 94], 95.00th=[ 103], | 99.00th=[ 114], 99.50th=[ 118], 99.90th=[ 121], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 351, max= 616, per=24.68%, avg=432.96, stdev=62.90, samples=23 iops : min= 87, max= 154, avg=107.87, stdev=15.84, samples=23 lat (msec) : 4=0.04%, 10=0.04%, 20=0.08%, 50=5.20%, 100=89.06% lat (msec) : 250=5.59% cpu : usr=0.28%, sys=2.84%, ctx=4699, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193670: Thu Oct 1 15:09:08 2026 read: IOPS=106, BW=426KiB/s (436kB/s)(5060KiB/11889msec) slat (usec): min=1475, max=16603, avg=4732.20, stdev=2015.75 clat (msec): min=27, max=127, avg=70.35, stdev=15.30 lat (msec): min=36, max=132, avg=75.09, stdev=15.87 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 82], 90.00th=[ 91], 95.00th=[ 100], | 99.00th=[ 116], 99.50th=[ 120], 99.90th=[ 127], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 335, max= 528, per=25.12%, avg=423.30, stdev=55.17, samples=23 iops : min= 83, max= 132, avg=105.48, stdev=13.85, samples=23 write: IOPS=108, BW=436KiB/s (446kB/s)(5180KiB/11889msec); 0 zone resets slat (usec): min=34, max=3288, avg=138.79, stdev=214.58 clat (msec): min=31, max=127, avg=72.55, stdev=15.65 lat (msec): min=31, max=127, avg=72.69, stdev=15.67 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 51], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 114], 99.50th=[ 118], 99.90th=[ 126], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 343, max= 520, per=24.63%, avg=432.09, stdev=53.83, samples=23 iops : min= 85, max= 130, avg=107.70, stdev=13.51, samples=23 lat (msec) : 50=5.43%, 100=89.30%, 250=5.27% cpu : usr=0.34%, sys=2.79%, ctx=4723, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1685KiB/s (1726kB/s), 418KiB/s-428KiB/s (428kB/s-438kB/s), io=19.6MiB (20.5MB), run=11802-11908msec WRITE: bw=1754KiB/s (1797kB/s), 436KiB/s-443KiB/s (446kB/s-453kB/s), io=20.4MiB (21.4MB), run=11802-11908msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=193675: Thu Oct 1 15:09:12 2026 read: IOPS=4, BW=168MiB/s (176MB/s)(40.0MiB/238msec) slat (nsec): min=17818k, max=17818k, avg=17817811.00, stdev= 0.00 clat (nsec): min=210198k, max=210198k, avg=210197840.00, stdev= 0.00 lat (nsec): min=228020k, max=228020k, avg=228019662.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.38%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=40.0MiB (41.9MB), run=238-238msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout