== sanity test 398c: run fio to test AIO ================= 16:59:20 (1783976360) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.469813 s, 89.3 MB/s osc.lustre-OST0000-osc-ffff8acc986eb000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179722: Mon Jul 13 16:59:44 2026 write: IOPS=173, BW=693KiB/s (710kB/s)(10.0MiB/14771msec); 0 zone resets slat (usec): min=36, max=6848, avg=224.74, stdev=386.94 clat (msec): min=7, max=175, avg=91.44, stdev=21.39 lat (msec): min=7, max=175, avg=91.67, stdev=21.39 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 107], 90.00th=[ 121], 95.00th=[ 131], | 99.00th=[ 146], 99.50th=[ 165], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 477, max= 934, per=25.08%, avg=693.69, stdev=116.77, samples=29 iops : min= 119, max= 233, avg=173.24, stdev=29.24, samples=29 lat (msec) : 10=0.04%, 20=0.08%, 50=1.17%, 100=68.28%, 250=30.43% cpu : usr=0.23%, sys=2.14%, ctx=2788, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179723: Mon Jul 13 16:59:44 2026 write: IOPS=172, BW=692KiB/s (708kB/s)(10.0MiB/14804msec); 0 zone resets slat (usec): min=39, max=7272, avg=246.79, stdev=450.56 clat (msec): min=10, max=178, avg=91.47, stdev=21.17 lat (msec): min=10, max=179, avg=91.72, stdev=21.17 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 106], 90.00th=[ 118], 95.00th=[ 130], | 99.00th=[ 153], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 425, max= 928, per=25.04%, avg=692.69, stdev=121.89, samples=29 iops : min= 106, max= 232, avg=173.00, stdev=30.51, samples=29 lat (msec) : 20=0.08%, 50=0.23%, 100=68.75%, 250=30.94% cpu : usr=0.30%, sys=2.18%, ctx=2848, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179724: Mon Jul 13 16:59:44 2026 write: IOPS=174, BW=697KiB/s (714kB/s)(10.0MiB/14696msec); 0 zone resets slat (usec): min=36, max=8101, avg=252.43, stdev=458.82 clat (msec): min=9, max=180, avg=91.46, stdev=21.10 lat (msec): min=9, max=180, avg=91.71, stdev=21.09 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 107], 90.00th=[ 121], 95.00th=[ 131], | 99.00th=[ 153], 99.50th=[ 157], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 534, max= 928, per=25.21%, avg=697.17, stdev=109.30, samples=29 iops : min= 133, max= 232, avg=174.10, stdev=27.39, samples=29 lat (msec) : 10=0.04%, 20=0.12%, 50=0.55%, 100=69.92%, 250=29.38% cpu : usr=0.27%, sys=2.29%, ctx=2927, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179725: Mon Jul 13 16:59:44 2026 write: IOPS=172, BW=692KiB/s (708kB/s)(10.0MiB/14801msec); 0 zone resets slat (usec): min=36, max=5494, avg=216.15, stdev=373.48 clat (msec): min=16, max=165, avg=91.56, stdev=21.20 lat (msec): min=16, max=165, avg=91.78, stdev=21.20 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 108], 90.00th=[ 120], 95.00th=[ 132], | 99.00th=[ 148], 99.50th=[ 163], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 425, max= 974, per=24.98%, avg=691.03, stdev=125.63, samples=29 iops : min= 106, max= 243, avg=172.59, stdev=31.45, samples=29 lat (msec) : 20=0.08%, 50=0.70%, 100=68.40%, 250=30.82% cpu : usr=0.28%, sys=1.95%, ctx=2922, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2767KiB/s (2833kB/s), 692KiB/s-697KiB/s (708kB/s-714kB/s), io=40.0MiB (41.9MB), run=14696-14804msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179737: Mon Jul 13 17:00:04 2026 read: IOPS=81, BW=326KiB/s (334kB/s)(5048KiB/15489msec) slat (usec): min=1533, max=32175, avg=6603.95, stdev=3478.02 clat (msec): min=42, max=191, avg=91.16, stdev=22.38 lat (msec): min=45, max=197, avg=97.77, stdev=23.37 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 58], 10.00th=[ 66], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 108], 90.00th=[ 121], 95.00th=[ 131], | 99.00th=[ 159], 99.50th=[ 176], 99.90th=[ 188], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 180, max= 496, per=25.07%, avg=323.37, stdev=64.43, samples=30 iops : min= 45, max= 124, avg=80.50, stdev=16.15, samples=30 write: IOPS=83, BW=335KiB/s (343kB/s)(5192KiB/15489msec); 0 zone resets slat (usec): min=38, max=4223, avg=223.93, stdev=355.38 clat (msec): min=38, max=172, avg=93.88, stdev=21.16 lat (msec): min=38, max=172, avg=94.10, stdev=21.14 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 61], 10.00th=[ 70], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 136], | 99.00th=[ 153], 99.50th=[ 157], 99.90th=[ 169], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 215, max= 504, per=24.68%, avg=331.47, stdev=63.62, samples=30 iops : min= 53, max= 126, avg=82.57, stdev=15.88, samples=30 lat (msec) : 50=2.11%, 100=65.94%, 250=31.95% cpu : usr=0.32%, sys=2.90%, ctx=4766, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179738: Mon Jul 13 17:00:04 2026 read: IOPS=80, BW=321KiB/s (329kB/s)(4976KiB/15509msec) slat (usec): min=82, max=27428, avg=6667.94, stdev=3478.97 clat (msec): min=18, max=168, avg=91.20, stdev=22.35 lat (msec): min=19, max=190, avg=97.87, stdev=23.24 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 57], 10.00th=[ 65], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 102], 80.00th=[ 110], 90.00th=[ 121], 95.00th=[ 131], | 99.00th=[ 155], 99.50th=[ 159], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 183, max= 480, per=24.49%, avg=315.87, stdev=69.91, samples=30 iops : min= 45, max= 120, avg=78.67, stdev=17.56, samples=30 write: IOPS=84, BW=339KiB/s (348kB/s)(5264KiB/15509msec); 0 zone resets slat (usec): min=53, max=7172, avg=210.00, stdev=368.76 clat (msec): min=39, max=197, avg=95.42, stdev=22.75 lat (msec): min=39, max=197, avg=95.63, stdev=22.75 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 79], | 30.00th=[ 85], 40.00th=[ 89], 50.00th=[ 94], 60.00th=[ 99], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 159], 99.50th=[ 165], 99.90th=[ 192], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 207, max= 417, per=25.05%, avg=336.40, stdev=60.62, samples=30 iops : min= 51, max= 104, avg=83.77, stdev=15.16, samples=30 lat (msec) : 20=0.04%, 50=1.95%, 100=63.24%, 250=34.77% cpu : usr=0.48%, sys=2.84%, ctx=4714, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179739: Mon Jul 13 17:00:04 2026 read: IOPS=80, BW=321KiB/s (328kB/s)(4984KiB/15542msec) slat (usec): min=1607, max=27584, avg=6693.69, stdev=3421.67 clat (msec): min=33, max=196, avg=91.56, stdev=22.25 lat (msec): min=36, max=204, avg=98.26, stdev=23.16 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 57], 10.00th=[ 66], 20.00th=[ 73], | 30.00th=[ 81], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 102], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 131], | 99.00th=[ 148], 99.50th=[ 157], 99.90th=[ 167], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 160, max= 464, per=24.59%, avg=317.17, stdev=64.90, samples=30 iops : min= 40, max= 116, avg=78.97, stdev=16.21, samples=30 write: IOPS=84, BW=338KiB/s (346kB/s)(5256KiB/15542msec); 0 zone resets slat (usec): min=45, max=10105, avg=221.55, stdev=419.21 clat (msec): min=25, max=164, avg=94.02, stdev=22.20 lat (msec): min=25, max=164, avg=94.25, stdev=22.20 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 58], 10.00th=[ 68], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 89], 50.00th=[ 93], 60.00th=[ 97], | 70.00th=[ 103], 80.00th=[ 111], 90.00th=[ 124], 95.00th=[ 136], | 99.00th=[ 153], 99.50th=[ 161], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 221, max= 504, per=24.66%, avg=331.13, stdev=72.43, samples=30 iops : min= 55, max= 126, avg=82.47, stdev=18.13, samples=30 lat (msec) : 50=2.73%, 100=64.30%, 250=32.97% cpu : usr=0.32%, sys=2.90%, ctx=4708, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179740: Mon Jul 13 17:00:04 2026 read: IOPS=81, BW=325KiB/s (333kB/s)(5060KiB/15548msec) slat (usec): min=1722, max=26855, avg=6684.23, stdev=3482.04 clat (msec): min=19, max=195, avg=91.15, stdev=23.19 lat (msec): min=22, max=203, avg=97.84, stdev=24.12 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 56], 10.00th=[ 64], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 109], 90.00th=[ 122], 95.00th=[ 133], | 99.00th=[ 159], 99.50th=[ 174], 99.90th=[ 182], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 180, max= 432, per=24.78%, avg=319.63, stdev=62.11, samples=30 iops : min= 45, max= 108, avg=79.63, stdev=15.52, samples=30 write: IOPS=83, BW=333KiB/s (341kB/s)(5180KiB/15548msec); 0 zone resets slat (usec): min=38, max=3417, avg=206.42, stdev=268.84 clat (msec): min=29, max=175, avg=94.65, stdev=23.21 lat (msec): min=29, max=175, avg=94.86, stdev=23.22 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 59], 10.00th=[ 67], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 92], 60.00th=[ 99], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 159], 99.50th=[ 171], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 207, max= 456, per=24.44%, avg=328.17, stdev=62.44, samples=30 iops : min= 51, max= 114, avg=81.70, stdev=15.70, samples=30 lat (msec) : 20=0.04%, 50=2.30%, 100=63.87%, 250=33.79% cpu : usr=0.33%, sys=2.92%, ctx=4679, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1291KiB/s (1322kB/s), 321KiB/s-326KiB/s (328kB/s-334kB/s), io=19.6MiB (20.5MB), run=15489-15548msec WRITE: bw=1344KiB/s (1376kB/s), 333KiB/s-339KiB/s (341kB/s-348kB/s), io=20.4MiB (21.4MB), run=15489-15548msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179746: Mon Jul 13 17:00:08 2026 read: IOPS=4, BW=196MiB/s (206MB/s)(40.0MiB/204msec) slat (nsec): min=13014k, max=13014k, avg=13014250.00, stdev= 0.00 clat (nsec): min=180843k, max=180843k, avg=180843422.00, stdev= 0.00 lat (nsec): min=193862k, max=193862k, avg=193862300.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.42%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=196MiB/s (206MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s), io=40.0MiB (41.9MB), run=204-204msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout