== sanity test 398c: run fio to test AIO ================= 20:13:12 (1772759592) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.12524 s, 335 MB/s osc.lustre-OST0000-osc-ffff8fc467a76000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=159660: Thu Mar 5 20:13:46 2026 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/30997msec); 0 zone resets slat (usec): min=40, max=1024, avg=236.76, stdev=106.03 clat (msec): min=42, max=326, avg=193.23, stdev=41.58 lat (msec): min=42, max=326, avg=193.46, stdev=41.59 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 128], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 266], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 326] bw ( KiB/s): min= 224, max= 626, per=24.91%, avg=328.56, stdev=59.85, samples=61 iops : min= 56, max= 156, avg=82.13, stdev=14.92, samples=61 lat (msec) : 50=0.04%, 100=2.81%, 250=89.10%, 500=8.05% cpu : usr=0.13%, sys=1.58%, ctx=2807, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159661: Thu Mar 5 20:13:46 2026 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31037msec); 0 zone resets slat (usec): min=41, max=1061, avg=230.16, stdev=115.76 clat (msec): min=66, max=326, avg=193.51, stdev=40.97 lat (msec): min=67, max=326, avg=193.74, stdev=40.99 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 128], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 226], 90.00th=[ 243], 95.00th=[ 262], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 313], 99.95th=[ 321], | 99.99th=[ 326] bw ( KiB/s): min= 224, max= 613, per=24.87%, avg=328.08, stdev=58.30, samples=61 iops : min= 56, max= 153, avg=82.02, stdev=14.55, samples=61 lat (msec) : 100=2.77%, 250=89.10%, 500=8.12% cpu : usr=0.11%, sys=1.51%, ctx=2762, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159662: Thu Mar 5 20:13:46 2026 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31037msec); 0 zone resets slat (usec): min=46, max=997, avg=232.39, stdev=110.15 clat (msec): min=62, max=330, avg=193.46, stdev=40.99 lat (msec): min=62, max=331, avg=193.69, stdev=41.01 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 203], | 70.00th=[ 211], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 266], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 330] bw ( KiB/s): min= 232, max= 613, per=24.84%, avg=327.69, stdev=58.69, samples=61 iops : min= 58, max= 153, avg=81.92, stdev=14.65, samples=61 lat (msec) : 100=2.62%, 250=89.49%, 500=7.89% cpu : usr=0.12%, sys=1.53%, ctx=2842, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159663: Thu Mar 5 20:13:46 2026 write: IOPS=82, BW=331KiB/s (339kB/s)(10.0MiB/30932msec); 0 zone resets slat (usec): min=42, max=990, avg=238.38, stdev=106.72 clat (msec): min=21, max=330, avg=192.97, stdev=42.12 lat (msec): min=21, max=331, avg=193.21, stdev=42.13 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 127], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 262], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 330] bw ( KiB/s): min= 216, max= 651, per=24.99%, avg=329.62, stdev=60.29, samples=61 iops : min= 54, max= 162, avg=82.39, stdev=15.01, samples=61 lat (msec) : 50=0.43%, 100=2.73%, 250=89.06%, 500=7.77% cpu : usr=0.14%, sys=1.61%, ctx=2870, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1320KiB/s (1351kB/s), 330KiB/s-331KiB/s (338kB/s-339kB/s), io=40.0MiB (41.9MB), run=30932-31037msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=159677: Thu Mar 5 20:14:08 2026 read: IOPS=66, BW=265KiB/s (271kB/s)(5048KiB/19067msec) slat (usec): min=1166, max=7411, avg=2481.75, stdev=772.12 clat (msec): min=3, max=142, avg=74.23, stdev=13.15 lat (msec): min=7, max=145, avg=76.72, stdev=13.15 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 65], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 93], | 99.00th=[ 102], 99.50th=[ 106], 99.90th=[ 140], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 184, max= 336, per=25.15%, avg=263.78, stdev=44.36, samples=37 iops : min= 46, max= 84, avg=65.95, stdev=11.09, samples=37 write: IOPS=68, BW=272KiB/s (279kB/s)(5192KiB/19067msec); 0 zone resets slat (usec): min=45, max=704, avg=186.59, stdev=90.60 clat (msec): min=71, max=269, avg=159.54, stdev=33.50 lat (msec): min=71, max=269, avg=159.73, stdev=33.50 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 239], 99.50th=[ 245], 99.90th=[ 268], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 224, max= 344, per=24.86%, avg=271.51, stdev=28.42, samples=37 iops : min= 56, max= 86, avg=67.86, stdev= 7.10, samples=37 lat (msec) : 4=0.04%, 10=0.04%, 20=0.31%, 50=0.70%, 100=49.53% lat (msec) : 250=49.22%, 500=0.16% cpu : usr=0.18%, sys=2.36%, ctx=4889, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159678: Thu Mar 5 20:14:08 2026 read: IOPS=65, BW=260KiB/s (266kB/s)(4976KiB/19125msec) slat (usec): min=1081, max=6414, avg=2453.46, stdev=746.12 clat (msec): min=2, max=119, avg=74.00, stdev=13.11 lat (msec): min=4, max=123, avg=76.46, stdev=13.13 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 89], 95.00th=[ 93], | 99.00th=[ 103], 99.50th=[ 105], 99.90th=[ 118], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 184, max= 368, per=24.81%, avg=260.22, stdev=49.17, samples=37 iops : min= 46, max= 92, avg=65.05, stdev=12.29, samples=37 write: IOPS=68, BW=275KiB/s (282kB/s)(5264KiB/19125msec); 0 zone resets slat (usec): min=59, max=726, avg=194.52, stdev=97.77 clat (msec): min=59, max=262, avg=159.36, stdev=32.70 lat (msec): min=59, max=262, avg=159.56, stdev=32.70 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 103], 10.00th=[ 116], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 186], 90.00th=[ 201], 95.00th=[ 215], | 99.00th=[ 236], 99.50th=[ 241], 99.90th=[ 255], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 216, max= 336, per=25.12%, avg=274.32, stdev=26.95, samples=37 iops : min= 54, max= 84, avg=68.57, stdev= 6.73, samples=37 lat (msec) : 4=0.16%, 10=0.04%, 20=0.47%, 50=0.66%, 100=48.40% lat (msec) : 250=50.16%, 500=0.12% cpu : usr=0.20%, sys=2.36%, ctx=4857, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159679: Thu Mar 5 20:14:08 2026 read: IOPS=65, BW=261KiB/s (267kB/s)(4984KiB/19124msec) slat (usec): min=1143, max=6451, avg=2480.99, stdev=781.25 clat (msec): min=2, max=117, avg=74.00, stdev=12.29 lat (msec): min=5, max=121, avg=76.48, stdev=12.33 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 88], 95.00th=[ 91], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 106], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 176, max= 360, per=24.88%, avg=260.97, stdev=55.28, samples=37 iops : min= 44, max= 90, avg=65.24, stdev=13.82, samples=37 write: IOPS=68, BW=275KiB/s (281kB/s)(5256KiB/19124msec); 0 zone resets slat (usec): min=49, max=1680, avg=188.30, stdev=98.81 clat (msec): min=59, max=252, avg=159.44, stdev=34.83 lat (msec): min=59, max=252, avg=159.63, stdev=34.83 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 130], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 251], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 200, max= 336, per=24.90%, avg=271.86, stdev=32.00, samples=37 iops : min= 50, max= 84, avg=67.95, stdev= 7.98, samples=37 lat (msec) : 4=0.04%, 10=0.20%, 20=0.23%, 50=0.70%, 100=49.61% lat (msec) : 250=49.10%, 500=0.12% cpu : usr=0.20%, sys=2.35%, ctx=4863, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159680: Thu Mar 5 20:14:08 2026 read: IOPS=66, BW=267KiB/s (273kB/s)(5060KiB/18967msec) slat (usec): min=72, max=6662, avg=2453.42, stdev=782.14 clat (msec): min=2, max=125, avg=73.84, stdev=13.23 lat (msec): min=4, max=128, avg=76.30, stdev=13.28 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 88], 95.00th=[ 92], | 99.00th=[ 101], 99.50th=[ 103], 99.90th=[ 109], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 168, max= 376, per=25.49%, avg=267.41, stdev=49.20, samples=37 iops : min= 42, max= 94, avg=66.84, stdev=12.29, samples=37 write: IOPS=68, BW=273KiB/s (280kB/s)(5180KiB/18967msec); 0 zone resets slat (usec): min=62, max=697, avg=186.02, stdev=85.80 clat (msec): min=35, max=255, avg=159.44, stdev=34.84 lat (msec): min=35, max=255, avg=159.62, stdev=34.84 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 216, max= 352, per=24.94%, avg=272.30, stdev=33.55, samples=37 iops : min= 54, max= 88, avg=68.05, stdev= 8.35, samples=37 lat (msec) : 4=0.08%, 10=0.27%, 20=0.39%, 50=0.66%, 100=49.88% lat (msec) : 250=48.59%, 500=0.12% cpu : usr=0.15%, sys=2.41%, ctx=4887, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1049KiB/s (1074kB/s), 260KiB/s-267KiB/s (266kB/s-273kB/s), io=19.6MiB (20.5MB), run=18967-19125msec WRITE: bw=1092KiB/s (1119kB/s), 272KiB/s-275KiB/s (279kB/s-282kB/s), io=20.4MiB (21.4MB), run=18967-19125msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=159686: Thu Mar 5 20:14:11 2026 read: IOPS=8, BW=345MiB/s (362MB/s)(40.0MiB/116msec) slat (nsec): min=8783.3k, max=8783.3k, avg=8783335.00, stdev= 0.00 clat (nsec): min=98710k, max=98710k, avg=98709640.00, stdev= 0.00 lat (nsec): min=107505k, max=107505k, avg=107505098.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[99091], 5.00th=[99091], 10.00th=[99091], 20.00th=[99091], | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=6.09%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=40.0MiB (41.9MB), run=116-116msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout