== sanity test 398c: run fio to test AIO ================= 20:06:26 (1787789186) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.511298 s, 82.0 MB/s osc.lustre-OST0000-osc-ffff9938f61d7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179413: Wed Aug 26 20:07:02 2026 write: IOPS=99, BW=397KiB/s (407kB/s)(10.0MiB/25794msec); 0 zone resets slat (usec): min=30, max=8254, avg=165.43, stdev=458.37 clat (msec): min=28, max=330, avg=160.23, stdev=41.46 lat (msec): min=28, max=330, avg=160.40, stdev=41.46 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 107], 10.00th=[ 115], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 218], 95.00th=[ 247], | 99.00th=[ 266], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 247, max= 641, per=25.01%, avg=396.94, stdev=77.30, samples=51 iops : min= 61, max= 160, avg=98.94, stdev=19.40, samples=51 lat (msec) : 50=0.12%, 100=2.30%, 250=94.10%, 500=3.48% cpu : usr=0.17%, sys=1.00%, ctx=2936, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179414: Wed Aug 26 20:07:02 2026 write: IOPS=100, BW=401KiB/s (410kB/s)(10.0MiB/25560msec); 0 zone resets slat (usec): min=22, max=9587, avg=181.53, stdev=541.66 clat (msec): min=13, max=327, avg=159.42, stdev=42.16 lat (msec): min=15, max=327, avg=159.60, stdev=42.14 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 220], 95.00th=[ 245], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 263, max= 594, per=25.24%, avg=400.58, stdev=80.25, samples=50 iops : min= 65, max= 148, avg=99.86, stdev=20.10, samples=50 lat (msec) : 20=0.08%, 50=0.31%, 100=4.02%, 250=92.77%, 500=2.81% cpu : usr=0.18%, sys=1.03%, ctx=2438, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179415: Wed Aug 26 20:07:02 2026 write: IOPS=99, BW=398KiB/s (407kB/s)(10.0MiB/25750msec); 0 zone resets slat (usec): min=21, max=14366, avg=205.87, stdev=799.59 clat (msec): min=20, max=327, avg=159.82, stdev=41.70 lat (msec): min=21, max=327, avg=160.02, stdev=41.70 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 108], 10.00th=[ 115], 20.00th=[ 127], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 222], 95.00th=[ 239], | 99.00th=[ 266], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 239, max= 633, per=25.01%, avg=396.96, stdev=82.17, samples=51 iops : min= 59, max= 158, avg=98.96, stdev=20.60, samples=51 lat (msec) : 50=0.31%, 100=2.81%, 250=93.55%, 500=3.32% cpu : usr=0.22%, sys=1.01%, ctx=2974, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179416: Wed Aug 26 20:07:02 2026 write: IOPS=99, BW=397KiB/s (407kB/s)(10.0MiB/25791msec); 0 zone resets slat (usec): min=28, max=16573, avg=156.64, stdev=672.41 clat (msec): min=13, max=340, avg=160.13, stdev=41.12 lat (msec): min=13, max=340, avg=160.29, stdev=41.14 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 215], 95.00th=[ 236], | 99.00th=[ 284], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 231, max= 633, per=24.99%, avg=396.55, stdev=89.08, samples=51 iops : min= 57, max= 158, avg=98.86, stdev=22.34, samples=51 lat (msec) : 20=0.04%, 50=0.04%, 100=3.48%, 250=93.16%, 500=3.28% cpu : usr=0.15%, sys=0.83%, ctx=2048, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1588KiB/s (1626kB/s), 397KiB/s-401KiB/s (407kB/s-410kB/s), io=40.0MiB (41.9MB), run=25560-25794msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179430: Wed Aug 26 20:07:44 2026 read: IOPS=35, BW=141KiB/s (145kB/s)(5048KiB/35686msec) slat (usec): min=2194, max=98162, avg=15514.84, stdev=10478.70 clat (msec): min=48, max=486, avg=211.37, stdev=72.88 lat (msec): min=57, max=555, avg=226.89, stdev=77.04 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 102], 10.00th=[ 126], 20.00th=[ 157], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 239], 80.00th=[ 266], 90.00th=[ 305], 95.00th=[ 347], | 99.00th=[ 430], 99.50th=[ 435], 99.90th=[ 460], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 31, max= 310, per=24.89%, avg=139.90, stdev=43.40, samples=70 iops : min= 7, max= 77, avg=34.50, stdev=10.90, samples=70 write: IOPS=36, BW=145KiB/s (149kB/s)(5192KiB/35686msec); 0 zone resets slat (usec): min=22, max=30142, avg=232.39, stdev=1122.57 clat (msec): min=50, max=483, avg=217.21, stdev=74.78 lat (msec): min=50, max=483, avg=217.44, stdev=74.80 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 107], 10.00th=[ 127], 20.00th=[ 157], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 226], | 70.00th=[ 251], 80.00th=[ 279], 90.00th=[ 317], 95.00th=[ 359], | 99.00th=[ 422], 99.50th=[ 435], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 64, max= 302, per=24.70%, avg=144.50, stdev=47.55, samples=70 iops : min= 16, max= 75, avg=35.64, stdev=11.93, samples=70 lat (msec) : 50=0.04%, 100=4.41%, 250=67.50%, 500=28.05% cpu : usr=0.29%, sys=1.51%, ctx=4318, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179431: Wed Aug 26 20:07:44 2026 read: IOPS=34, BW=140KiB/s (143kB/s)(4976KiB/35637msec) slat (usec): min=1886, max=91045, avg=15246.71, stdev=10586.42 clat (msec): min=24, max=452, avg=209.84, stdev=72.60 lat (msec): min=29, max=525, avg=225.09, stdev=76.62 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 101], 10.00th=[ 122], 20.00th=[ 150], | 30.00th=[ 171], 40.00th=[ 188], 50.00th=[ 203], 60.00th=[ 220], | 70.00th=[ 241], 80.00th=[ 264], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 409], 99.50th=[ 426], 99.90th=[ 443], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 55, max= 302, per=24.62%, avg=138.36, stdev=47.03, samples=70 iops : min= 13, max= 75, avg=34.11, stdev=11.78, samples=70 write: IOPS=36, BW=148KiB/s (151kB/s)(5264KiB/35637msec); 0 zone resets slat (usec): min=28, max=19316, avg=246.87, stdev=1063.76 clat (msec): min=38, max=462, avg=218.40, stdev=71.57 lat (msec): min=38, max=462, avg=218.65, stdev=71.61 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 105], 10.00th=[ 133], 20.00th=[ 161], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 211], 60.00th=[ 230], | 70.00th=[ 251], 80.00th=[ 275], 90.00th=[ 313], 95.00th=[ 355], | 99.00th=[ 418], 99.50th=[ 435], 99.90th=[ 447], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 63, max= 326, per=25.15%, avg=147.10, stdev=49.60, samples=70 iops : min= 15, max= 81, avg=36.31, stdev=12.47, samples=70 lat (msec) : 50=0.12%, 100=4.38%, 250=67.62%, 500=27.89% cpu : usr=0.26%, sys=1.67%, ctx=4290, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179432: Wed Aug 26 20:07:44 2026 read: IOPS=35, BW=140KiB/s (144kB/s)(4984KiB/35556msec) slat (usec): min=41, max=282584, avg=15717.77, stdev=12996.29 clat (msec): min=14, max=561, avg=210.41, stdev=77.02 lat (msec): min=14, max=575, avg=226.13, stdev=81.11 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 99], 10.00th=[ 124], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 199], 60.00th=[ 218], | 70.00th=[ 241], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 355], | 99.00th=[ 464], 99.50th=[ 472], 99.90th=[ 542], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 64, max= 302, per=24.82%, avg=139.49, stdev=50.78, samples=70 iops : min= 16, max= 75, avg=34.39, stdev=12.73, samples=70 write: IOPS=36, BW=148KiB/s (151kB/s)(5256KiB/35556msec); 0 zone resets slat (usec): min=23, max=20599, avg=220.97, stdev=1029.87 clat (msec): min=43, max=541, avg=217.90, stdev=75.13 lat (msec): min=44, max=541, avg=218.12, stdev=75.15 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 107], 10.00th=[ 131], 20.00th=[ 161], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 209], 60.00th=[ 228], | 70.00th=[ 249], 80.00th=[ 275], 90.00th=[ 313], 95.00th=[ 351], | 99.00th=[ 426], 99.50th=[ 464], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 47, max= 294, per=25.05%, avg=146.51, stdev=48.79, samples=70 iops : min= 11, max= 73, avg=36.14, stdev=12.23, samples=70 lat (msec) : 20=0.04%, 50=0.16%, 100=4.38%, 250=67.30%, 500=27.85% lat (msec) : 750=0.27% cpu : usr=0.17%, sys=1.74%, ctx=4309, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179433: Wed Aug 26 20:07:44 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(5060KiB/35664msec) slat (usec): min=2174, max=92219, avg=15783.62, stdev=10991.92 clat (msec): min=18, max=447, avg=211.81, stdev=73.51 lat (msec): min=22, max=513, avg=227.60, stdev=77.79 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 105], 10.00th=[ 124], 20.00th=[ 153], | 30.00th=[ 171], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 220], | 70.00th=[ 243], 80.00th=[ 271], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 422], 99.50th=[ 435], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 63, max= 326, per=25.06%, avg=140.84, stdev=47.10, samples=70 iops : min= 15, max= 81, avg=34.73, stdev=11.82, samples=70 write: IOPS=36, BW=145KiB/s (149kB/s)(5180KiB/35664msec); 0 zone resets slat (usec): min=24, max=20440, avg=240.61, stdev=1036.48 clat (msec): min=54, max=492, avg=216.35, stdev=71.92 lat (msec): min=54, max=492, avg=216.59, stdev=71.94 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 101], 10.00th=[ 133], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 245], 80.00th=[ 275], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 405], 99.50th=[ 426], 99.90th=[ 481], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 39, max= 262, per=24.61%, avg=143.97, stdev=44.75, samples=70 iops : min= 9, max= 65, avg=35.51, stdev=11.23, samples=70 lat (msec) : 20=0.04%, 50=0.16%, 100=4.41%, 250=67.77%, 500=27.62% cpu : usr=0.22%, sys=1.84%, ctx=4531, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=562KiB/s (576kB/s), 140KiB/s-142KiB/s (143kB/s-145kB/s), io=19.6MiB (20.5MB), run=35556-35686msec WRITE: bw=585KiB/s (599kB/s), 145KiB/s-148KiB/s (149kB/s-151kB/s), io=20.4MiB (21.4MB), run=35556-35686msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179443: Wed Aug 26 20:07:54 2026 read: IOPS=2, BW=102MiB/s (107MB/s)(40.0MiB/393msec) slat (nsec): min=24760k, max=24760k, avg=24760198.00, stdev= 0.00 clat (nsec): min=350827k, max=350827k, avg=350827399.00, stdev= 0.00 lat (nsec): min=375592k, max=375592k, avg=375592022.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.04%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=40.0MiB (41.9MB), run=393-393msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout