== sanity test 398c: run fio to test AIO ================= 22:51:23 (1787712683) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.484891 s, 86.5 MB/s osc.lustre-OST0000-osc-ffff96576cec3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179633: Tue Aug 25 22:52:02 2026 write: IOPS=88, BW=353KiB/s (362kB/s)(10.0MiB/28978msec); 0 zone resets slat (usec): min=28, max=16924, avg=244.14, stdev=827.53 clat (msec): min=38, max=329, avg=180.54, stdev=47.56 lat (msec): min=38, max=330, avg=180.78, stdev=47.58 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 114], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 174], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 275], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 190, max= 704, per=24.82%, avg=350.47, stdev=87.02, samples=57 iops : min= 47, max= 176, avg=87.16, stdev=21.78, samples=57 lat (msec) : 50=0.43%, 100=3.44%, 250=88.20%, 500=7.93% cpu : usr=0.22%, sys=1.05%, ctx=2514, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179634: Tue Aug 25 22:52:02 2026 write: IOPS=88, BW=353KiB/s (362kB/s)(10.0MiB/29001msec); 0 zone resets slat (usec): min=23, max=8983, avg=212.27, stdev=616.30 clat (msec): min=14, max=330, avg=180.73, stdev=48.69 lat (msec): min=14, max=332, avg=180.95, stdev=48.71 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 113], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 239, max= 699, per=24.85%, avg=350.88, stdev=81.35, samples=57 iops : min= 59, max= 174, avg=87.26, stdev=20.31, samples=57 lat (msec) : 20=0.12%, 50=0.78%, 100=3.24%, 250=86.95%, 500=8.91% cpu : usr=0.17%, sys=0.93%, ctx=1944, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179635: Tue Aug 25 22:52:02 2026 write: IOPS=88, BW=354KiB/s (362kB/s)(10.0MiB/28934msec); 0 zone resets slat (usec): min=27, max=23181, avg=229.08, stdev=848.66 clat (msec): min=18, max=333, avg=180.49, stdev=47.88 lat (msec): min=19, max=334, avg=180.72, stdev=47.87 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 109], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 175, max= 712, per=24.86%, avg=351.00, stdev=85.56, samples=57 iops : min= 43, max= 178, avg=87.30, stdev=21.46, samples=57 lat (msec) : 20=0.04%, 50=0.39%, 100=4.02%, 250=86.72%, 500=8.83% cpu : usr=0.21%, sys=0.98%, ctx=2406, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179636: Tue Aug 25 22:52:02 2026 write: IOPS=88, BW=353KiB/s (362kB/s)(10.0MiB/29004msec); 0 zone resets slat (usec): min=29, max=19106, avg=232.68, stdev=860.74 clat (msec): min=9, max=330, avg=180.67, stdev=47.19 lat (msec): min=15, max=333, avg=180.91, stdev=47.18 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 113], 10.00th=[ 134], 20.00th=[ 146], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 218], 90.00th=[ 245], 95.00th=[ 264], | 99.00th=[ 321], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 215, max= 637, per=24.79%, avg=349.96, stdev=77.16, samples=57 iops : min= 53, max= 159, avg=87.04, stdev=19.31, samples=57 lat (msec) : 10=0.04%, 50=0.08%, 100=3.71%, 250=88.20%, 500=7.97% cpu : usr=0.15%, sys=0.93%, ctx=2285, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1412KiB/s (1446kB/s), 353KiB/s-354KiB/s (362kB/s-362kB/s), io=40.0MiB (41.9MB), run=28934-29004msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179650: Tue Aug 25 22:52:42 2026 read: IOPS=38, BW=155KiB/s (158kB/s)(5048KiB/32617msec) slat (usec): min=3175, max=65751, avg=14313.28, stdev=8467.08 clat (msec): min=7, max=401, avg=193.13, stdev=50.46 lat (msec): min=10, max=421, avg=207.46, stdev=52.48 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 14, max= 245, per=24.97%, avg=153.55, stdev=40.78, samples=64 iops : min= 3, max= 61, avg=37.92, stdev=10.28, samples=64 write: IOPS=39, BW=159KiB/s (163kB/s)(5192KiB/32617msec); 0 zone resets slat (usec): min=30, max=9070, avg=214.65, stdev=562.53 clat (msec): min=22, max=352, avg=196.21, stdev=45.45 lat (msec): min=23, max=352, avg=196.43, stdev=45.43 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 77, max= 245, per=25.17%, avg=161.11, stdev=39.26, samples=63 iops : min= 19, max= 61, avg=39.84, stdev= 9.80, samples=63 lat (msec) : 10=0.04%, 50=0.16%, 100=0.47%, 250=86.17%, 500=13.16% cpu : usr=0.28%, sys=1.85%, ctx=4614, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179651: Tue Aug 25 22:52:42 2026 read: IOPS=38, BW=153KiB/s (156kB/s)(4976KiB/32591msec) slat (msec): min=2, max=192, avg=14.27, stdev=10.28 clat (msec): min=21, max=485, avg=191.59, stdev=51.61 lat (msec): min=35, max=501, avg=205.86, stdev=54.10 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 123], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 255], 95.00th=[ 279], | 99.00th=[ 342], 99.50th=[ 384], 99.90th=[ 477], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 29, max= 224, per=24.62%, avg=151.39, stdev=40.04, samples=64 iops : min= 7, max= 56, avg=37.41, stdev=10.08, samples=64 write: IOPS=40, BW=162KiB/s (165kB/s)(5264KiB/32591msec); 0 zone resets slat (usec): min=30, max=8562, avg=245.00, stdev=629.83 clat (msec): min=90, max=517, avg=198.87, stdev=51.14 lat (msec): min=92, max=517, avg=199.12, stdev=51.15 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 133], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 234], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 355], 99.50th=[ 460], 99.90th=[ 498], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 81, max= 262, per=25.22%, avg=161.39, stdev=40.05, samples=64 iops : min= 20, max= 65, avg=39.91, stdev=10.02, samples=64 lat (msec) : 50=0.12%, 100=0.23%, 250=87.07%, 500=12.54%, 750=0.04% cpu : usr=0.28%, sys=1.80%, ctx=4497, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179652: Tue Aug 25 22:52:42 2026 read: IOPS=38, BW=154KiB/s (158kB/s)(4984KiB/32316msec) slat (usec): min=29, max=61924, avg=14311.97, stdev=8199.28 clat (msec): min=22, max=374, avg=192.43, stdev=48.49 lat (msec): min=34, max=382, avg=206.75, stdev=50.76 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 122], 10.00th=[ 134], 20.00th=[ 150], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 257], 95.00th=[ 288], | 99.00th=[ 317], 99.50th=[ 326], 99.90th=[ 342], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 86, max= 252, per=25.05%, avg=154.03, stdev=38.27, samples=64 iops : min= 21, max= 63, avg=38.06, stdev= 9.65, samples=64 write: IOPS=40, BW=163KiB/s (167kB/s)(5256KiB/32316msec); 0 zone resets slat (usec): min=28, max=23680, avg=308.67, stdev=1136.93 clat (msec): min=70, max=399, avg=196.27, stdev=46.99 lat (msec): min=70, max=400, avg=196.58, stdev=47.00 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 132], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 85, max= 270, per=25.21%, avg=161.36, stdev=43.60, samples=64 iops : min= 21, max= 67, avg=39.89, stdev=10.95, samples=64 lat (msec) : 50=0.12%, 100=0.51%, 250=86.76%, 500=12.62% cpu : usr=0.25%, sys=1.82%, ctx=4344, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179653: Tue Aug 25 22:52:42 2026 read: IOPS=38, BW=155KiB/s (159kB/s)(5060KiB/32552msec) slat (usec): min=2523, max=49366, avg=13866.91, stdev=7540.12 clat (msec): min=27, max=374, avg=191.53, stdev=46.79 lat (msec): min=42, max=384, avg=205.40, stdev=48.82 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 368], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 30, max= 240, per=25.23%, avg=155.17, stdev=41.80, samples=64 iops : min= 7, max= 60, avg=38.36, stdev=10.55, samples=64 write: IOPS=39, BW=159KiB/s (163kB/s)(5180KiB/32552msec); 0 zone resets slat (usec): min=29, max=10527, avg=227.69, stdev=569.54 clat (msec): min=46, max=347, avg=198.42, stdev=44.41 lat (msec): min=46, max=347, avg=198.65, stdev=44.43 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 53, max= 245, per=24.79%, avg=158.67, stdev=36.29, samples=64 iops : min= 13, max= 61, avg=39.23, stdev= 9.06, samples=64 lat (msec) : 50=0.08%, 100=0.31%, 250=87.70%, 500=11.91% cpu : usr=0.24%, sys=1.76%, ctx=4513, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=615KiB/s (630kB/s), 153KiB/s-155KiB/s (156kB/s-159kB/s), io=19.6MiB (20.5MB), run=32316-32617msec WRITE: bw=641KiB/s (656kB/s), 159KiB/s-163KiB/s (163kB/s-167kB/s), io=20.4MiB (21.4MB), run=32316-32617msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179662: Tue Aug 25 22:52:50 2026 read: IOPS=2, BW=90.5MiB/s (94.9MB/s)(40.0MiB/442msec) slat (nsec): min=16182k, max=16182k, avg=16181519.00, stdev= 0.00 clat (nsec): min=396587k, max=396587k, avg=396586859.00, stdev= 0.00 lat (nsec): min=419612k, max=419612k, avg=419611575.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.95%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=90.5MiB/s (94.9MB/s), 90.5MiB/s-90.5MiB/s (94.9MB/s-94.9MB/s), io=40.0MiB (41.9MB), run=442-442msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout