== sanity test 398c: run fio to test AIO ================= 18:54:48 (1786834488) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.569204 s, 73.7 MB/s osc.lustre-OST0000-osc-ffff8a87b9f28800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179234: Sat Aug 15 18:55:23 2026 write: IOPS=100, BW=404KiB/s (414kB/s)(10.0MiB/25350msec); 0 zone resets slat (usec): min=46, max=27009, avg=254.60, stdev=823.59 clat (msec): min=37, max=282, avg=157.81, stdev=43.14 lat (msec): min=37, max=284, avg=158.07, stdev=43.16 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 93], 10.00th=[ 108], 20.00th=[ 122], | 30.00th=[ 133], 40.00th=[ 144], 50.00th=[ 157], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 194], 90.00th=[ 215], 95.00th=[ 232], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 284] bw ( KiB/s): min= 254, max= 689, per=25.13%, avg=402.12, stdev=97.64, samples=50 iops : min= 63, max= 172, avg=100.24, stdev=24.45, samples=50 lat (msec) : 50=1.25%, 100=4.96%, 250=92.27%, 500=1.52% cpu : usr=0.13%, sys=1.08%, ctx=1815, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179235: Sat Aug 15 18:55:23 2026 write: IOPS=100, BW=400KiB/s (410kB/s)(10.0MiB/25591msec); 0 zone resets slat (usec): min=47, max=9592, avg=291.37, stdev=647.23 clat (msec): min=15, max=307, avg=158.79, stdev=41.60 lat (msec): min=16, max=309, avg=159.08, stdev=41.62 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 192], 90.00th=[ 213], 95.00th=[ 236], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 254, max= 631, per=24.91%, avg=398.58, stdev=82.45, samples=50 iops : min= 63, max= 157, avg=99.32, stdev=20.65, samples=50 lat (msec) : 20=0.08%, 50=0.27%, 100=4.61%, 250=93.28%, 500=1.76% cpu : usr=0.18%, sys=1.42%, ctx=2974, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179236: Sat Aug 15 18:55:23 2026 write: IOPS=100, BW=401KiB/s (411kB/s)(10.0MiB/25512msec); 0 zone resets slat (usec): min=48, max=15829, avg=274.33, stdev=688.23 clat (msec): min=20, max=314, avg=158.40, stdev=41.75 lat (msec): min=20, max=314, avg=158.68, stdev=41.74 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 124], | 30.00th=[ 133], 40.00th=[ 144], 50.00th=[ 157], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 192], 90.00th=[ 213], 95.00th=[ 232], | 99.00th=[ 259], 99.50th=[ 275], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 256, max= 670, per=24.98%, avg=399.72, stdev=87.86, samples=50 iops : min= 64, max= 167, avg=99.62, stdev=22.01, samples=50 lat (msec) : 50=0.51%, 100=4.92%, 250=92.66%, 500=1.91% cpu : usr=0.18%, sys=1.32%, ctx=2797, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179237: Sat Aug 15 18:55:23 2026 write: IOPS=100, BW=400KiB/s (410kB/s)(10.0MiB/25600msec); 0 zone resets slat (usec): min=46, max=13549, avg=288.16, stdev=712.50 clat (msec): min=20, max=313, avg=158.68, stdev=41.31 lat (msec): min=20, max=313, avg=158.97, stdev=41.33 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 109], 20.00th=[ 124], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 194], 90.00th=[ 215], 95.00th=[ 232], | 99.00th=[ 259], 99.50th=[ 268], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 255, max= 757, per=24.87%, avg=397.86, stdev=93.05, samples=50 iops : min= 63, max= 189, avg=99.16, stdev=23.34, samples=50 lat (msec) : 50=0.12%, 100=5.00%, 250=93.09%, 500=1.80% cpu : usr=0.21%, sys=1.27%, ctx=2791, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1600KiB/s (1638kB/s), 400KiB/s-404KiB/s (410kB/s-414kB/s), io=40.0MiB (41.9MB), run=25350-25600msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179251: Sat Aug 15 18:56:01 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(5048KiB/30578msec) slat (usec): min=2451, max=53018, avg=13450.95, stdev=7377.93 clat (msec): min=72, max=339, avg=179.03, stdev=46.75 lat (msec): min=78, max=373, avg=192.49, stdev=49.14 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 112], 10.00th=[ 123], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 222], 90.00th=[ 241], 95.00th=[ 264], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 85, max= 304, per=25.10%, avg=164.42, stdev=44.78, samples=60 iops : min= 21, max= 76, avg=40.58, stdev=11.28, samples=60 write: IOPS=42, BW=170KiB/s (174kB/s)(5192KiB/30578msec); 0 zone resets slat (usec): min=52, max=15735, avg=413.69, stdev=1096.75 clat (msec): min=88, max=328, avg=186.39, stdev=44.98 lat (msec): min=88, max=328, avg=186.80, stdev=45.00 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 228], 90.00th=[ 251], 95.00th=[ 266], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 70, max= 255, per=24.93%, avg=169.78, stdev=43.28, samples=60 iops : min= 17, max= 63, avg=41.93, stdev=10.83, samples=60 lat (msec) : 100=1.21%, 250=89.84%, 500=8.95% cpu : usr=0.30%, sys=2.14%, ctx=4610, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179252: Sat Aug 15 18:56:01 2026 read: IOPS=40, BW=163KiB/s (167kB/s)(4976KiB/30574msec) slat (usec): min=2611, max=62733, avg=13840.73, stdev=7674.77 clat (msec): min=17, max=342, avg=180.44, stdev=46.84 lat (msec): min=28, max=370, avg=194.29, stdev=49.30 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 243], 95.00th=[ 266], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 330], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 70, max= 248, per=24.70%, avg=161.80, stdev=40.11, samples=60 iops : min= 17, max= 62, avg=39.95, stdev=10.06, samples=60 write: IOPS=43, BW=172KiB/s (176kB/s)(5264KiB/30574msec); 0 zone resets slat (usec): min=51, max=16552, avg=374.93, stdev=939.85 clat (msec): min=83, max=309, avg=185.03, stdev=45.28 lat (msec): min=83, max=309, avg=185.41, stdev=45.35 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 79, max= 278, per=25.29%, avg=172.25, stdev=46.43, samples=60 iops : min= 19, max= 69, avg=42.53, stdev=11.70, samples=60 lat (msec) : 20=0.04%, 50=0.08%, 100=1.48%, 250=89.84%, 500=8.55% cpu : usr=0.24%, sys=2.19%, ctx=4498, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179253: Sat Aug 15 18:56:01 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(4984KiB/30285msec) slat (usec): min=99, max=70879, avg=13499.77, stdev=7634.99 clat (msec): min=43, max=323, avg=178.89, stdev=46.13 lat (msec): min=43, max=343, avg=192.40, stdev=48.50 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 112], 10.00th=[ 122], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 317], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 103, max= 255, per=25.06%, avg=164.12, stdev=38.25, samples=60 iops : min= 25, max= 63, avg=40.52, stdev= 9.61, samples=60 write: IOPS=43, BW=174KiB/s (178kB/s)(5256KiB/30285msec); 0 zone resets slat (usec): min=53, max=12997, avg=396.37, stdev=960.99 clat (msec): min=82, max=338, avg=185.31, stdev=44.67 lat (msec): min=82, max=338, avg=185.71, stdev=44.72 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 249], 95.00th=[ 264], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 79, max= 288, per=25.32%, avg=172.45, stdev=47.22, samples=60 iops : min= 19, max= 72, avg=42.58, stdev=11.86, samples=60 lat (msec) : 50=0.12%, 100=1.33%, 250=90.23%, 500=8.32% cpu : usr=0.30%, sys=2.18%, ctx=4570, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179254: Sat Aug 15 18:56:01 2026 read: IOPS=41, BW=165KiB/s (169kB/s)(5060KiB/30638msec) slat (usec): min=164, max=69854, avg=13698.38, stdev=8211.51 clat (msec): min=46, max=325, avg=179.71, stdev=45.91 lat (msec): min=46, max=356, avg=193.41, stdev=48.51 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 245], 95.00th=[ 262], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 71, max= 280, per=25.06%, avg=164.13, stdev=43.09, samples=60 iops : min= 17, max= 70, avg=40.50, stdev=10.83, samples=60 write: IOPS=42, BW=169KiB/s (173kB/s)(5180KiB/30638msec); 0 zone resets slat (usec): min=51, max=18923, avg=403.63, stdev=954.23 clat (msec): min=23, max=312, avg=186.09, stdev=46.89 lat (msec): min=24, max=313, avg=186.49, stdev=46.91 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 116], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 194], | 70.00th=[ 211], 80.00th=[ 230], 90.00th=[ 253], 95.00th=[ 271], | 99.00th=[ 296], 99.50th=[ 300], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 85, max= 270, per=24.63%, avg=167.70, stdev=39.24, samples=60 iops : min= 21, max= 67, avg=41.42, stdev= 9.82, samples=60 lat (msec) : 50=0.16%, 100=1.76%, 250=88.52%, 500=9.57% cpu : usr=0.29%, sys=2.15%, ctx=4670, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=655KiB/s (671kB/s), 163KiB/s-165KiB/s (167kB/s-169kB/s), io=19.6MiB (20.5MB), run=30285-30638msec WRITE: bw=682KiB/s (698kB/s), 169KiB/s-174KiB/s (173kB/s-178kB/s), io=20.4MiB (21.4MB), run=30285-30638msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179263: Sat Aug 15 18:56:11 2026 read: IOPS=1, BW=76.2MiB/s (79.9MB/s)(40.0MiB/525msec) slat (nsec): min=26985k, max=26985k, avg=26984555.00, stdev= 0.00 clat (nsec): min=463496k, max=463496k, avg=463496214.00, stdev= 0.00 lat (nsec): min=490485k, max=490485k, avg=490485417.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.91%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.2MiB/s (79.9MB/s), 76.2MiB/s-76.2MiB/s (79.9MB/s-79.9MB/s), io=40.0MiB (41.9MB), run=525-525msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout