== sanity test 398c: run fio to test AIO ================= 11:56:26 (1787586986) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.437367 s, 95.9 MB/s osc.lustre-OST0000-osc-ffff95801c21a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180316: Mon Aug 24 11:57:04 2026 write: IOPS=89, BW=357KiB/s (366kB/s)(10.0MiB/28683msec); 0 zone resets slat (usec): min=28, max=10051, avg=230.41, stdev=639.82 clat (msec): min=29, max=309, avg=178.36, stdev=39.09 lat (msec): min=30, max=309, avg=178.59, stdev=39.09 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 167], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 247, max= 504, per=24.91%, avg=354.91, stdev=59.31, samples=57 iops : min= 61, max= 126, avg=88.33, stdev=14.85, samples=57 lat (msec) : 50=0.31%, 100=0.94%, 250=94.45%, 500=4.30% cpu : usr=0.18%, sys=1.02%, ctx=2798, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180317: Mon Aug 24 11:57:04 2026 write: IOPS=89, BW=358KiB/s (366kB/s)(10.0MiB/28635msec); 0 zone resets slat (usec): min=29, max=17152, avg=247.57, stdev=888.28 clat (msec): min=49, max=310, avg=178.54, stdev=38.18 lat (msec): min=49, max=310, avg=178.79, stdev=38.18 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 249], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 239, max= 484, per=25.02%, avg=356.55, stdev=59.95, samples=56 iops : min= 59, max= 121, avg=88.75, stdev=15.01, samples=56 lat (msec) : 50=0.04%, 100=1.33%, 250=93.91%, 500=4.73% cpu : usr=0.24%, sys=0.94%, ctx=2741, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180318: Mon Aug 24 11:57:04 2026 write: IOPS=89, BW=356KiB/s (365kB/s)(10.0MiB/28734msec); 0 zone resets slat (usec): min=28, max=31930, avg=260.03, stdev=1020.29 clat (msec): min=29, max=319, avg=178.43, stdev=37.62 lat (msec): min=29, max=319, avg=178.69, stdev=37.65 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 126], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 243], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 223, max= 504, per=24.92%, avg=355.14, stdev=58.77, samples=57 iops : min= 55, max= 126, avg=88.42, stdev=14.73, samples=57 lat (msec) : 50=0.04%, 100=1.13%, 250=95.00%, 500=3.83% cpu : usr=0.14%, sys=1.05%, ctx=2872, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180319: Mon Aug 24 11:57:04 2026 write: IOPS=89, BW=356KiB/s (365kB/s)(10.0MiB/28732msec); 0 zone resets slat (usec): min=28, max=17648, avg=221.97, stdev=754.79 clat (msec): min=20, max=331, avg=178.51, stdev=38.34 lat (msec): min=20, max=331, avg=178.74, stdev=38.32 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 121], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 143, max= 496, per=24.92%, avg=355.16, stdev=58.34, samples=57 iops : min= 35, max= 124, avg=88.42, stdev=14.62, samples=57 lat (msec) : 50=0.12%, 100=1.72%, 250=94.49%, 500=3.67% cpu : usr=0.18%, sys=0.97%, ctx=2542, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1425KiB/s (1460kB/s), 356KiB/s-358KiB/s (365kB/s-366kB/s), io=40.0MiB (41.9MB), run=28635-28734msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180334: Mon Aug 24 11:57:42 2026 read: IOPS=43, BW=173KiB/s (177kB/s)(5048KiB/29149msec) slat (usec): min=2807, max=52552, avg=12784.83, stdev=6769.63 clat (msec): min=24, max=305, avg=170.74, stdev=37.09 lat (msec): min=39, max=319, avg=183.54, stdev=38.43 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 239], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 76, max= 238, per=25.05%, avg=172.32, stdev=31.49, samples=57 iops : min= 19, max= 59, avg=42.65, stdev= 7.88, samples=57 write: IOPS=44, BW=178KiB/s (182kB/s)(5192KiB/29149msec); 0 zone resets slat (usec): min=30, max=11240, avg=236.01, stdev=668.31 clat (msec): min=24, max=301, avg=178.19, stdev=35.08 lat (msec): min=25, max=302, avg=178.43, stdev=35.05 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 126], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 239], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 53, max= 272, per=24.82%, avg=177.72, stdev=36.82, samples=57 iops : min= 13, max= 68, avg=43.98, stdev= 9.27, samples=57 lat (msec) : 50=0.20%, 100=0.39%, 250=96.76%, 500=2.66% cpu : usr=0.25%, sys=1.72%, ctx=4291, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180335: Mon Aug 24 11:57:42 2026 read: IOPS=42, BW=172KiB/s (176kB/s)(4976KiB/28989msec) slat (usec): min=56, max=60100, avg=12340.47, stdev=6500.20 clat (msec): min=11, max=474, avg=172.77, stdev=41.10 lat (msec): min=17, max=485, avg=185.12, stdev=42.28 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 121], 10.00th=[ 129], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 220], 95.00th=[ 239], | 99.00th=[ 288], 99.50th=[ 326], 99.90th=[ 464], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 95, max= 255, per=24.94%, avg=171.60, stdev=33.59, samples=57 iops : min= 23, max= 63, avg=42.46, stdev= 8.48, samples=57 write: IOPS=45, BW=182KiB/s (186kB/s)(5264KiB/28989msec); 0 zone resets slat (usec): min=34, max=10109, avg=235.49, stdev=700.07 clat (msec): min=46, max=419, avg=176.56, stdev=37.13 lat (msec): min=47, max=419, avg=176.80, stdev=37.16 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 125], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 111, max= 296, per=25.47%, avg=182.35, stdev=38.95, samples=57 iops : min= 27, max= 74, avg=45.12, stdev= 9.81, samples=57 lat (msec) : 20=0.08%, 50=0.08%, 100=0.70%, 250=95.70%, 500=3.44% cpu : usr=0.28%, sys=1.86%, ctx=4439, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180336: Mon Aug 24 11:57:42 2026 read: IOPS=42, BW=172KiB/s (176kB/s)(4984KiB/29045msec) slat (usec): min=2610, max=58137, avg=12351.85, stdev=6331.90 clat (msec): min=79, max=406, avg=170.53, stdev=36.46 lat (msec): min=86, max=436, avg=182.89, stdev=37.31 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 118], 10.00th=[ 127], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 236], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 380], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 91, max= 256, per=25.07%, avg=172.47, stdev=33.29, samples=57 iops : min= 22, max= 64, avg=42.67, stdev= 8.45, samples=57 write: IOPS=45, BW=181KiB/s (185kB/s)(5256KiB/29045msec); 0 zone resets slat (usec): min=32, max=22230, avg=228.05, stdev=779.26 clat (msec): min=90, max=466, avg=178.08, stdev=38.86 lat (msec): min=91, max=466, avg=178.31, stdev=38.87 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 148], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 279], 99.50th=[ 347], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 84, max= 269, per=25.31%, avg=181.19, stdev=39.08, samples=57 iops : min= 21, max= 67, avg=44.86, stdev= 9.78, samples=57 lat (msec) : 100=0.39%, 250=96.91%, 500=2.70% cpu : usr=0.25%, sys=1.81%, ctx=4379, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180337: Mon Aug 24 11:57:42 2026 read: IOPS=43, BW=174KiB/s (178kB/s)(5060KiB/29073msec) slat (usec): min=2551, max=42243, avg=12353.93, stdev=6470.63 clat (msec): min=20, max=333, avg=170.84, stdev=35.65 lat (msec): min=31, max=350, avg=183.20, stdev=36.97 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 121], 10.00th=[ 128], 20.00th=[ 140], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 218], 95.00th=[ 232], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 321], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 77, max= 239, per=25.34%, avg=174.33, stdev=34.44, samples=57 iops : min= 19, max= 59, avg=43.14, stdev= 8.63, samples=57 write: IOPS=44, BW=178KiB/s (182kB/s)(5180KiB/29073msec); 0 zone resets slat (usec): min=28, max=12088, avg=251.39, stdev=762.53 clat (msec): min=97, max=386, avg=178.04, stdev=34.88 lat (msec): min=97, max=386, avg=178.29, stdev=34.91 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 127], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 93, max= 255, per=24.86%, avg=177.96, stdev=31.29, samples=57 iops : min= 23, max= 63, avg=44.05, stdev= 7.92, samples=57 lat (msec) : 50=0.04%, 100=0.27%, 250=97.42%, 500=2.27% cpu : usr=0.24%, sys=1.91%, ctx=4461, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=688KiB/s (705kB/s), 172KiB/s-174KiB/s (176kB/s-178kB/s), io=19.6MiB (20.5MB), run=28989-29149msec WRITE: bw=717KiB/s (734kB/s), 178KiB/s-182KiB/s (182kB/s-186kB/s), io=20.4MiB (21.4MB), run=28989-29149msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180345: Mon Aug 24 11:57:50 2026 read: IOPS=1, BW=63.4MiB/s (66.5MB/s)(40.0MiB/631msec) slat (nsec): min=34507k, max=34507k, avg=34506536.00, stdev= 0.00 clat (nsec): min=571894k, max=571894k, avg=571893533.00, stdev= 0.00 lat (nsec): min=606404k, max=606404k, avg=606403749.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=2.54%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.4MiB/s (66.5MB/s), 63.4MiB/s-63.4MiB/s (66.5MB/s-66.5MB/s), io=40.0MiB (41.9MB), run=631-631msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout