== sanity test 398c: run fio to test AIO ================= 10:17:38 (1787062658) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.142533 s, 294 MB/s osc.lustre-OST0000-osc-ffff986e20c74800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=161287: Tue Aug 18 10:18:12 2026 write: IOPS=83, BW=334KiB/s (342kB/s)(10.0MiB/30658msec); 0 zone resets slat (usec): min=41, max=947, avg=220.61, stdev=104.79 clat (msec): min=35, max=297, avg=191.27, stdev=33.71 lat (msec): min=35, max=298, avg=191.49, stdev=33.73 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 138], 10.00th=[ 157], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 218], 90.00th=[ 230], 95.00th=[ 239], | 99.00th=[ 268], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 248, max= 612, per=24.88%, avg=331.93, stdev=48.63, samples=60 iops : min= 62, max= 153, avg=82.98, stdev=12.16, samples=60 lat (msec) : 50=0.62%, 100=1.88%, 250=95.20%, 500=2.30% cpu : usr=0.12%, sys=1.52%, ctx=2887, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=161288: Tue Aug 18 10:18:12 2026 write: IOPS=83, BW=334KiB/s (342kB/s)(10.0MiB/30658msec); 0 zone resets slat (usec): min=45, max=837, avg=237.88, stdev=100.98 clat (msec): min=21, max=297, avg=191.18, stdev=33.45 lat (msec): min=22, max=297, avg=191.42, stdev=33.45 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 138], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 218], 90.00th=[ 230], 95.00th=[ 239], | 99.00th=[ 266], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 256, max= 625, per=24.88%, avg=331.88, stdev=50.75, samples=60 iops : min= 64, max= 156, avg=82.97, stdev=12.66, samples=60 lat (msec) : 50=0.39%, 100=1.80%, 250=95.78%, 500=2.03% cpu : usr=0.16%, sys=1.61%, ctx=2998, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=161289: Tue Aug 18 10:18:12 2026 write: IOPS=83, BW=334KiB/s (342kB/s)(10.0MiB/30690msec); 0 zone resets slat (usec): min=41, max=819, avg=233.15, stdev=98.45 clat (msec): min=46, max=300, avg=191.35, stdev=32.67 lat (msec): min=46, max=301, avg=191.59, stdev=32.67 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 134], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 207], 80.00th=[ 218], 90.00th=[ 230], 95.00th=[ 241], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 256, max= 580, per=24.84%, avg=331.34, stdev=44.82, samples=61 iops : min= 64, max= 145, avg=82.84, stdev=11.21, samples=61 lat (msec) : 50=0.04%, 100=1.91%, 250=95.66%, 500=2.38% cpu : usr=0.11%, sys=1.63%, ctx=2877, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=161290: Tue Aug 18 10:18:12 2026 write: IOPS=83, BW=334KiB/s (342kB/s)(10.0MiB/30690msec); 0 zone resets slat (usec): min=52, max=1074, avg=235.59, stdev=104.97 clat (msec): min=61, max=297, avg=191.38, stdev=32.33 lat (msec): min=61, max=297, avg=191.62, stdev=32.33 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 178], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 207], 80.00th=[ 218], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 256, max= 574, per=24.84%, avg=331.38, stdev=46.41, samples=61 iops : min= 64, max= 143, avg=82.84, stdev=11.56, samples=61 lat (msec) : 100=1.91%, 250=95.74%, 500=2.34% cpu : usr=0.14%, sys=1.60%, ctx=2965, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1335KiB/s (1367kB/s), 334KiB/s-334KiB/s (342kB/s-342kB/s), io=40.0MiB (41.9MB), run=30658-30690msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=161304: Tue Aug 18 10:18:35 2026 read: IOPS=63, BW=256KiB/s (262kB/s)(5048KiB/19733msec) slat (usec): min=40, max=6337, avg=2545.35, stdev=838.57 clat (msec): min=3, max=209, avg=77.44, stdev=14.61 lat (msec): min=6, max=212, avg=79.98, stdev=14.71 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 81], | 70.00th=[ 84], 80.00th=[ 88], 90.00th=[ 92], 95.00th=[ 99], | 99.00th=[ 110], 99.50th=[ 115], 99.90th=[ 180], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 176, max= 344, per=25.36%, avg=256.13, stdev=43.86, samples=39 iops : min= 44, max= 86, avg=64.03, stdev=10.97, samples=39 write: IOPS=65, BW=263KiB/s (269kB/s)(5192KiB/19733msec); 0 zone resets slat (usec): min=42, max=923, avg=196.54, stdev=97.91 clat (msec): min=26, max=301, avg=165.06, stdev=40.11 lat (msec): min=26, max=301, avg=165.26, stdev=40.11 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 103], 10.00th=[ 114], 20.00th=[ 130], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 215], 95.00th=[ 234], | 99.00th=[ 264], 99.50th=[ 268], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 184, max= 368, per=24.94%, avg=262.36, stdev=40.09, samples=39 iops : min= 46, max= 92, avg=65.59, stdev=10.02, samples=39 lat (msec) : 4=0.04%, 10=0.08%, 20=0.16%, 50=0.98%, 100=48.09% lat (msec) : 250=49.45%, 500=1.21% cpu : usr=0.19%, sys=2.41%, ctx=4747, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=161305: Tue Aug 18 10:18:35 2026 read: IOPS=62, BW=251KiB/s (257kB/s)(4976KiB/19857msec) slat (usec): min=1138, max=5827, avg=2560.03, stdev=784.14 clat (msec): min=4, max=175, avg=78.01, stdev=14.24 lat (msec): min=6, max=177, avg=80.58, stdev=14.24 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 89], 90.00th=[ 93], 95.00th=[ 99], | 99.00th=[ 108], 99.50th=[ 134], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 128, max= 384, per=24.80%, avg=250.46, stdev=53.13, samples=39 iops : min= 32, max= 96, avg=62.62, stdev=13.28, samples=39 write: IOPS=66, BW=265KiB/s (271kB/s)(5264KiB/19857msec); 0 zone resets slat (usec): min=59, max=689, avg=193.67, stdev=87.89 clat (msec): min=34, max=278, avg=164.70, stdev=39.05 lat (msec): min=34, max=278, avg=164.89, stdev=39.05 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 234], | 99.00th=[ 257], 99.50th=[ 264], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 168, max= 368, per=25.20%, avg=265.10, stdev=42.46, samples=39 iops : min= 42, max= 92, avg=66.26, stdev=10.61, samples=39 lat (msec) : 10=0.12%, 20=0.16%, 50=0.55%, 100=47.66%, 250=50.66% lat (msec) : 500=0.86% cpu : usr=0.23%, sys=2.36%, ctx=4769, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=161306: Tue Aug 18 10:18:35 2026 read: IOPS=62, BW=251KiB/s (257kB/s)(4984KiB/19857msec) slat (usec): min=1171, max=5724, avg=2527.09, stdev=774.10 clat (msec): min=6, max=137, avg=77.57, stdev=13.23 lat (msec): min=9, max=139, avg=80.10, stdev=13.28 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 88], 90.00th=[ 92], 95.00th=[ 97], | 99.00th=[ 109], 99.50th=[ 112], 99.90th=[ 131], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 160, max= 384, per=24.90%, avg=251.54, stdev=52.67, samples=39 iops : min= 40, max= 96, avg=62.87, stdev=13.17, samples=39 write: IOPS=66, BW=265KiB/s (271kB/s)(5256KiB/19857msec); 0 zone resets slat (usec): min=64, max=3478, avg=198.52, stdev=133.65 clat (msec): min=52, max=274, avg=165.17, stdev=37.77 lat (msec): min=52, max=274, avg=165.37, stdev=37.78 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 107], 10.00th=[ 117], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 236], | 99.00th=[ 257], 99.50th=[ 266], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 192, max= 336, per=24.98%, avg=262.82, stdev=37.47, samples=39 iops : min= 48, max= 84, avg=65.69, stdev= 9.38, samples=39 lat (msec) : 10=0.16%, 20=0.16%, 50=0.43%, 100=47.93%, 250=50.35% lat (msec) : 500=0.98% cpu : usr=0.27%, sys=2.32%, ctx=4858, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=161307: Tue Aug 18 10:18:35 2026 read: IOPS=63, BW=255KiB/s (261kB/s)(5060KiB/19837msec) slat (usec): min=1122, max=6296, avg=2548.32, stdev=820.20 clat (msec): min=5, max=171, avg=77.99, stdev=13.43 lat (msec): min=9, max=172, avg=80.54, stdev=13.45 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 59], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 88], 90.00th=[ 94], 95.00th=[ 99], | 99.00th=[ 109], 99.50th=[ 114], 99.90th=[ 132], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 184, max= 384, per=25.38%, avg=256.38, stdev=48.44, samples=39 iops : min= 46, max= 96, avg=64.08, stdev=12.12, samples=39 write: IOPS=65, BW=261KiB/s (267kB/s)(5180KiB/19837msec); 0 zone resets slat (usec): min=65, max=742, avg=189.47, stdev=88.47 clat (msec): min=77, max=278, avg=165.70, stdev=38.60 lat (msec): min=77, max=278, avg=165.89, stdev=38.60 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 234], | 99.00th=[ 262], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 192, max= 328, per=24.75%, avg=260.41, stdev=35.03, samples=39 iops : min= 48, max= 82, avg=65.10, stdev= 8.76, samples=39 lat (msec) : 10=0.12%, 20=0.16%, 50=0.27%, 100=48.63%, 250=49.57% lat (msec) : 500=1.25% cpu : usr=0.18%, sys=2.38%, ctx=4883, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1011KiB/s (1035kB/s), 251KiB/s-256KiB/s (257kB/s-262kB/s), io=19.6MiB (20.5MB), run=19733-19857msec WRITE: bw=1052KiB/s (1077kB/s), 261KiB/s-265KiB/s (267kB/s-271kB/s), io=20.4MiB (21.4MB), run=19733-19857msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=161313: Tue Aug 18 10:18:39 2026 read: IOPS=8, BW=357MiB/s (374MB/s)(40.0MiB/112msec) slat (nsec): min=10148k, max=10148k, avg=10148239.00, stdev= 0.00 clat (nsec): min=97022k, max=97022k, avg=97022015.00, stdev= 0.00 lat (nsec): min=107174k, max=107174k, avg=107174061.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[96994], 5.00th=[96994], 10.00th=[96994], 20.00th=[96994], | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994], | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.21%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=40.0MiB (41.9MB), run=112-112msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout