== sanity test 398c: run fio to test AIO ================= 18:14:54 (1787609694) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.544413 s, 77.0 MB/s osc.lustre-OST0000-osc-ffff8e2d82b27800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=175815: Mon Aug 24 18:18:02 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177028msec); 0 zone resets slat (usec): min=29, max=17037, avg=288.92, stdev=933.11 clat (msec): min=320, max=2608, avg=1105.06, stdev=302.82 lat (msec): min=320, max=2608, avg=1105.36, stdev=302.84 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 684], 10.00th=[ 776], 20.00th=[ 869], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 2056], 99.50th=[ 2232], 99.90th=[ 2400], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 128, per=24.75%, avg=57.18, stdev=24.77, samples=351 iops : min= 1, max= 32, avg=13.73, stdev= 6.24, samples=351 lat (msec) : 500=1.29%, 750=6.33%, 1000=30.70%, 2000=60.51%, >=2000=1.17% cpu : usr=0.05%, sys=0.25%, ctx=1799, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175816: Mon Aug 24 18:18:02 2026 write: IOPS=14, BW=57.0KiB/s (59.4kB/s)(10.0MiB/176650msec); 0 zone resets slat (usec): min=29, max=24786, avg=356.90, stdev=1362.64 clat (msec): min=228, max=2498, avg=1103.22, stdev=310.83 lat (msec): min=228, max=2498, avg=1103.58, stdev=310.85 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 642], 10.00th=[ 760], 20.00th=[ 869], | 30.00th=[ 944], 40.00th=[ 1003], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1670], | 99.00th=[ 2140], 99.50th=[ 2232], 99.90th=[ 2400], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=25.86%, avg=59.75, stdev=30.26, samples=336 iops : min= 1, max= 32, avg=14.36, stdev= 7.63, samples=336 lat (msec) : 250=0.62%, 500=0.08%, 750=8.05%, 1000=30.04%, 2000=59.65% lat (msec) : >=2000=1.56% cpu : usr=0.05%, sys=0.25%, ctx=1680, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175817: Mon Aug 24 18:18:02 2026 write: IOPS=14, BW=57.9KiB/s (59.2kB/s)(10.0MiB/176992msec); 0 zone resets slat (usec): min=29, max=15016, avg=310.71, stdev=1033.71 clat (msec): min=233, max=2532, avg=1104.91, stdev=302.52 lat (msec): min=234, max=2532, avg=1105.23, stdev=302.51 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 667], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1083], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1485], 95.00th=[ 1670], | 99.00th=[ 1989], 99.50th=[ 2165], 99.90th=[ 2299], 99.95th=[ 2500], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 128, per=25.71%, avg=59.39, stdev=33.07, samples=338 iops : min= 1, max= 32, avg=14.29, stdev= 8.28, samples=338 lat (msec) : 250=0.62%, 500=0.70%, 750=7.73%, 1000=29.77%, 2000=60.31% lat (msec) : >=2000=0.86% cpu : usr=0.05%, sys=0.24%, ctx=1579, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=175818: Mon Aug 24 18:18:02 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177029msec); 0 zone resets slat (usec): min=30, max=16404, avg=297.38, stdev=981.53 clat (msec): min=160, max=2532, avg=1104.98, stdev=305.90 lat (msec): min=160, max=2532, avg=1105.28, stdev=305.89 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 667], 10.00th=[ 776], 20.00th=[ 869], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1070], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 2089], 99.50th=[ 2265], 99.90th=[ 2299], 99.95th=[ 2500], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 128, per=25.28%, avg=58.39, stdev=27.41, samples=344 iops : min= 1, max= 32, avg=14.04, stdev= 6.89, samples=344 lat (msec) : 250=0.08%, 500=1.52%, 750=7.23%, 1000=30.08%, 2000=59.73% lat (msec) : >=2000=1.37% cpu : usr=0.05%, sys=0.24%, ctx=1708, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=231KiB/s (237kB/s), 57.8KiB/s-57.0KiB/s (59.2kB/s-59.4kB/s), io=40.0MiB (41.9MB), run=176650-177029msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=175852: Mon Aug 24 18:20:25 2026 read: IOPS=9, BW=36.9KiB/s (37.8kB/s)(5048KiB/136842msec) slat (usec): min=40, max=92388, avg=18684.39, stdev=11456.18 clat (msec): min=37, max=1506, avg=535.91, stdev=168.28 lat (msec): min=49, max=1517, avg=554.61, stdev=170.65 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 321], 10.00th=[ 359], 20.00th=[ 405], | 30.00th=[ 439], 40.00th=[ 481], 50.00th=[ 510], 60.00th=[ 550], | 70.00th=[ 600], 80.00th=[ 651], 90.00th=[ 751], 95.00th=[ 835], | 99.00th=[ 1116], 99.50th=[ 1150], 99.90th=[ 1234], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 103, per=25.74%, avg=37.58, stdev=20.36, samples=261 iops : min= 1, max= 25, avg= 8.79, stdev= 5.13, samples=261 write: IOPS=9, BW=37.9KiB/s (38.9kB/s)(5192KiB/136842msec); 0 zone resets slat (usec): min=26, max=51153, avg=483.36, stdev=2066.63 clat (msec): min=100, max=3050, avg=1146.08, stdev=374.39 lat (msec): min=100, max=3050, avg=1146.57, stdev=374.55 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 667], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1368], 90.00th=[ 1653], 95.00th=[ 1838], | 99.00th=[ 2500], 99.50th=[ 2601], 99.90th=[ 2802], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 96, per=25.12%, avg=38.18, stdev=19.50, samples=264 iops : min= 1, max= 24, avg= 8.93, stdev= 4.89, samples=264 lat (msec) : 50=0.04%, 100=0.04%, 250=0.98%, 500=22.93%, 750=25.31% lat (msec) : 1000=18.05%, 2000=31.13%, >=2000=1.52% cpu : usr=0.11%, sys=0.74%, ctx=4365, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175853: Mon Aug 24 18:20:25 2026 read: IOPS=9, BW=36.2KiB/s (37.1kB/s)(4976KiB/137399msec) slat (usec): min=3164, max=88511, avg=18585.20, stdev=11197.56 clat (msec): min=11, max=1353, avg=541.01, stdev=164.64 lat (msec): min=21, max=1377, avg=559.61, stdev=167.05 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 326], 10.00th=[ 368], 20.00th=[ 409], | 30.00th=[ 443], 40.00th=[ 485], 50.00th=[ 518], 60.00th=[ 558], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 743], 95.00th=[ 827], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1318], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 119, per=25.41%, avg=37.10, stdev=18.95, samples=260 iops : min= 1, max= 29, avg= 8.67, stdev= 4.75, samples=260 write: IOPS=9, BW=38.3KiB/s (39.2kB/s)(5264KiB/137399msec); 0 zone resets slat (usec): min=31, max=24179, avg=406.95, stdev=1248.25 clat (msec): min=211, max=2782, avg=1139.40, stdev=375.14 lat (msec): min=211, max=2782, avg=1139.81, stdev=375.14 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 860], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1636], 95.00th=[ 1888], | 99.00th=[ 2500], 99.50th=[ 2601], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 95, per=24.88%, avg=37.81, stdev=19.19, samples=271 iops : min= 1, max= 23, avg= 8.84, stdev= 4.83, samples=271 lat (msec) : 20=0.08%, 100=0.20%, 250=0.43%, 500=21.76%, 750=26.84% lat (msec) : 1000=17.97%, 2000=31.09%, >=2000=1.64% cpu : usr=0.11%, sys=0.60%, ctx=4130, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175854: Mon Aug 24 18:20:25 2026 read: IOPS=9, BW=36.3KiB/s (37.1kB/s)(4984KiB/137399msec) slat (usec): min=3753, max=79944, avg=19020.09, stdev=10875.60 clat (msec): min=14, max=1458, avg=544.71, stdev=166.79 lat (msec): min=27, max=1477, avg=563.74, stdev=168.93 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 321], 10.00th=[ 363], 20.00th=[ 418], | 30.00th=[ 456], 40.00th=[ 489], 50.00th=[ 523], 60.00th=[ 567], | 70.00th=[ 609], 80.00th=[ 659], 90.00th=[ 743], 95.00th=[ 844], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1435], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 95, per=24.79%, avg=36.19, stdev=18.07, samples=268 iops : min= 1, max= 23, avg= 8.44, stdev= 4.53, samples=268 write: IOPS=9, BW=38.3KiB/s (39.2kB/s)(5256KiB/137399msec); 0 zone resets slat (usec): min=35, max=37537, avg=448.68, stdev=1673.17 clat (msec): min=106, max=2997, avg=1136.09, stdev=365.18 lat (msec): min=108, max=2997, avg=1136.55, stdev=365.28 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 667], 10.00th=[ 751], 20.00th=[ 860], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1636], 95.00th=[ 1854], | 99.00th=[ 2433], 99.50th=[ 2534], 99.90th=[ 2802], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 96, per=25.03%, avg=38.05, stdev=18.45, samples=268 iops : min= 1, max= 24, avg= 8.90, stdev= 4.63, samples=268 lat (msec) : 20=0.08%, 50=0.04%, 250=0.59%, 500=21.05%, 750=27.62% lat (msec) : 1000=18.12%, 2000=30.66%, >=2000=1.84% cpu : usr=0.11%, sys=0.60%, ctx=4069, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=175855: Mon Aug 24 18:20:25 2026 read: IOPS=9, BW=37.0KiB/s (37.9kB/s)(5060KiB/136745msec) slat (usec): min=3016, max=97026, avg=18856.95, stdev=11500.45 clat (msec): min=32, max=1473, avg=542.01, stdev=179.55 lat (msec): min=52, max=1486, avg=560.87, stdev=181.93 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 321], 10.00th=[ 351], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 481], 50.00th=[ 518], 60.00th=[ 558], | 70.00th=[ 600], 80.00th=[ 659], 90.00th=[ 751], 95.00th=[ 835], | 99.00th=[ 1234], 99.50th=[ 1435], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 95, per=25.83%, avg=37.71, stdev=19.26, samples=261 iops : min= 1, max= 23, avg= 8.82, stdev= 4.84, samples=261 write: IOPS=9, BW=37.9KiB/s (38.8kB/s)(5180KiB/136745msec); 0 zone resets slat (usec): min=34, max=36457, avg=438.53, stdev=1547.82 clat (msec): min=108, max=2967, avg=1138.92, stdev=369.39 lat (msec): min=108, max=2967, avg=1139.36, stdev=369.42 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 684], 10.00th=[ 760], 20.00th=[ 869], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1653], 95.00th=[ 1888], | 99.00th=[ 2433], 99.50th=[ 2500], 99.90th=[ 2836], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 96, per=25.17%, avg=38.25, stdev=18.44, samples=263 iops : min= 1, max= 24, avg= 8.95, stdev= 4.65, samples=263 lat (msec) : 50=0.04%, 100=0.16%, 250=0.59%, 500=22.50%, 750=25.82% lat (msec) : 1000=18.63%, 2000=30.59%, >=2000=1.68% cpu : usr=0.11%, sys=0.63%, ctx=4302, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=146KiB/s (150kB/s), 36.2KiB/s-37.0KiB/s (37.1kB/s-37.9kB/s), io=19.6MiB (20.5MB), run=136745-137399msec WRITE: bw=152KiB/s (156kB/s), 37.9KiB/s-38.3KiB/s (38.8kB/s-39.2kB/s), io=20.4MiB (21.4MB), run=136745-137399msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=175882: Mon Aug 24 18:20:35 2026 read: IOPS=1, BW=62.7MiB/s (65.7MB/s)(40.0MiB/638msec) slat (nsec): min=34933k, max=34933k, avg=34932572.00, stdev= 0.00 clat (nsec): min=573664k, max=573664k, avg=573664432.00, stdev= 0.00 lat (nsec): min=608604k, max=608604k, avg=608604097.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.04%, ctx=3, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.7MiB/s (65.7MB/s), 62.7MiB/s-62.7MiB/s (65.7MB/s-65.7MB/s), io=40.0MiB (41.9MB), run=638-638msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout