== sanity test 398c: run fio to test AIO ================= 22:19:38 (1786760378) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.508966 s, 82.4 MB/s osc.lustre-OST0000-osc-ffff88db054dd800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145032: Fri Aug 14 22:22:01 2026 write: IOPS=19, BW=78.3KiB/s (80.1kB/s)(10.0MiB/130858msec); 0 zone resets slat (usec): min=35, max=19894, avg=298.85, stdev=844.44 clat (msec): min=132, max=1822, avg=816.41, stdev=252.64 lat (msec): min=132, max=1826, avg=816.71, stdev=252.66 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 414], 10.00th=[ 542], 20.00th=[ 634], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 911], 80.00th=[ 1011], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1519], 99.50th=[ 1670], 99.90th=[ 1770], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 256, per=25.17%, avg=78.54, stdev=34.32, samples=257 iops : min= 1, max= 64, avg=19.21, stdev= 8.60, samples=257 lat (msec) : 250=0.62%, 500=7.42%, 750=33.79%, 1000=36.91%, 2000=21.25% cpu : usr=0.06%, sys=0.33%, ctx=2454, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145033: Fri Aug 14 22:22:01 2026 write: IOPS=19, BW=78.1KiB/s (79.0kB/s)(10.0MiB/131081msec); 0 zone resets slat (usec): min=37, max=23724, avg=281.74, stdev=939.61 clat (msec): min=120, max=1917, avg=818.11, stdev=245.20 lat (msec): min=120, max=1917, avg=818.40, stdev=245.18 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 422], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 911], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1536], 99.50th=[ 1603], 99.90th=[ 1770], 99.95th=[ 1854], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 247, per=24.78%, avg=77.32, stdev=34.40, samples=261 iops : min= 1, max= 61, avg=18.91, stdev= 8.61, samples=261 lat (msec) : 250=0.98%, 500=6.21%, 750=34.84%, 1000=36.95%, 2000=21.02% cpu : usr=0.05%, sys=0.32%, ctx=2524, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145034: Fri Aug 14 22:22:01 2026 write: IOPS=19, BW=78.4KiB/s (80.3kB/s)(10.0MiB/130595msec); 0 zone resets slat (usec): min=35, max=10949, avg=294.32, stdev=686.92 clat (msec): min=70, max=1902, avg=815.44, stdev=251.66 lat (msec): min=71, max=1903, avg=815.74, stdev=251.62 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 409], 10.00th=[ 542], 20.00th=[ 634], | 30.00th=[ 684], 40.00th=[ 735], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 919], 80.00th=[ 1020], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1569], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 263, per=24.96%, avg=77.86, stdev=35.21, samples=259 iops : min= 1, max= 65, avg=19.05, stdev= 8.82, samples=259 lat (msec) : 100=0.08%, 250=1.02%, 500=6.84%, 750=35.74%, 1000=35.08% lat (msec) : 2000=21.25% cpu : usr=0.06%, sys=0.36%, ctx=2788, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145035: Fri Aug 14 22:22:01 2026 write: IOPS=19, BW=78.1KiB/s (79.0kB/s)(10.0MiB/131078msec); 0 zone resets slat (usec): min=35, max=20355, avg=285.30, stdev=916.10 clat (msec): min=119, max=1909, avg=817.77, stdev=249.47 lat (msec): min=119, max=1909, avg=818.05, stdev=249.45 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 422], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 911], 80.00th=[ 995], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1536], 99.50th=[ 1720], 99.90th=[ 1854], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 231, per=25.07%, avg=78.22, stdev=36.67, samples=258 iops : min= 1, max= 57, avg=19.14, stdev= 9.20, samples=258 lat (msec) : 250=0.86%, 500=6.17%, 750=34.34%, 1000=38.67%, 2000=19.96% cpu : usr=0.05%, sys=0.33%, ctx=2256, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=312KiB/s (320kB/s), 78.1KiB/s-78.4KiB/s (79.0kB/s-80.3kB/s), io=40.0MiB (41.9MB), run=130595-131081msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145064: Fri Aug 14 22:24:00 2026 read: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5048KiB/111822msec) slat (usec): min=2508, max=84648, avg=15090.58, stdev=9240.09 clat (msec): min=12, max=1713, avg=440.20, stdev=179.00 lat (msec): min=27, max=1745, avg=455.30, stdev=182.07 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 226], 10.00th=[ 268], 20.00th=[ 313], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 481], 80.00th=[ 542], 90.00th=[ 651], 95.00th=[ 776], | 99.00th=[ 1083], 99.50th=[ 1284], 99.90th=[ 1519], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 135, per=25.19%, avg=45.09, stdev=23.18, samples=219 iops : min= 1, max= 33, avg=10.72, stdev= 5.83, samples=219 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(5192KiB/111822msec); 0 zone resets slat (usec): min=42, max=22991, avg=408.26, stdev=1186.18 clat (msec): min=166, max=2399, avg=932.56, stdev=362.45 lat (msec): min=166, max=2399, avg=932.97, stdev=362.48 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 510], 10.00th=[ 550], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 776], 50.00th=[ 844], 60.00th=[ 944], | 70.00th=[ 1036], 80.00th=[ 1183], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 2165], 99.50th=[ 2299], 99.90th=[ 2333], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 103, per=24.88%, avg=46.28, stdev=22.53, samples=219 iops : min= 1, max= 25, avg=11.01, stdev= 5.66, samples=219 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=3.75%, 500=35.12% lat (msec) : 750=25.90%, 1000=16.95%, 2000=17.19%, >=2000=0.86% cpu : usr=0.11%, sys=0.71%, ctx=4722, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145065: Fri Aug 14 22:24:00 2026 read: IOPS=11, BW=44.5KiB/s (45.6kB/s)(4976KiB/111785msec) slat (usec): min=3065, max=66447, avg=15049.68, stdev=9195.99 clat (msec): min=9, max=1739, avg=441.44, stdev=189.87 lat (msec): min=17, max=1775, avg=456.50, stdev=192.77 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 228], 10.00th=[ 271], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 435], | 70.00th=[ 468], 80.00th=[ 531], 90.00th=[ 659], 95.00th=[ 810], | 99.00th=[ 1234], 99.50th=[ 1351], 99.90th=[ 1536], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 112, per=25.39%, avg=45.44, stdev=24.14, samples=214 iops : min= 1, max= 28, avg=10.81, stdev= 6.09, samples=214 write: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5264KiB/111785msec); 0 zone resets slat (usec): min=42, max=19103, avg=350.32, stdev=984.98 clat (msec): min=252, max=2502, avg=924.11, stdev=360.28 lat (msec): min=252, max=2502, avg=924.46, stdev=360.28 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 493], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 709], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 927], | 70.00th=[ 1036], 80.00th=[ 1167], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 2072], 99.50th=[ 2333], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 120, per=25.51%, avg=47.46, stdev=22.67, samples=217 iops : min= 1, max= 30, avg=11.30, stdev= 5.71, samples=217 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.23%, 250=2.93% lat (msec) : 500=36.33%, 750=24.77%, 1000=17.70%, 2000=17.15%, >=2000=0.70% cpu : usr=0.10%, sys=0.67%, ctx=4504, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145066: Fri Aug 14 22:24:00 2026 read: IOPS=11, BW=44.6KiB/s (45.7kB/s)(4984KiB/111710msec) slat (usec): min=85, max=70283, avg=15164.51, stdev=9652.36 clat (msec): min=8, max=1734, avg=435.26, stdev=189.46 lat (msec): min=16, max=1748, avg=450.44, stdev=192.40 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 218], 10.00th=[ 259], 20.00th=[ 309], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 472], 80.00th=[ 535], 90.00th=[ 642], 95.00th=[ 760], | 99.00th=[ 1133], 99.50th=[ 1401], 99.90th=[ 1720], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 136, per=25.55%, avg=45.73, stdev=24.36, samples=214 iops : min= 1, max= 34, avg=10.88, stdev= 6.13, samples=214 write: IOPS=11, BW=47.0KiB/s (48.2kB/s)(5256KiB/111710msec); 0 zone resets slat (usec): min=38, max=15813, avg=433.22, stdev=1337.87 clat (msec): min=127, max=2502, avg=931.14, stdev=365.22 lat (msec): min=127, max=2502, avg=931.58, stdev=365.26 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 485], 10.00th=[ 542], 20.00th=[ 642], | 30.00th=[ 718], 40.00th=[ 776], 50.00th=[ 852], 60.00th=[ 944], | 70.00th=[ 1053], 80.00th=[ 1167], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 2140], 99.50th=[ 2265], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=25.21%, avg=46.89, stdev=23.19, samples=218 iops : min= 1, max= 32, avg=11.17, stdev= 5.84, samples=218 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.27%, 250=3.67% lat (msec) : 500=35.43%, 750=25.04%, 1000=16.91%, 2000=17.27%, >=2000=1.13% cpu : usr=0.09%, sys=0.73%, ctx=4869, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145067: Fri Aug 14 22:24:00 2026 read: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5060KiB/111699msec) slat (usec): min=2919, max=71726, avg=15141.54, stdev=9310.56 clat (msec): min=13, max=1744, avg=441.18, stdev=195.23 lat (msec): min=21, max=1755, avg=456.33, stdev=198.39 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 222], 10.00th=[ 264], 20.00th=[ 313], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 430], | 70.00th=[ 477], 80.00th=[ 542], 90.00th=[ 676], 95.00th=[ 844], | 99.00th=[ 1116], 99.50th=[ 1301], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 119, per=25.53%, avg=45.71, stdev=22.81, samples=217 iops : min= 1, max= 29, avg=10.87, stdev= 5.75, samples=217 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(5180KiB/111699msec); 0 zone resets slat (usec): min=45, max=22573, avg=385.90, stdev=1289.93 clat (msec): min=251, max=2503, avg=930.32, stdev=357.65 lat (msec): min=251, max=2503, avg=930.71, stdev=357.74 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 485], 10.00th=[ 558], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 776], 50.00th=[ 852], 60.00th=[ 944], | 70.00th=[ 1053], 80.00th=[ 1167], 90.00th=[ 1418], 95.00th=[ 1620], | 99.00th=[ 2165], 99.50th=[ 2299], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=24.92%, avg=46.35, stdev=22.94, samples=218 iops : min= 1, max= 32, avg=11.03, stdev= 5.77, samples=218 lat (msec) : 20=0.12%, 50=0.20%, 100=0.20%, 250=3.40%, 500=35.55% lat (msec) : 750=25.20%, 1000=16.48%, 2000=18.01%, >=2000=0.86% cpu : usr=0.12%, sys=0.73%, ctx=5022, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=179KiB/s (184kB/s), 44.5KiB/s-45.3KiB/s (45.6kB/s-46.4kB/s), io=19.6MiB (20.5MB), run=111699-111822msec WRITE: bw=187KiB/s (191kB/s), 46.4KiB/s-47.1KiB/s (47.5kB/s-48.2kB/s), io=20.4MiB (21.4MB), run=111699-111822msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145087: Fri Aug 14 22:24:12 2026 read: IOPS=1, BW=59.2MiB/s (62.0MB/s)(40.0MiB/676msec) slat (nsec): min=19065k, max=19065k, avg=19064561.00, stdev= 0.00 clat (nsec): min=638540k, max=638540k, avg=638539914.00, stdev= 0.00 lat (nsec): min=657634k, max=657634k, avg=657633597.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.33%, ctx=44, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=59.2MiB/s (62.0MB/s), 59.2MiB/s-59.2MiB/s (62.0MB/s-62.0MB/s), io=40.0MiB (41.9MB), run=676-676msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout