== sanity test 398c: run fio to test AIO ================= 13:13:25 (1786036405) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.61136 s, 68.6 MB/s osc.lustre-OST0000-osc-ffff9dc2a6647800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145020: Thu Aug 6 13:16:11 2026 write: IOPS=16, BW=67.9KiB/s (69.6kB/s)(10.0MiB/150742msec); 0 zone resets slat (usec): min=37, max=17609, avg=294.91, stdev=957.98 clat (msec): min=243, max=1873, avg=941.30, stdev=258.10 lat (msec): min=249, max=1875, avg=941.60, stdev=258.12 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 969], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1452], | 99.00th=[ 1737], 99.50th=[ 1770], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 216, per=25.13%, avg=68.11, stdev=33.87, samples=294 iops : min= 1, max= 54, avg=16.49, stdev= 8.47, samples=294 lat (msec) : 250=0.16%, 500=3.55%, 750=14.69%, 1000=47.73%, 2000=33.87% cpu : usr=0.05%, sys=0.29%, ctx=1666, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145021: Thu Aug 6 13:16:11 2026 write: IOPS=16, BW=67.8KiB/s (69.5kB/s)(10.0MiB/150976msec); 0 zone resets slat (usec): min=38, max=24399, avg=335.65, stdev=1128.83 clat (msec): min=345, max=1962, avg=941.80, stdev=253.15 lat (msec): min=345, max=1962, avg=942.14, stdev=253.18 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 818], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 1703], 99.50th=[ 1754], 99.90th=[ 1888], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 128, per=25.30%, avg=68.55, stdev=32.66, samples=293 iops : min= 1, max= 32, avg=16.59, stdev= 8.18, samples=293 lat (msec) : 500=3.24%, 750=15.43%, 1000=48.28%, 2000=33.05% cpu : usr=0.05%, sys=0.32%, ctx=1772, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145022: Thu Aug 6 13:16:11 2026 write: IOPS=16, BW=67.0KiB/s (69.6kB/s)(10.0MiB/150675msec); 0 zone resets slat (usec): min=38, max=25205, avg=364.18, stdev=1193.75 clat (msec): min=217, max=1960, avg=940.10, stdev=256.72 lat (msec): min=218, max=1960, avg=940.47, stdev=256.74 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 550], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 961], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1418], | 99.00th=[ 1754], 99.50th=[ 1804], 99.90th=[ 1905], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 198, per=25.03%, avg=67.83, stdev=27.21, samples=296 iops : min= 1, max= 49, avg=16.41, stdev= 6.85, samples=296 lat (msec) : 250=0.08%, 500=3.59%, 750=13.98%, 1000=49.18%, 2000=33.16% cpu : usr=0.07%, sys=0.38%, ctx=2232, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145023: Thu Aug 6 13:16:11 2026 write: IOPS=16, BW=67.8KiB/s (69.5kB/s)(10.0MiB/150977msec); 0 zone resets slat (usec): min=39, max=30405, avg=362.74, stdev=1310.22 clat (msec): min=299, max=1834, avg=941.77, stdev=251.66 lat (msec): min=300, max=1834, avg=942.14, stdev=251.71 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1250], 95.00th=[ 1418], | 99.00th=[ 1720], 99.50th=[ 1737], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 128, per=24.85%, avg=67.35, stdev=27.34, samples=298 iops : min= 1, max= 32, avg=16.30, stdev= 6.84, samples=298 lat (msec) : 500=2.93%, 750=16.25%, 1000=47.42%, 2000=33.40% cpu : usr=0.06%, sys=0.38%, ctx=2344, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=271KiB/s (278kB/s), 67.8KiB/s-67.0KiB/s (69.5kB/s-69.6kB/s), io=40.0MiB (41.9MB), run=150675-150977msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145055: Thu Aug 6 13:18:18 2026 read: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5048KiB/115932msec) slat (usec): min=86, max=63421, avg=16661.99, stdev=9049.24 clat (msec): min=14, max=1249, avg=454.34, stdev=136.15 lat (msec): min=21, max=1261, avg=471.01, stdev=137.74 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 268], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 667], | 99.00th=[ 877], 99.50th=[ 953], 99.90th=[ 1167], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 104, per=25.40%, avg=43.44, stdev=20.88, samples=227 iops : min= 1, max= 26, avg=10.28, stdev= 5.24, samples=227 write: IOPS=11, BW=44.8KiB/s (45.9kB/s)(5192KiB/115932msec); 0 zone resets slat (usec): min=47, max=19751, avg=425.65, stdev=1108.68 clat (msec): min=149, max=2298, avg=969.68, stdev=305.30 lat (msec): min=150, max=2298, avg=970.11, stdev=305.30 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 718], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1888], 99.50th=[ 2232], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=44.39, stdev=19.70, samples=228 iops : min= 1, max= 25, avg=10.52, stdev= 4.94, samples=228 lat (msec) : 20=0.08%, 50=0.20%, 100=0.20%, 250=1.48%, 500=33.67% lat (msec) : 750=24.06%, 1000=18.12%, 2000=21.84%, >=2000=0.35% cpu : usr=0.10%, sys=0.73%, ctx=4667, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145056: Thu Aug 6 13:18:18 2026 read: IOPS=10, BW=42.6KiB/s (43.7kB/s)(4976KiB/116721msec) slat (usec): min=2913, max=77316, avg=16645.68, stdev=8614.17 clat (msec): min=8, max=962, avg=454.41, stdev=128.96 lat (msec): min=30, max=999, avg=471.07, stdev=130.20 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 844], 99.50th=[ 877], 99.90th=[ 944], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 119, per=24.63%, avg=42.12, stdev=19.74, samples=230 iops : min= 1, max= 29, avg= 9.95, stdev= 4.96, samples=230 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5264KiB/116721msec); 0 zone resets slat (usec): min=52, max=24273, avg=462.07, stdev=1356.17 clat (msec): min=199, max=2299, avg=970.20, stdev=299.15 lat (msec): min=200, max=2299, avg=970.67, stdev=299.12 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 518], 10.00th=[ 609], 20.00th=[ 718], | 30.00th=[ 802], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 1888], 99.50th=[ 1938], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 103, per=25.29%, avg=45.01, stdev=20.87, samples=229 iops : min= 1, max= 25, avg=10.67, stdev= 5.22, samples=229 lat (msec) : 10=0.04%, 50=0.08%, 100=0.31%, 250=0.94%, 500=33.79% lat (msec) : 750=24.69%, 1000=18.63%, 2000=21.29%, >=2000=0.23% cpu : usr=0.12%, sys=0.83%, ctx=5080, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145057: Thu Aug 6 13:18:18 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(4984KiB/116692msec) slat (usec): min=3629, max=63145, avg=16292.90, stdev=8318.57 clat (msec): min=19, max=970, avg=445.54, stdev=126.73 lat (msec): min=28, max=999, avg=461.84, stdev=128.26 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 435], 60.00th=[ 456], | 70.00th=[ 493], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 659], | 99.00th=[ 810], 99.50th=[ 902], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 152, per=25.03%, avg=42.80, stdev=21.85, samples=228 iops : min= 1, max= 38, avg=10.12, stdev= 5.51, samples=228 write: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5256KiB/116692msec); 0 zone resets slat (usec): min=42, max=18958, avg=447.40, stdev=1234.79 clat (msec): min=254, max=2289, avg=978.83, stdev=297.86 lat (msec): min=255, max=2290, avg=979.28, stdev=297.86 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 518], 10.00th=[ 617], 20.00th=[ 726], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1888], 99.50th=[ 2022], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 103, per=25.21%, avg=44.88, stdev=19.44, samples=228 iops : min= 1, max= 25, avg=10.64, stdev= 4.86, samples=228 lat (msec) : 20=0.04%, 50=0.16%, 100=0.35%, 250=1.13%, 500=35.23% lat (msec) : 750=22.03%, 1000=18.95%, 2000=21.84%, >=2000=0.27% cpu : usr=0.11%, sys=0.76%, ctx=4656, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145058: Thu Aug 6 13:18:18 2026 read: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5060KiB/116366msec) slat (usec): min=3482, max=66089, avg=17009.45, stdev=8961.11 clat (msec): min=34, max=1002, avg=450.88, stdev=128.30 lat (msec): min=56, max=1018, avg=467.91, stdev=129.73 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 684], | 99.00th=[ 860], 99.50th=[ 911], 99.90th=[ 986], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 103, per=25.50%, avg=43.61, stdev=20.02, samples=227 iops : min= 1, max= 25, avg=10.33, stdev= 5.00, samples=227 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5180KiB/116366msec); 0 zone resets slat (usec): min=37, max=23480, avg=442.94, stdev=1390.17 clat (msec): min=206, max=2270, avg=977.25, stdev=294.91 lat (msec): min=207, max=2272, avg=977.69, stdev=295.01 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1871], 99.50th=[ 1938], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 103, per=24.77%, avg=44.10, stdev=19.58, samples=229 iops : min= 1, max= 25, avg=10.45, stdev= 4.88, samples=229 lat (msec) : 50=0.08%, 100=0.27%, 250=1.29%, 500=34.61%, 750=23.32% lat (msec) : 1000=18.55%, 2000=21.76%, >=2000=0.12% cpu : usr=0.10%, sys=0.78%, ctx=4753, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.6KiB/s-43.5KiB/s (43.7kB/s-44.6kB/s), io=19.6MiB (20.5MB), run=115932-116721msec WRITE: bw=179KiB/s (183kB/s), 44.5KiB/s-45.1KiB/s (45.6kB/s-46.2kB/s), io=20.4MiB (21.4MB), run=115932-116721msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145080: Thu Aug 6 13:18:28 2026 read: IOPS=1, BW=64.7MiB/s (67.9MB/s)(40.0MiB/618msec) slat (nsec): min=45506k, max=45506k, avg=45506082.00, stdev= 0.00 clat (nsec): min=542428k, max=542428k, avg=542427733.00, stdev= 0.00 lat (nsec): min=587938k, max=587938k, avg=587938273.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.94%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.7MiB/s (67.9MB/s), 64.7MiB/s-64.7MiB/s (67.9MB/s-67.9MB/s), io=40.0MiB (41.9MB), run=618-618msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout