== sanity test 398c: run fio to test AIO ================= 15:01:10 (1779994870) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.536478 s, 78.2 MB/s osc.lustre-OST0000-osc-ffff88f104d20800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145875: Thu May 28 15:03:50 2026 write: IOPS=17, BW=68.6KiB/s (70.2kB/s)(10.0MiB/149297msec); 0 zone resets slat (usec): min=49, max=12256, avg=257.46, stdev=676.97 clat (msec): min=127, max=1556, avg=932.26, stdev=196.41 lat (msec): min=127, max=1556, avg=932.52, stdev=196.41 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 651], 10.00th=[ 709], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 969], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 135, per=24.92%, avg=68.04, stdev=25.29, samples=296 iops : min= 1, max= 33, avg=16.52, stdev= 6.35, samples=296 lat (msec) : 250=0.27%, 500=0.94%, 750=15.62%, 1000=48.63%, 2000=34.53% cpu : usr=0.05%, sys=0.28%, ctx=2293, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145876: Thu May 28 15:03:50 2026 write: IOPS=17, BW=68.5KiB/s (70.1kB/s)(10.0MiB/149482msec); 0 zone resets slat (usec): min=47, max=22905, avg=288.16, stdev=950.53 clat (msec): min=436, max=1565, avg=932.64, stdev=191.18 lat (msec): min=436, max=1569, avg=932.93, stdev=191.18 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 651], 10.00th=[ 709], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 128, per=26.37%, avg=72.00, stdev=28.93, samples=280 iops : min= 1, max= 32, avg=17.51, stdev= 7.24, samples=280 lat (msec) : 500=1.37%, 750=14.49%, 1000=49.49%, 2000=34.65% cpu : usr=0.04%, sys=0.29%, ctx=1928, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145877: Thu May 28 15:03:50 2026 write: IOPS=17, BW=68.5KiB/s (70.2kB/s)(10.0MiB/149432msec); 0 zone resets slat (usec): min=39, max=13391, avg=240.44, stdev=660.64 clat (msec): min=325, max=1549, avg=932.55, stdev=194.20 lat (msec): min=325, max=1549, avg=932.79, stdev=194.20 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 667], 10.00th=[ 718], 20.00th=[ 768], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 969], | 70.00th=[ 1028], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1418], 99.50th=[ 1485], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 135, per=25.98%, avg=70.93, stdev=33.17, samples=284 iops : min= 1, max= 33, avg=17.25, stdev= 8.33, samples=284 lat (msec) : 500=0.78%, 750=16.17%, 1000=48.98%, 2000=34.06% cpu : usr=0.04%, sys=0.26%, ctx=1594, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145878: Thu May 28 15:03:50 2026 write: IOPS=17, BW=68.5KiB/s (70.1kB/s)(10.0MiB/149528msec); 0 zone resets slat (usec): min=36, max=22367, avg=256.68, stdev=779.79 clat (msec): min=456, max=1554, avg=932.90, stdev=190.10 lat (msec): min=456, max=1554, avg=933.16, stdev=190.10 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 667], 10.00th=[ 718], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 953], | 70.00th=[ 1011], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1519], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 128, per=26.00%, avg=70.98, stdev=33.68, samples=284 iops : min= 1, max= 32, avg=17.25, stdev= 8.46, samples=284 lat (msec) : 500=0.39%, 750=15.12%, 1000=52.81%, 2000=31.68% cpu : usr=0.04%, sys=0.27%, ctx=1633, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=274KiB/s (281kB/s), 68.5KiB/s-68.6KiB/s (70.1kB/s-70.2kB/s), io=40.0MiB (41.9MB), run=149297-149528msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145910: Thu May 28 15:05:54 2026 read: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5048KiB/115086msec) slat (usec): min=56, max=88087, avg=15621.91, stdev=9467.10 clat (msec): min=13, max=1286, avg=451.25, stdev=138.70 lat (msec): min=19, max=1293, avg=466.88, stdev=140.17 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 266], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 405], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 852], 99.50th=[ 911], 99.90th=[ 1183], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7, max= 117, per=25.38%, avg=44.16, stdev=20.44, samples=224 iops : min= 1, max= 29, avg=10.56, stdev= 5.14, samples=224 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5192KiB/115086msec); 0 zone resets slat (usec): min=57, max=26946, avg=355.19, stdev=1194.74 clat (msec): min=120, max=2017, avg=963.15, stdev=300.72 lat (msec): min=124, max=2017, avg=963.50, stdev=300.77 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 802], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1217], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1770], 99.50th=[ 1921], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 103, per=24.61%, avg=44.55, stdev=20.05, samples=228 iops : min= 1, max= 25, avg=10.67, stdev= 5.03, samples=228 lat (msec) : 20=0.04%, 50=0.35%, 100=0.27%, 250=1.33%, 500=34.10% lat (msec) : 750=24.53%, 1000=19.14%, 2000=20.12%, >=2000=0.12% cpu : usr=0.08%, sys=0.59%, ctx=5005, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145911: Thu May 28 15:05:54 2026 read: IOPS=10, BW=43.2KiB/s (44.2kB/s)(4976KiB/115234msec) slat (usec): min=2854, max=75922, avg=15541.61, stdev=8763.83 clat (msec): min=21, max=895, avg=452.18, stdev=130.35 lat (msec): min=42, max=918, avg=467.72, stdev=132.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 827], 99.50th=[ 860], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 111, per=24.81%, avg=43.17, stdev=21.48, samples=226 iops : min= 1, max= 27, avg=10.32, stdev= 5.37, samples=226 write: IOPS=11, BW=45.7KiB/s (46.8kB/s)(5264KiB/115234msec); 0 zone resets slat (usec): min=52, max=15472, avg=336.55, stdev=867.71 clat (msec): min=313, max=1959, avg=956.44, stdev=294.89 lat (msec): min=314, max=1959, avg=956.77, stdev=294.88 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 527], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 103, per=25.19%, avg=45.60, stdev=20.77, samples=226 iops : min= 1, max= 25, avg=10.93, stdev= 5.23, samples=226 lat (msec) : 50=0.16%, 100=0.23%, 250=1.09%, 500=33.95%, 750=25.62% lat (msec) : 1000=19.10%, 2000=19.84% cpu : usr=0.07%, sys=0.57%, ctx=4342, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145912: Thu May 28 15:05:54 2026 read: IOPS=10, BW=43.2KiB/s (44.3kB/s)(4984KiB/115241msec) slat (usec): min=2759, max=55573, avg=15110.53, stdev=8257.00 clat (msec): min=8, max=934, avg=446.21, stdev=130.20 lat (msec): min=21, max=955, avg=461.33, stdev=132.06 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 535], 90.00th=[ 617], 95.00th=[ 701], | 99.00th=[ 835], 99.50th=[ 885], 99.90th=[ 927], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 128, per=24.73%, avg=43.03, stdev=21.74, samples=228 iops : min= 1, max= 32, avg=10.29, stdev= 5.49, samples=228 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(5256KiB/115241msec); 0 zone resets slat (usec): min=57, max=14289, avg=366.90, stdev=968.70 clat (msec): min=319, max=1975, avg=962.70, stdev=295.46 lat (msec): min=327, max=1975, avg=963.07, stdev=295.43 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 550], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 1955], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 103, per=24.84%, avg=44.95, stdev=20.02, samples=228 iops : min= 1, max= 25, avg=10.77, stdev= 5.03, samples=228 lat (msec) : 10=0.04%, 50=0.16%, 100=0.23%, 250=0.98%, 500=35.00% lat (msec) : 750=24.14%, 1000=18.87%, 2000=20.59% cpu : usr=0.08%, sys=0.63%, ctx=4651, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145913: Thu May 28 15:05:54 2026 read: IOPS=10, BW=43.9KiB/s (44.0kB/s)(5060KiB/115239msec) slat (usec): min=2827, max=71409, avg=15685.47, stdev=9347.62 clat (msec): min=23, max=933, avg=455.37, stdev=126.67 lat (msec): min=35, max=948, avg=471.06, stdev=128.69 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 617], 95.00th=[ 693], | 99.00th=[ 793], 99.50th=[ 844], 99.90th=[ 894], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 112, per=25.07%, avg=43.61, stdev=19.98, samples=228 iops : min= 1, max= 28, avg=10.43, stdev= 5.05, samples=228 write: IOPS=11, BW=44.9KiB/s (46.0kB/s)(5180KiB/115239msec); 0 zone resets slat (usec): min=53, max=8935, avg=294.99, stdev=581.04 clat (msec): min=205, max=1970, avg=961.26, stdev=292.22 lat (msec): min=206, max=1970, avg=961.55, stdev=292.23 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1720], 99.50th=[ 1770], 99.90th=[ 1955], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 103, per=24.51%, avg=44.36, stdev=19.77, samples=228 iops : min= 1, max= 25, avg=10.61, stdev= 4.97, samples=228 lat (msec) : 50=0.08%, 100=0.23%, 250=0.94%, 500=33.83%, 750=25.27% lat (msec) : 1000=20.70%, 2000=18.95% cpu : usr=0.08%, sys=0.60%, ctx=4957, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=174KiB/s (178kB/s), 43.2KiB/s-43.9KiB/s (44.2kB/s-44.0kB/s), io=19.6MiB (20.5MB), run=115086-115241msec WRITE: bw=181KiB/s (186kB/s), 44.9KiB/s-45.7KiB/s (46.0kB/s-46.8kB/s), io=20.4MiB (21.4MB), run=115086-115241msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145934: Thu May 28 15:06:04 2026 read: IOPS=2, BW=90.9MiB/s (95.3MB/s)(40.0MiB/440msec) slat (nsec): min=48453k, max=48453k, avg=48453305.00, stdev= 0.00 clat (nsec): min=378646k, max=378646k, avg=378645725.00, stdev= 0.00 lat (nsec): min=427104k, max=427104k, avg=427103882.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] lat (msec) : 500=100.00% cpu : usr=0.46%, sys=3.42%, ctx=27, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=90.9MiB/s (95.3MB/s), 90.9MiB/s-90.9MiB/s (95.3MB/s-95.3MB/s), io=40.0MiB (41.9MB), run=440-440msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout