== sanity test 398c: run fio to test AIO ================= 18:27:42 (1788647262) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.364918 s, 115 MB/s osc.lustre-OST0000-osc-ffff94f2c4994000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145741: Sat Sep 5 18:30:14 2026 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144522msec); 0 zone resets slat (usec): min=23, max=11772, avg=162.39, stdev=568.60 clat (msec): min=230, max=1814, avg=902.21, stdev=252.55 lat (msec): min=230, max=1814, avg=902.37, stdev=252.56 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 542], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1200], 95.00th=[ 1385], | 99.00th=[ 1653], 99.50th=[ 1720], 99.90th=[ 1804], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 254, per=25.17%, avg=71.23, stdev=33.21, samples=283 iops : min= 1, max= 63, avg=17.34, stdev= 8.33, samples=283 lat (msec) : 250=0.62%, 500=3.79%, 750=20.20%, 1000=46.60%, 2000=28.79% cpu : usr=0.04%, sys=0.22%, ctx=1515, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145742: Sat Sep 5 18:30:14 2026 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144524msec); 0 zone resets slat (usec): min=23, max=23869, avg=164.45, stdev=752.91 clat (msec): min=250, max=1925, avg=902.27, stdev=251.63 lat (msec): min=250, max=1925, avg=902.43, stdev=251.65 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 514], 10.00th=[ 659], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1217], 95.00th=[ 1351], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 1888], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 254, per=24.90%, avg=70.48, stdev=28.86, samples=286 iops : min= 1, max= 63, avg=17.15, stdev= 7.24, samples=286 lat (msec) : 500=4.84%, 750=19.02%, 1000=46.76%, 2000=29.38% cpu : usr=0.06%, sys=0.23%, ctx=1776, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145743: Sat Sep 5 18:30:14 2026 write: IOPS=17, BW=70.0KiB/s (72.7kB/s)(10.0MiB/144256msec); 0 zone resets slat (usec): min=23, max=21451, avg=184.27, stdev=837.88 clat (msec): min=167, max=1941, avg=900.98, stdev=263.52 lat (msec): min=168, max=1941, avg=901.17, stdev=263.57 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 439], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1351], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 1888], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 254, per=24.97%, avg=70.66, stdev=33.46, samples=285 iops : min= 1, max= 63, avg=17.20, stdev= 8.41, samples=285 lat (msec) : 250=1.09%, 500=4.18%, 750=18.52%, 1000=45.31%, 2000=30.90% cpu : usr=0.05%, sys=0.23%, ctx=1717, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145744: Sat Sep 5 18:30:14 2026 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144525msec); 0 zone resets slat (usec): min=24, max=23629, avg=173.43, stdev=809.74 clat (msec): min=134, max=1813, avg=902.32, stdev=255.37 lat (msec): min=134, max=1813, avg=902.49, stdev=255.33 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 502], 10.00th=[ 651], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 1003], 80.00th=[ 1070], 90.00th=[ 1217], 95.00th=[ 1351], | 99.00th=[ 1720], 99.50th=[ 1737], 99.90th=[ 1804], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 246, per=25.25%, avg=71.47, stdev=32.96, samples=282 iops : min= 1, max= 61, avg=17.41, stdev= 8.28, samples=282 lat (msec) : 250=0.51%, 500=4.53%, 750=18.91%, 1000=45.27%, 2000=30.78% cpu : usr=0.05%, sys=0.22%, ctx=1897, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=283KiB/s (290kB/s), 70.9KiB/s-70.0KiB/s (72.6kB/s-72.7kB/s), io=40.0MiB (41.9MB), run=144256-144525msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145775: Sat Sep 5 18:32:22 2026 read: IOPS=10, BW=42.3KiB/s (43.4kB/s)(5048KiB/119234msec) slat (msec): min=2, max=179, avg=15.71, stdev=11.13 clat (msec): min=27, max=1518, avg=459.28, stdev=198.24 lat (msec): min=40, max=1534, avg=475.00, stdev=202.37 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 481], 80.00th=[ 550], 90.00th=[ 743], 95.00th=[ 919], | 99.00th=[ 1167], 99.50th=[ 1301], 99.90th=[ 1435], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 159, per=25.27%, avg=42.45, stdev=23.32, samples=233 iops : min= 1, max= 39, avg=10.13, stdev= 5.89, samples=233 write: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5192KiB/119234msec); 0 zone resets slat (usec): min=30, max=18336, avg=253.17, stdev=1013.83 clat (msec): min=301, max=3836, avg=1004.18, stdev=428.38 lat (msec): min=301, max=3836, avg=1004.44, stdev=428.38 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 969], | 70.00th=[ 1083], 80.00th=[ 1217], 90.00th=[ 1435], 95.00th=[ 1754], | 99.00th=[ 2735], 99.50th=[ 3540], 99.90th=[ 3742], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 7, max= 96, per=25.03%, avg=43.81, stdev=21.19, samples=231 iops : min= 1, max= 24, avg=10.48, stdev= 5.37, samples=231 lat (msec) : 50=0.08%, 100=0.16%, 250=1.33%, 500=35.47%, 750=19.30% lat (msec) : 1000=23.75%, 2000=18.44%, >=2000=1.48% cpu : usr=0.10%, sys=0.54%, ctx=4453, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145776: Sat Sep 5 18:32:22 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(4976KiB/119094msec) slat (usec): min=34, max=177880, avg=15343.67, stdev=10359.16 clat (msec): min=10, max=1639, avg=460.39, stdev=199.32 lat (msec): min=14, max=1676, avg=475.74, stdev=202.87 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 259], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 481], 80.00th=[ 558], 90.00th=[ 726], 95.00th=[ 911], | 99.00th=[ 1116], 99.50th=[ 1301], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 127, per=25.02%, avg=42.03, stdev=22.16, samples=231 iops : min= 1, max= 31, avg=10.03, stdev= 5.60, samples=231 write: IOPS=11, BW=44.2KiB/s (45.3kB/s)(5264KiB/119094msec); 0 zone resets slat (usec): min=25, max=19524, avg=262.11, stdev=1031.45 clat (msec): min=243, max=3837, avg=997.39, stdev=433.77 lat (msec): min=243, max=3837, avg=997.66, stdev=433.80 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 894], 60.00th=[ 969], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1435], 95.00th=[ 1787], | 99.00th=[ 2735], 99.50th=[ 3473], 99.90th=[ 3675], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 4, max= 119, per=25.17%, avg=44.05, stdev=22.00, samples=233 iops : min= 1, max= 29, avg=10.54, stdev= 5.58, samples=233 lat (msec) : 20=0.08%, 50=0.08%, 100=0.20%, 250=1.64%, 500=35.08% lat (msec) : 750=20.98%, 1000=21.33%, 2000=18.91%, >=2000=1.72% cpu : usr=0.09%, sys=0.56%, ctx=4299, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145777: Sat Sep 5 18:32:22 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(4984KiB/119298msec) slat (usec): min=3139, max=77040, avg=15770.76, stdev=10018.38 clat (msec): min=16, max=1666, avg=455.63, stdev=201.79 lat (msec): min=22, max=1684, avg=471.40, stdev=205.04 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 435], | 70.00th=[ 477], 80.00th=[ 535], 90.00th=[ 693], 95.00th=[ 877], | 99.00th=[ 1200], 99.50th=[ 1469], 99.90th=[ 1653], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 120, per=25.71%, avg=43.19, stdev=23.28, samples=226 iops : min= 1, max= 30, avg=10.32, stdev= 5.87, samples=226 write: IOPS=11, BW=44.1KiB/s (45.1kB/s)(5256KiB/119298msec); 0 zone resets slat (usec): min=31, max=18009, avg=247.84, stdev=977.67 clat (msec): min=194, max=3901, avg=1001.97, stdev=428.58 lat (msec): min=194, max=3901, avg=1002.22, stdev=428.62 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 961], | 70.00th=[ 1099], 80.00th=[ 1234], 90.00th=[ 1452], 95.00th=[ 1787], | 99.00th=[ 2869], 99.50th=[ 3239], 99.90th=[ 3876], 99.95th=[ 3910], | 99.99th=[ 3910] bw ( KiB/s): min= 7, max= 104, per=25.34%, avg=44.34, stdev=20.80, samples=231 iops : min= 1, max= 26, avg=10.61, stdev= 5.27, samples=231 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=1.84%, 500=35.20% lat (msec) : 750=19.61%, 1000=22.70%, 2000=18.95%, >=2000=1.52% cpu : usr=0.09%, sys=0.52%, ctx=4476, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145778: Sat Sep 5 18:32:22 2026 read: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5060KiB/119089msec) slat (msec): min=2, max=124, avg=15.50, stdev=10.25 clat (msec): min=65, max=1614, avg=450.58, stdev=188.99 lat (msec): min=76, max=1643, avg=466.09, stdev=192.28 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 430], | 70.00th=[ 468], 80.00th=[ 518], 90.00th=[ 701], 95.00th=[ 877], | 99.00th=[ 1183], 99.50th=[ 1234], 99.90th=[ 1485], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 136, per=26.13%, avg=43.89, stdev=22.07, samples=226 iops : min= 1, max= 34, avg=10.50, stdev= 5.59, samples=226 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5180KiB/119089msec); 0 zone resets slat (usec): min=30, max=16369, avg=196.25, stdev=632.09 clat (msec): min=284, max=3748, avg=1012.07, stdev=433.93 lat (msec): min=284, max=3748, avg=1012.27, stdev=433.95 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 844], 50.00th=[ 902], 60.00th=[ 969], | 70.00th=[ 1099], 80.00th=[ 1234], 90.00th=[ 1435], 95.00th=[ 1754], | 99.00th=[ 2937], 99.50th=[ 3373], 99.90th=[ 3708], 99.95th=[ 3742], | 99.99th=[ 3742] bw ( KiB/s): min= 7, max= 104, per=25.06%, avg=43.85, stdev=20.77, samples=230 iops : min= 1, max= 26, avg=10.48, stdev= 5.23, samples=230 lat (msec) : 100=0.12%, 250=1.29%, 500=36.91%, 750=19.22%, 1000=22.66% lat (msec) : 2000=18.05%, >=2000=1.76% cpu : usr=0.09%, sys=0.56%, ctx=4451, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=168KiB/s (172kB/s), 41.8KiB/s-42.5KiB/s (42.8kB/s-43.5kB/s), io=19.6MiB (20.5MB), run=119089-119298msec WRITE: bw=175KiB/s (179kB/s), 43.5KiB/s-44.2KiB/s (44.5kB/s-45.3kB/s), io=20.4MiB (21.4MB), run=119089-119298msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145798: Sat Sep 5 18:32:31 2026 read: IOPS=1, BW=71.8MiB/s (75.3MB/s)(40.0MiB/557msec) slat (nsec): min=35993k, max=35993k, avg=35993099.00, stdev= 0.00 clat (nsec): min=497174k, max=497174k, avg=497174318.00, stdev= 0.00 lat (nsec): min=533172k, max=533172k, avg=533172484.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 498], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.52%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.8MiB/s (75.3MB/s), 71.8MiB/s-71.8MiB/s (75.3MB/s-75.3MB/s), io=40.0MiB (41.9MB), run=557-557msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout