== sanity test 398c: run fio to test AIO ================= 22:31:40 (1781231500) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.550628 s, 76.2 MB/s osc.lustre-OST0000-osc-ffff924cd8130800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146344: Thu Jun 11 22:34:40 2026 write: IOPS=15, BW=60.4KiB/s (61.8kB/s)(10.0MiB/169561msec); 0 zone resets slat (usec): min=40, max=17624, avg=423.12, stdev=1165.86 clat (msec): min=385, max=2442, avg=1058.27, stdev=333.47 lat (msec): min=385, max=2442, avg=1058.69, stdev=333.45 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 634], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1485], 95.00th=[ 1821], | 99.00th=[ 2165], 99.50th=[ 2333], 99.90th=[ 2366], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=25.13%, avg=60.57, stdev=27.16, samples=332 iops : min= 1, max= 32, avg=14.64, stdev= 6.84, samples=332 lat (msec) : 500=1.37%, 750=12.23%, 1000=35.86%, 2000=48.75%, >=2000=1.80% cpu : usr=0.06%, sys=0.35%, ctx=2155, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146345: Thu Jun 11 22:34:40 2026 write: IOPS=15, BW=60.4KiB/s (61.8kB/s)(10.0MiB/169635msec); 0 zone resets slat (usec): min=38, max=25707, avg=378.20, stdev=1098.64 clat (msec): min=362, max=2447, avg=1058.56, stdev=323.32 lat (msec): min=365, max=2447, avg=1058.94, stdev=323.34 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1502], 95.00th=[ 1787], | 99.00th=[ 2056], 99.50th=[ 2165], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=25.84%, avg=62.27, stdev=30.73, samples=323 iops : min= 1, max= 32, avg=15.07, stdev= 7.70, samples=323 lat (msec) : 500=0.70%, 750=11.68%, 1000=37.27%, 2000=48.79%, >=2000=1.56% cpu : usr=0.04%, sys=0.32%, ctx=1765, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146346: Thu Jun 11 22:34:40 2026 write: IOPS=15, BW=60.4KiB/s (61.8kB/s)(10.0MiB/169567msec); 0 zone resets slat (usec): min=53, max=26508, avg=456.27, stdev=1386.85 clat (msec): min=382, max=2446, avg=1058.14, stdev=322.46 lat (msec): min=382, max=2447, avg=1058.60, stdev=322.46 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1485], 95.00th=[ 1770], | 99.00th=[ 2106], 99.50th=[ 2265], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=25.21%, avg=60.76, stdev=29.42, samples=331 iops : min= 1, max= 32, avg=14.69, stdev= 7.40, samples=331 lat (msec) : 500=0.66%, 750=11.68%, 1000=37.81%, 2000=48.36%, >=2000=1.48% cpu : usr=0.06%, sys=0.35%, ctx=2289, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146347: Thu Jun 11 22:34:40 2026 write: IOPS=15, BW=60.5KiB/s (61.0kB/s)(10.0MiB/169201msec); 0 zone resets slat (usec): min=52, max=27838, avg=444.58, stdev=1313.19 clat (msec): min=172, max=2440, avg=1056.39, stdev=330.90 lat (msec): min=172, max=2440, avg=1056.84, stdev=330.89 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1502], 95.00th=[ 1787], | 99.00th=[ 2140], 99.50th=[ 2232], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=24.98%, avg=60.20, stdev=25.19, samples=334 iops : min= 1, max= 32, avg=14.55, stdev= 6.33, samples=334 lat (msec) : 250=0.12%, 500=1.29%, 750=11.91%, 1000=36.99%, 2000=48.12% lat (msec) : >=2000=1.56% cpu : usr=0.05%, sys=0.36%, ctx=2297, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=241KiB/s (247kB/s), 60.4KiB/s-60.5KiB/s (61.8kB/s-61.0kB/s), io=40.0MiB (41.9MB), run=169201-169635msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146381: Thu Jun 11 22:36:43 2026 read: IOPS=11, BW=44.1KiB/s (45.2kB/s)(5048KiB/114444msec) slat (usec): min=3949, max=55190, avg=16050.04, stdev=8465.87 clat (msec): min=15, max=818, avg=446.34, stdev=106.01 lat (msec): min=47, max=845, avg=462.40, stdev=106.69 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 493], 80.00th=[ 535], 90.00th=[ 592], 95.00th=[ 625], | 99.00th=[ 701], 99.50th=[ 735], 99.90th=[ 793], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 96, per=25.52%, avg=44.66, stdev=20.04, samples=221 iops : min= 1, max= 24, avg=10.62, stdev= 5.02, samples=221 write: IOPS=11, BW=45.4KiB/s (46.5kB/s)(5192KiB/114444msec); 0 zone resets slat (usec): min=55, max=14021, avg=495.66, stdev=1153.73 clat (msec): min=274, max=1871, avg=957.78, stdev=281.42 lat (msec): min=274, max=1871, avg=958.28, stdev=281.36 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 701], | 30.00th=[ 785], 40.00th=[ 869], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1687], 99.50th=[ 1737], 99.90th=[ 1854], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 104, per=24.62%, avg=44.80, stdev=18.88, samples=226 iops : min= 1, max= 26, avg=10.66, stdev= 4.71, samples=226 lat (msec) : 20=0.04%, 100=0.08%, 250=0.66%, 500=36.64%, 750=24.41% lat (msec) : 1000=16.02%, 2000=22.15% cpu : usr=0.11%, sys=0.78%, ctx=4354, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146382: Thu Jun 11 22:36:43 2026 read: IOPS=10, BW=43.5KiB/s (44.6kB/s)(4976KiB/114338msec) slat (usec): min=3040, max=60978, avg=16013.71, stdev=8337.89 clat (msec): min=22, max=843, avg=443.46, stdev=109.93 lat (msec): min=28, max=849, avg=459.49, stdev=110.80 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 418], 50.00th=[ 439], 60.00th=[ 464], | 70.00th=[ 489], 80.00th=[ 527], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 726], 99.50th=[ 768], 99.90th=[ 827], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 7, max= 111, per=24.61%, avg=43.07, stdev=22.28, samples=225 iops : min= 1, max= 27, avg=10.23, stdev= 5.60, samples=225 write: IOPS=11, BW=46.0KiB/s (47.1kB/s)(5264KiB/114338msec); 0 zone resets slat (usec): min=55, max=14786, avg=525.80, stdev=1288.71 clat (msec): min=315, max=1861, avg=952.61, stdev=278.01 lat (msec): min=316, max=1861, avg=953.14, stdev=277.93 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 523], 10.00th=[ 575], 20.00th=[ 701], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 944], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1603], 99.50th=[ 1703], 99.90th=[ 1787], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8, max= 111, per=24.93%, avg=45.37, stdev=19.91, samples=227 iops : min= 2, max= 27, avg=10.81, stdev= 4.96, samples=227 lat (msec) : 50=0.31%, 100=0.12%, 250=0.39%, 500=36.33%, 750=24.96% lat (msec) : 1000=15.20%, 2000=22.70% cpu : usr=0.10%, sys=0.83%, ctx=4455, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146383: Thu Jun 11 22:36:43 2026 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(4984KiB/114331msec) slat (usec): min=3156, max=55098, avg=15966.45, stdev=8394.16 clat (msec): min=22, max=817, avg=448.78, stdev=105.64 lat (msec): min=47, max=848, avg=464.75, stdev=106.24 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 296], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 535], 90.00th=[ 592], 95.00th=[ 634], | 99.00th=[ 735], 99.50th=[ 751], 99.90th=[ 810], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 110, per=24.77%, avg=43.35, stdev=20.43, samples=225 iops : min= 1, max= 27, avg=10.30, stdev= 5.15, samples=225 write: IOPS=11, BW=45.0KiB/s (47.1kB/s)(5256KiB/114331msec); 0 zone resets slat (usec): min=61, max=25878, avg=613.20, stdev=1720.60 clat (msec): min=243, max=1834, avg=948.14, stdev=266.90 lat (msec): min=243, max=1834, avg=948.76, stdev=266.83 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 542], 10.00th=[ 600], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 944], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1603], 99.50th=[ 1703], 99.90th=[ 1821], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 103, per=24.90%, avg=45.31, stdev=21.06, samples=226 iops : min= 1, max= 25, avg=10.78, stdev= 5.27, samples=226 lat (msec) : 50=0.04%, 100=0.12%, 250=0.59%, 500=35.51%, 750=25.08% lat (msec) : 1000=16.99%, 2000=21.68% cpu : usr=0.12%, sys=0.83%, ctx=4675, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146384: Thu Jun 11 22:36:43 2026 read: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5060KiB/113585msec) slat (usec): min=114, max=56603, avg=16133.63, stdev=8559.20 clat (msec): min=48, max=811, avg=443.35, stdev=113.93 lat (msec): min=56, max=848, avg=459.50, stdev=115.14 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 435], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 531], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 743], 99.50th=[ 760], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 104, per=25.73%, avg=45.03, stdev=19.48, samples=220 iops : min= 1, max= 26, avg=10.72, stdev= 4.87, samples=220 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(5180KiB/113585msec); 0 zone resets slat (usec): min=54, max=18908, avg=463.84, stdev=1136.36 clat (msec): min=151, max=1882, avg=953.14, stdev=285.77 lat (msec): min=152, max=1882, avg=953.61, stdev=285.75 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 860], 50.00th=[ 953], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1687], 99.50th=[ 1720], 99.90th=[ 1854], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 104, per=24.62%, avg=44.81, stdev=20.82, samples=225 iops : min= 1, max= 26, avg=10.67, stdev= 5.22, samples=225 lat (msec) : 50=0.04%, 100=0.47%, 250=0.98%, 500=35.62%, 750=25.47% lat (msec) : 1000=15.20%, 2000=22.23% cpu : usr=0.11%, sys=0.84%, ctx=4214, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=175KiB/s (180kB/s), 43.5KiB/s-44.5KiB/s (44.6kB/s-45.6kB/s), io=19.6MiB (20.5MB), run=113585-114444msec WRITE: bw=183KiB/s (187kB/s), 45.4KiB/s-46.0KiB/s (46.5kB/s-47.1kB/s), io=20.4MiB (21.4MB), run=113585-114444msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146406: Thu Jun 11 22:36:54 2026 read: IOPS=2, BW=84.6MiB/s (88.7MB/s)(40.0MiB/473msec) slat (nsec): min=23265k, max=23265k, avg=23264679.00, stdev= 0.00 clat (nsec): min=426257k, max=426257k, avg=426256855.00, stdev= 0.00 lat (nsec): min=449526k, max=449526k, avg=449526159.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.21%, sys=2.75%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=84.6MiB/s (88.7MB/s), 84.6MiB/s-84.6MiB/s (88.7MB/s-88.7MB/s), io=40.0MiB (41.9MB), run=473-473msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout