== sanity test 398c: run fio to test AIO ================= 22:34:57 (1788748497) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.511323 s, 82.0 MB/s osc.lustre-OST0000-osc-ffff941bb8fd7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146069: Sun Sep 6 22:37:44 2026 write: IOPS=16, BW=64.9KiB/s (66.4kB/s)(10.0MiB/157881msec); 0 zone resets slat (usec): min=29, max=17355, avg=232.60, stdev=772.23 clat (msec): min=154, max=1956, avg=985.39, stdev=263.62 lat (msec): min=154, max=1956, avg=985.62, stdev=263.62 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 575], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 1770], 99.95th=[ 1888], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 252, per=25.08%, avg=64.95, stdev=33.17, samples=310 iops : min= 1, max= 63, avg=15.77, stdev= 8.33, samples=310 lat (msec) : 250=2.50%, 500=0.78%, 750=10.39%, 1000=40.59%, 2000=45.74% cpu : usr=0.05%, sys=0.24%, ctx=1937, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146070: Sun Sep 6 22:37:44 2026 write: IOPS=16, BW=64.9KiB/s (66.5kB/s)(10.0MiB/157789msec); 0 zone resets slat (usec): min=25, max=16134, avg=225.35, stdev=817.53 clat (msec): min=85, max=2017, avg=985.21, stdev=264.09 lat (msec): min=86, max=2017, avg=985.44, stdev=264.08 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 575], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1653], 99.50th=[ 1821], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 260, per=24.91%, avg=64.52, stdev=31.35, samples=312 iops : min= 1, max= 65, avg=15.66, stdev= 7.87, samples=312 lat (msec) : 100=0.04%, 250=1.88%, 500=1.88%, 750=10.04%, 1000=40.51% lat (msec) : 2000=45.59%, >=2000=0.08% cpu : usr=0.04%, sys=0.23%, ctx=1876, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146071: Sun Sep 6 22:37:44 2026 write: IOPS=16, BW=64.0KiB/s (66.5kB/s)(10.0MiB/157616msec); 0 zone resets slat (usec): min=29, max=15902, avg=213.28, stdev=777.95 clat (msec): min=144, max=1999, avg=983.76, stdev=262.96 lat (msec): min=144, max=1999, avg=983.98, stdev=262.95 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 584], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1687], 99.50th=[ 1754], 99.90th=[ 1955], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 252, per=25.25%, avg=65.39, stdev=33.55, samples=308 iops : min= 1, max= 63, avg=15.88, stdev= 8.42, samples=308 lat (msec) : 250=2.42%, 500=1.29%, 750=9.14%, 1000=40.39%, 2000=46.76% cpu : usr=0.04%, sys=0.22%, ctx=1770, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146072: Sun Sep 6 22:37:44 2026 write: IOPS=16, BW=64.9KiB/s (66.5kB/s)(10.0MiB/157788msec); 0 zone resets slat (usec): min=27, max=17789, avg=230.31, stdev=924.86 clat (msec): min=108, max=1921, avg=985.50, stdev=264.28 lat (msec): min=109, max=1922, avg=985.73, stdev=264.23 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 575], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1452], | 99.00th=[ 1670], 99.50th=[ 1770], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 315, per=25.16%, avg=65.16, stdev=34.15, samples=309 iops : min= 1, max= 78, avg=15.82, stdev= 8.56, samples=309 lat (msec) : 250=2.34%, 500=1.37%, 750=9.88%, 1000=39.53%, 2000=46.88% cpu : usr=0.05%, sys=0.23%, ctx=1863, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=259KiB/s (266kB/s), 64.9KiB/s-64.0KiB/s (66.4kB/s-66.5kB/s), io=40.0MiB (41.9MB), run=157616-157881msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146104: Sun Sep 6 22:39:31 2026 read: IOPS=12, BW=51.0KiB/s (52.3kB/s)(5048KiB/98891msec) slat (usec): min=31, max=501017, avg=13187.00, stdev=15484.93 clat (msec): min=11, max=1164, avg=383.50, stdev=101.56 lat (msec): min=17, max=1175, avg=396.70, stdev=102.38 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 251], 10.00th=[ 284], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 422], 80.00th=[ 456], 90.00th=[ 498], 95.00th=[ 542], | 99.00th=[ 676], 99.50th=[ 785], 99.90th=[ 1036], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 104, per=25.07%, avg=50.64, stdev=21.45, samples=196 iops : min= 1, max= 26, avg=12.22, stdev= 5.41, samples=196 write: IOPS=13, BW=52.5KiB/s (53.8kB/s)(5192KiB/98891msec); 0 zone resets slat (usec): min=30, max=12183, avg=266.04, stdev=754.45 clat (msec): min=206, max=1753, avg=831.78, stdev=209.93 lat (msec): min=206, max=1753, avg=832.05, stdev=209.94 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 514], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 768], 50.00th=[ 827], 60.00th=[ 885], | 70.00th=[ 944], 80.00th=[ 1011], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1334], 99.50th=[ 1401], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 111, per=24.86%, avg=52.20, stdev=19.82, samples=195 iops : min= 1, max= 27, avg=12.61, stdev= 4.96, samples=195 lat (msec) : 20=0.12%, 50=0.23%, 100=0.20%, 250=1.91%, 500=43.98% lat (msec) : 750=21.05%, 1000=21.80%, 2000=10.70% cpu : usr=0.10%, sys=0.71%, ctx=4737, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146105: Sun Sep 6 22:39:31 2026 read: IOPS=12, BW=50.2KiB/s (51.4kB/s)(4976KiB/99067msec) slat (usec): min=2526, max=51248, avg=12715.67, stdev=6389.59 clat (msec): min=23, max=823, avg=379.85, stdev=88.42 lat (msec): min=34, max=832, avg=392.57, stdev=89.16 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 259], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 447], 90.00th=[ 498], 95.00th=[ 531], | 99.00th=[ 651], 99.50th=[ 701], 99.90th=[ 726], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 119, per=24.68%, avg=49.85, stdev=22.60, samples=196 iops : min= 1, max= 29, avg=12.03, stdev= 5.71, samples=196 write: IOPS=13, BW=53.1KiB/s (54.4kB/s)(5264KiB/99067msec); 0 zone resets slat (usec): min=31, max=18886, avg=309.50, stdev=1021.91 clat (msec): min=220, max=1435, avg=831.04, stdev=201.64 lat (msec): min=220, max=1435, avg=831.35, stdev=201.62 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 514], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 944], 80.00th=[ 1003], 90.00th=[ 1099], 95.00th=[ 1150], | 99.00th=[ 1318], 99.50th=[ 1401], 99.90th=[ 1418], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 104, per=25.10%, avg=52.71, stdev=18.67, samples=196 iops : min= 1, max= 26, avg=12.74, stdev= 4.71, samples=196 lat (msec) : 50=0.16%, 250=1.72%, 500=44.34%, 750=20.86%, 1000=22.34% lat (msec) : 2000=10.59% cpu : usr=0.10%, sys=0.69%, ctx=4710, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146106: Sun Sep 6 22:39:31 2026 read: IOPS=12, BW=50.4KiB/s (51.6kB/s)(4984KiB/98882msec) slat (usec): min=2782, max=50224, avg=12877.12, stdev=6782.59 clat (msec): min=17, max=1165, avg=380.59, stdev=98.26 lat (msec): min=22, max=1176, avg=393.47, stdev=99.24 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 368], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 451], 90.00th=[ 502], 95.00th=[ 542], | 99.00th=[ 676], 99.50th=[ 718], 99.90th=[ 1028], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 152, per=24.96%, avg=50.41, stdev=23.34, samples=195 iops : min= 1, max= 38, avg=12.16, stdev= 5.86, samples=195 write: IOPS=13, BW=53.2KiB/s (54.4kB/s)(5256KiB/98882msec); 0 zone resets slat (usec): min=31, max=20762, avg=299.55, stdev=1053.88 clat (msec): min=295, max=1766, avg=827.66, stdev=203.18 lat (msec): min=295, max=1766, avg=827.97, stdev=203.23 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 510], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 827], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1003], 90.00th=[ 1083], 95.00th=[ 1150], | 99.00th=[ 1334], 99.50th=[ 1401], 99.90th=[ 1687], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 111, per=24.99%, avg=52.47, stdev=18.99, samples=196 iops : min= 1, max= 27, avg=12.68, stdev= 4.78, samples=196 lat (msec) : 20=0.08%, 50=0.20%, 100=0.35%, 250=1.48%, 500=43.67% lat (msec) : 750=21.41%, 1000=22.46%, 2000=10.35% cpu : usr=0.10%, sys=0.63%, ctx=4291, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146107: Sun Sep 6 22:39:31 2026 read: IOPS=12, BW=51.0KiB/s (52.3kB/s)(5060KiB/99159msec) slat (msec): min=3, max=506, avg=13.55, stdev=19.81 clat (msec): min=18, max=1519, avg=380.19, stdev=106.38 lat (msec): min=30, max=1529, avg=393.75, stdev=109.14 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 447], 90.00th=[ 498], 95.00th=[ 527], | 99.00th=[ 684], 99.50th=[ 760], 99.90th=[ 1301], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 143, per=25.22%, avg=50.94, stdev=23.62, samples=195 iops : min= 1, max= 35, avg=12.30, stdev= 5.95, samples=195 write: IOPS=13, BW=52.2KiB/s (53.5kB/s)(5180KiB/99159msec); 0 zone resets slat (usec): min=30, max=13306, avg=237.31, stdev=685.72 clat (msec): min=154, max=1853, avg=837.45, stdev=219.92 lat (msec): min=154, max=1854, avg=837.69, stdev=219.91 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 498], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 776], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1020], 90.00th=[ 1099], 95.00th=[ 1183], | 99.00th=[ 1385], 99.50th=[ 1703], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 127, per=24.84%, avg=52.15, stdev=19.99, samples=195 iops : min= 1, max= 31, avg=12.60, stdev= 5.01, samples=195 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=1.99%, 500=45.12% lat (msec) : 750=20.04%, 1000=20.55%, 2000=11.91% cpu : usr=0.10%, sys=0.63%, ctx=4209, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=202KiB/s (207kB/s), 50.2KiB/s-51.0KiB/s (51.4kB/s-52.3kB/s), io=19.6MiB (20.5MB), run=98882-99159msec WRITE: bw=211KiB/s (216kB/s), 52.2KiB/s-53.2KiB/s (53.5kB/s-54.4kB/s), io=20.4MiB (21.4MB), run=98882-99159msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146126: Sun Sep 6 22:39:41 2026 read: IOPS=2, BW=93.0MiB/s (97.5MB/s)(40.0MiB/430msec) slat (nsec): min=21155k, max=21155k, avg=21154696.00, stdev= 0.00 clat (nsec): min=384920k, max=384920k, avg=384919617.00, stdev= 0.00 lat (nsec): min=406078k, max=406078k, avg=406077887.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.26%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=93.0MiB/s (97.5MB/s), 93.0MiB/s-93.0MiB/s (97.5MB/s-97.5MB/s), io=40.0MiB (41.9MB), run=430-430msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout