== sanity test 398c: run fio to test AIO ================= 11:15:18 (1790694918) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.832196 s, 50.4 MB/s osc.lustre-OST0000-osc-ffff8bca9d4fc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145553: Tue Sep 29 11:18:30 2026 write: IOPS=14, BW=57.3KiB/s (58.7kB/s)(10.0MiB/178752msec); 0 zone resets slat (usec): min=23, max=32181, avg=267.03, stdev=1141.57 clat (msec): min=144, max=2281, avg=1116.23, stdev=310.11 lat (msec): min=149, max=2281, avg=1116.51, stdev=310.10 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 659], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1003], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1653], | 99.00th=[ 1921], 99.50th=[ 1989], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 128, per=25.86%, avg=58.95, stdev=33.58, samples=341 iops : min= 1, max= 32, avg=14.23, stdev= 8.38, samples=341 lat (msec) : 250=0.23%, 500=2.54%, 750=6.17%, 1000=30.08%, 2000=60.59% lat (msec) : >=2000=0.39% cpu : usr=0.05%, sys=0.22%, ctx=1741, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145554: Tue Sep 29 11:18:30 2026 write: IOPS=14, BW=57.2KiB/s (58.6kB/s)(10.0MiB/178939msec); 0 zone resets slat (usec): min=23, max=23193, avg=281.48, stdev=1149.12 clat (msec): min=418, max=2094, avg=1116.72, stdev=296.81 lat (msec): min=418, max=2094, avg=1117.00, stdev=296.78 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 667], 10.00th=[ 785], 20.00th=[ 885], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1368], 90.00th=[ 1519], 95.00th=[ 1653], | 99.00th=[ 1871], 99.50th=[ 1938], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 168, per=25.35%, avg=57.79, stdev=26.72, samples=348 iops : min= 1, max= 42, avg=13.95, stdev= 6.71, samples=348 lat (msec) : 500=2.19%, 750=5.35%, 1000=30.04%, 2000=62.15%, >=2000=0.27% cpu : usr=0.06%, sys=0.27%, ctx=1893, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145555: Tue Sep 29 11:18:30 2026 write: IOPS=14, BW=57.2KiB/s (58.6kB/s)(10.0MiB/178939msec); 0 zone resets slat (usec): min=25, max=17326, avg=275.33, stdev=966.04 clat (msec): min=416, max=2261, avg=1117.03, stdev=299.29 lat (msec): min=416, max=2261, avg=1117.30, stdev=299.29 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 693], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1250], 80.00th=[ 1368], 90.00th=[ 1519], 95.00th=[ 1653], | 99.00th=[ 1854], 99.50th=[ 1955], 99.90th=[ 2198], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 128, per=25.42%, avg=57.95, stdev=27.53, samples=347 iops : min= 1, max= 32, avg=13.99, stdev= 6.90, samples=347 lat (msec) : 500=1.88%, 750=6.33%, 1000=30.51%, 2000=60.98%, >=2000=0.31% cpu : usr=0.06%, sys=0.23%, ctx=1664, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145556: Tue Sep 29 11:18:30 2026 write: IOPS=14, BW=57.3KiB/s (58.7kB/s)(10.0MiB/178617msec); 0 zone resets slat (usec): min=23, max=22830, avg=275.54, stdev=1051.10 clat (msec): min=80, max=2199, avg=1115.23, stdev=304.54 lat (msec): min=80, max=2199, avg=1115.51, stdev=304.52 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 634], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1385], 90.00th=[ 1519], 95.00th=[ 1653], | 99.00th=[ 1838], 99.50th=[ 1905], 99.90th=[ 2106], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 160, per=25.12%, avg=57.28, stdev=26.04, samples=351 iops : min= 1, max= 40, avg=13.82, stdev= 6.54, samples=351 lat (msec) : 100=0.04%, 250=0.08%, 500=2.03%, 750=6.17%, 1000=30.74% lat (msec) : 2000=60.59%, >=2000=0.35% cpu : usr=0.04%, sys=0.25%, ctx=1813, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=229KiB/s (234kB/s), 57.2KiB/s-57.3KiB/s (58.6kB/s-58.7kB/s), io=40.0MiB (41.9MB), run=178617-178939msec 4% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145592: Tue Sep 29 11:20:47 2026 read: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5048KiB/123436msec) slat (usec): min=33, max=82265, avg=16864.11, stdev=9782.06 clat (msec): min=40, max=1250, avg=468.20, stdev=151.49 lat (msec): min=48, max=1273, avg=485.07, stdev=153.88 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 651], 95.00th=[ 743], | 99.00th=[ 969], 99.50th=[ 1099], 99.90th=[ 1200], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 112, per=25.70%, avg=41.37, stdev=20.20, samples=238 iops : min= 1, max= 28, avg= 9.80, stdev= 5.08, samples=238 write: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5192KiB/123436msec); 0 zone resets slat (usec): min=29, max=17669, avg=325.83, stdev=985.12 clat (msec): min=198, max=2469, avg=1048.67, stdev=326.42 lat (msec): min=198, max=2470, avg=1049.00, stdev=326.41 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 860], 40.00th=[ 936], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1502], 95.00th=[ 1636], | 99.00th=[ 2005], 99.50th=[ 2198], 99.90th=[ 2333], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 96, per=25.06%, avg=42.10, stdev=19.60, samples=240 iops : min= 1, max= 24, avg= 9.98, stdev= 4.92, samples=240 lat (msec) : 50=0.04%, 100=0.20%, 250=1.29%, 500=31.91%, 750=22.81% lat (msec) : 1000=16.88%, 2000=26.37%, >=2000=0.51% cpu : usr=0.09%, sys=0.68%, ctx=4378, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145593: Tue Sep 29 11:20:47 2026 read: IOPS=10, BW=40.1KiB/s (41.0kB/s)(4976KiB/124224msec) slat (usec): min=2961, max=67533, avg=17309.83, stdev=10192.45 clat (msec): min=25, max=1448, avg=473.02, stdev=157.75 lat (msec): min=33, max=1469, avg=490.34, stdev=160.13 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 477], | 70.00th=[ 518], 80.00th=[ 567], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 1150], 99.50th=[ 1183], 99.90th=[ 1267], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 5, max= 104, per=24.64%, avg=39.67, stdev=22.04, samples=244 iops : min= 1, max= 26, avg= 9.37, stdev= 5.57, samples=244 write: IOPS=10, BW=42.4KiB/s (43.4kB/s)(5264KiB/124224msec); 0 zone resets slat (usec): min=30, max=16627, avg=331.82, stdev=984.86 clat (msec): min=261, max=2370, avg=1044.72, stdev=322.86 lat (msec): min=261, max=2370, avg=1045.05, stdev=322.86 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 1989], 99.50th=[ 2198], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 112, per=25.10%, avg=42.17, stdev=19.59, samples=244 iops : min= 1, max= 28, avg= 9.99, stdev= 4.91, samples=244 lat (msec) : 50=0.12%, 100=0.08%, 250=1.21%, 500=31.91%, 750=22.50% lat (msec) : 1000=17.11%, 2000=26.56%, >=2000=0.51% cpu : usr=0.09%, sys=0.63%, ctx=3880, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145594: Tue Sep 29 11:20:47 2026 read: IOPS=10, BW=40.1KiB/s (41.1kB/s)(4984KiB/124230msec) slat (usec): min=3094, max=83857, avg=17035.70, stdev=9789.77 clat (msec): min=15, max=1365, avg=471.24, stdev=155.62 lat (msec): min=31, max=1383, avg=488.29, stdev=158.38 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 751], | 99.00th=[ 1003], 99.50th=[ 1150], 99.90th=[ 1334], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7, max= 96, per=24.92%, avg=40.12, stdev=20.42, samples=243 iops : min= 1, max= 24, avg= 9.48, stdev= 5.14, samples=243 write: IOPS=10, BW=42.3KiB/s (43.3kB/s)(5256KiB/124230msec); 0 zone resets slat (usec): min=32, max=13998, avg=383.60, stdev=1121.99 clat (msec): min=322, max=2365, avg=1045.73, stdev=320.08 lat (msec): min=328, max=2365, avg=1046.12, stdev=320.09 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 609], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 1020], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1653], | 99.00th=[ 2005], 99.50th=[ 2198], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 111, per=25.06%, avg=42.10, stdev=18.67, samples=243 iops : min= 1, max= 27, avg= 9.98, stdev= 4.67, samples=243 lat (msec) : 20=0.08%, 50=0.20%, 100=0.08%, 250=0.78%, 500=31.91% lat (msec) : 750=22.77%, 1000=16.95%, 2000=26.68%, >=2000=0.55% cpu : usr=0.12%, sys=0.64%, ctx=4372, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145595: Tue Sep 29 11:20:47 2026 read: IOPS=10, BW=41.0KiB/s (42.0kB/s)(5060KiB/123355msec) slat (usec): min=2260, max=77374, avg=16984.60, stdev=9974.77 clat (msec): min=13, max=1472, avg=473.21, stdev=159.46 lat (msec): min=21, max=1484, avg=490.20, stdev=161.74 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 567], 90.00th=[ 659], 95.00th=[ 776], | 99.00th=[ 1062], 99.50th=[ 1167], 99.90th=[ 1418], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 119, per=25.82%, avg=41.57, stdev=20.46, samples=238 iops : min= 1, max= 29, avg= 9.85, stdev= 5.12, samples=238 write: IOPS=10, BW=41.0KiB/s (43.0kB/s)(5180KiB/123355msec); 0 zone resets slat (usec): min=27, max=15820, avg=344.44, stdev=1020.55 clat (msec): min=337, max=2369, avg=1041.49, stdev=323.40 lat (msec): min=337, max=2369, avg=1041.84, stdev=323.44 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 609], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 844], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1250], 90.00th=[ 1485], 95.00th=[ 1636], | 99.00th=[ 1989], 99.50th=[ 2056], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 5, max= 102, per=24.94%, avg=41.90, stdev=19.06, samples=241 iops : min= 1, max= 25, avg= 9.93, stdev= 4.82, samples=241 lat (msec) : 20=0.04%, 100=0.08%, 250=1.17%, 500=32.11%, 750=23.59% lat (msec) : 1000=16.41%, 2000=26.17%, >=2000=0.43% cpu : usr=0.10%, sys=0.67%, ctx=4611, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=162KiB/s (165kB/s), 40.1KiB/s-41.0KiB/s (41.0kB/s-42.0kB/s), io=19.6MiB (20.5MB), run=123355-124230msec WRITE: bw=168KiB/s (172kB/s), 41.0KiB/s-42.4KiB/s (43.0kB/s-43.4kB/s), io=20.4MiB (21.4MB), run=123355-124230msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145619: Tue Sep 29 11:20:59 2026 read: IOPS=1, BW=71.8MiB/s (75.3MB/s)(40.0MiB/557msec) slat (nsec): min=25693k, max=25693k, avg=25692805.00, stdev= 0.00 clat (nsec): min=511553k, max=511553k, avg=511553262.00, stdev= 0.00 lat (nsec): min=537251k, max=537251k, avg=537251358.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.52%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.8MiB/s (75.3MB/s), 71.8MiB/s-71.8MiB/s (75.3MB/s-75.3MB/s), io=40.0MiB (41.9MB), run=557-557msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout