== sanity test 398c: run fio to test AIO ================= 08:07:20 (1787746040) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.748909 s, 56.0 MB/s osc.lustre-OST0000-osc-ffff8d98adef2800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145795: Wed Aug 26 08:10:17 2026 write: IOPS=15, BW=60.8KiB/s (62.2kB/s)(10.0MiB/168475msec); 0 zone resets slat (usec): min=23, max=12138, avg=167.95, stdev=654.31 clat (msec): min=76, max=2600, avg=1051.87, stdev=312.01 lat (msec): min=76, max=2600, avg=1052.04, stdev=312.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 625], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 2022], 99.50th=[ 2232], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 128, per=25.45%, avg=61.84, stdev=29.48, samples=326 iops : min= 1, max= 32, avg=15.04, stdev= 7.39, samples=326 lat (msec) : 100=0.04%, 250=0.62%, 500=0.94%, 750=12.23%, 1000=33.24% lat (msec) : 2000=51.88%, >=2000=1.05% cpu : usr=0.04%, sys=0.19%, ctx=1532, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145796: Wed Aug 26 08:10:17 2026 write: IOPS=15, BW=60.8KiB/s (62.3kB/s)(10.0MiB/168373msec); 0 zone resets slat (usec): min=22, max=17091, avg=191.27, stdev=910.23 clat (msec): min=180, max=2380, avg=1051.28, stdev=309.38 lat (msec): min=181, max=2381, avg=1051.47, stdev=309.39 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 1921], 99.50th=[ 2232], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 135, per=25.37%, avg=61.64, stdev=31.54, samples=327 iops : min= 1, max= 33, avg=14.99, stdev= 7.90, samples=327 lat (msec) : 250=0.04%, 500=1.56%, 750=12.54%, 1000=31.60%, 2000=53.59% lat (msec) : >=2000=0.66% cpu : usr=0.04%, sys=0.21%, ctx=1684, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145797: Wed Aug 26 08:10:17 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168078msec); 0 zone resets slat (usec): min=27, max=16809, avg=224.89, stdev=938.68 clat (msec): min=53, max=2479, avg=1049.80, stdev=319.18 lat (msec): min=53, max=2479, avg=1050.03, stdev=319.17 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1620], | 99.00th=[ 1921], 99.50th=[ 2366], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 237, per=24.96%, avg=60.66, stdev=27.16, samples=332 iops : min= 1, max= 59, avg=14.75, stdev= 6.80, samples=332 lat (msec) : 100=0.35%, 250=0.47%, 500=1.29%, 750=11.37%, 1000=32.73% lat (msec) : 2000=52.93%, >=2000=0.86% cpu : usr=0.05%, sys=0.20%, ctx=2002, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145798: Wed Aug 26 08:10:17 2026 write: IOPS=15, BW=60.8KiB/s (62.3kB/s)(10.0MiB/168372msec); 0 zone resets slat (usec): min=28, max=17412, avg=192.94, stdev=833.07 clat (msec): min=199, max=2485, avg=1051.24, stdev=311.13 lat (msec): min=199, max=2489, avg=1051.44, stdev=311.17 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 2022], 99.50th=[ 2232], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 208, per=25.29%, avg=61.46, stdev=29.58, samples=328 iops : min= 1, max= 52, avg=14.95, stdev= 7.40, samples=328 lat (msec) : 250=0.39%, 500=1.52%, 750=11.84%, 1000=31.95%, 2000=53.28% lat (msec) : >=2000=1.02% cpu : usr=0.04%, sys=0.19%, ctx=2003, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=243KiB/s (249kB/s), 60.8KiB/s-60.9KiB/s (62.2kB/s-62.4kB/s), io=40.0MiB (41.9MB), run=168078-168475msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145834: Wed Aug 26 08:12:27 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5048KiB/120583msec) slat (usec): min=2804, max=81960, avg=15152.76, stdev=9574.16 clat (msec): min=13, max=925, avg=467.09, stdev=125.28 lat (msec): min=29, max=942, avg=482.25, stdev=127.00 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 284], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 634], 95.00th=[ 676], | 99.00th=[ 793], 99.50th=[ 818], 99.90th=[ 885], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 143, per=25.02%, avg=41.53, stdev=21.64, samples=238 iops : min= 1, max= 35, avg= 9.92, stdev= 5.40, samples=238 write: IOPS=10, BW=43.1KiB/s (44.1kB/s)(5192KiB/120583msec); 0 zone resets slat (usec): min=32, max=10919, avg=192.29, stdev=546.76 clat (msec): min=287, max=2236, avg=1014.74, stdev=298.68 lat (msec): min=287, max=2236, avg=1014.93, stdev=298.69 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 2005], 99.50th=[ 2089], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 120, per=24.55%, avg=42.46, stdev=20.29, samples=239 iops : min= 1, max= 30, avg=10.15, stdev= 5.09, samples=239 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=1.05%, 500=31.45% lat (msec) : 750=23.83%, 1000=19.30%, 2000=23.63%, >=2000=0.51% cpu : usr=0.08%, sys=0.43%, ctx=4220, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145835: Wed Aug 26 08:12:27 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4976KiB/120490msec) slat (usec): min=52, max=62566, avg=15605.49, stdev=9283.87 clat (msec): min=7, max=1042, avg=466.75, stdev=129.75 lat (msec): min=7, max=1062, avg=482.36, stdev=131.18 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 634], 95.00th=[ 676], | 99.00th=[ 802], 99.50th=[ 852], 99.90th=[ 944], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 111, per=24.67%, avg=40.95, stdev=20.37, samples=238 iops : min= 1, max= 27, avg= 9.77, stdev= 5.08, samples=238 write: IOPS=10, BW=43.7KiB/s (44.7kB/s)(5264KiB/120490msec); 0 zone resets slat (usec): min=29, max=19790, avg=239.54, stdev=1036.41 clat (msec): min=111, max=2201, avg=1007.44, stdev=302.70 lat (msec): min=112, max=2201, avg=1007.68, stdev=302.68 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 852], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1502], | 99.00th=[ 1989], 99.50th=[ 2056], 99.90th=[ 2123], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8, max= 112, per=25.21%, avg=43.61, stdev=18.94, samples=236 iops : min= 2, max= 28, avg=10.44, stdev= 4.77, samples=236 lat (msec) : 10=0.04%, 50=0.27%, 100=0.12%, 250=1.29%, 500=30.66% lat (msec) : 750=24.06%, 1000=19.06%, 2000=23.98%, >=2000=0.51% cpu : usr=0.08%, sys=0.51%, ctx=4590, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145836: Wed Aug 26 08:12:27 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4984KiB/120581msec) slat (usec): min=2314, max=97227, avg=15939.69, stdev=10151.79 clat (msec): min=10, max=967, avg=464.54, stdev=127.10 lat (msec): min=22, max=1002, avg=480.48, stdev=128.95 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 676], | 99.00th=[ 802], 99.50th=[ 827], 99.90th=[ 894], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 119, per=25.18%, avg=41.80, stdev=19.88, samples=234 iops : min= 1, max= 29, avg= 9.98, stdev= 4.97, samples=234 write: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5256KiB/120581msec); 0 zone resets slat (usec): min=24, max=9254, avg=192.04, stdev=519.38 clat (msec): min=80, max=2203, avg=1010.11, stdev=289.99 lat (msec): min=80, max=2203, avg=1010.30, stdev=290.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1921], 99.50th=[ 2005], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 104, per=24.74%, avg=42.80, stdev=19.42, samples=239 iops : min= 1, max= 26, avg=10.23, stdev= 4.85, samples=239 lat (msec) : 20=0.16%, 50=0.16%, 100=0.08%, 250=0.70%, 500=31.13% lat (msec) : 750=23.71%, 1000=19.88%, 2000=23.75%, >=2000=0.43% cpu : usr=0.08%, sys=0.48%, ctx=4480, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145837: Wed Aug 26 08:12:27 2026 read: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5060KiB/120234msec) slat (msec): min=3, max=522, avg=16.06, stdev=17.18 clat (msec): min=20, max=1119, avg=470.82, stdev=128.61 lat (msec): min=32, max=1138, avg=486.89, stdev=129.50 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 288], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 397], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 827], 99.50th=[ 894], 99.90th=[ 961], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 103, per=25.47%, avg=42.27, stdev=18.76, samples=235 iops : min= 1, max= 25, avg=10.10, stdev= 4.73, samples=235 write: IOPS=10, BW=43.1KiB/s (44.1kB/s)(5180KiB/120234msec); 0 zone resets slat (usec): min=29, max=22716, avg=267.67, stdev=1309.84 clat (msec): min=268, max=2190, avg=1006.80, stdev=299.41 lat (msec): min=269, max=2190, avg=1007.07, stdev=299.40 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 978], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1502], | 99.00th=[ 1955], 99.50th=[ 2056], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 120, per=24.67%, avg=42.68, stdev=18.74, samples=237 iops : min= 1, max= 30, avg=10.21, stdev= 4.71, samples=237 lat (msec) : 50=0.20%, 100=0.16%, 250=0.78%, 500=30.78%, 750=24.77% lat (msec) : 1000=19.45%, 2000=23.44%, >=2000=0.43% cpu : usr=0.07%, sys=0.46%, ctx=4266, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.3KiB/s-42.1KiB/s (42.3kB/s-43.1kB/s), io=19.6MiB (20.5MB), run=120234-120583msec WRITE: bw=173KiB/s (177kB/s), 43.1KiB/s-43.7KiB/s (44.1kB/s-44.7kB/s), io=20.4MiB (21.4MB), run=120234-120583msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145859: Wed Aug 26 08:12:33 2026 read: IOPS=1, BW=77.8MiB/s (81.6MB/s)(40.0MiB/514msec) slat (nsec): min=17648k, max=17648k, avg=17648239.00, stdev= 0.00 clat (nsec): min=484149k, max=484149k, avg=484148724.00, stdev= 0.00 lat (nsec): min=501800k, max=501800k, avg=501800305.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.34%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.8MiB/s (81.6MB/s), 77.8MiB/s-77.8MiB/s (81.6MB/s-81.6MB/s), io=40.0MiB (41.9MB), run=514-514msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout