== sanity test 398c: run fio to test AIO ================= 10:36:41 (1787755001) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.564904 s, 74.2 MB/s osc.lustre-OST0000-osc-ffff9ffcc889d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179491: Wed Aug 26 10:37:28 2026 write: IOPS=74, BW=300KiB/s (307kB/s)(10.0MiB/34144msec); 0 zone resets slat (usec): min=23, max=16587, avg=227.20, stdev=860.69 clat (msec): min=45, max=432, avg=212.90, stdev=54.50 lat (msec): min=45, max=432, avg=213.13, stdev=54.49 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 255], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 435] bw ( KiB/s): min= 164, max= 438, per=25.08%, avg=299.97, stdev=67.54, samples=67 iops : min= 41, max= 109, avg=74.64, stdev=16.88, samples=67 lat (msec) : 50=0.04%, 100=1.25%, 250=77.50%, 500=21.21% cpu : usr=0.20%, sys=0.83%, ctx=2523, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179492: Wed Aug 26 10:37:28 2026 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34221msec); 0 zone resets slat (usec): min=21, max=33505, avg=239.01, stdev=1132.96 clat (msec): min=28, max=399, avg=212.38, stdev=53.89 lat (msec): min=28, max=400, avg=212.62, stdev=53.90 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 384], 99.50th=[ 388], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 401] bw ( KiB/s): min= 151, max= 484, per=24.90%, avg=297.84, stdev=65.62, samples=67 iops : min= 37, max= 121, avg=74.10, stdev=16.44, samples=67 lat (msec) : 50=0.16%, 100=0.94%, 250=78.48%, 500=20.43% cpu : usr=0.17%, sys=0.92%, ctx=2593, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179493: Wed Aug 26 10:37:28 2026 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34221msec); 0 zone resets slat (usec): min=28, max=19066, avg=225.77, stdev=879.78 clat (msec): min=39, max=414, avg=212.58, stdev=53.24 lat (msec): min=39, max=417, avg=212.81, stdev=53.23 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 368], 99.50th=[ 380], 99.90th=[ 393], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 184, max= 456, per=24.86%, avg=297.37, stdev=63.93, samples=67 iops : min= 46, max= 114, avg=73.97, stdev=15.96, samples=67 lat (msec) : 50=0.20%, 100=0.62%, 250=77.97%, 500=21.21% cpu : usr=0.16%, sys=0.91%, ctx=2471, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179494: Wed Aug 26 10:37:28 2026 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34216msec); 0 zone resets slat (usec): min=22, max=14328, avg=153.11, stdev=557.96 clat (msec): min=75, max=397, avg=212.57, stdev=52.17 lat (msec): min=75, max=397, avg=212.72, stdev=52.16 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 144], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 249], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 393], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 128, max= 398, per=24.94%, avg=298.22, stdev=67.11, samples=67 iops : min= 32, max= 99, avg=74.19, stdev=16.78, samples=67 lat (msec) : 100=0.90%, 250=79.30%, 500=19.80% cpu : usr=0.11%, sys=0.60%, ctx=991, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1197KiB/s (1226kB/s), 299KiB/s-300KiB/s (306kB/s-307kB/s), io=40.0MiB (41.9MB), run=34144-34221msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179509: Wed Aug 26 10:38:14 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(5048KiB/36570msec) slat (usec): min=31, max=168544, avg=15859.05, stdev=11554.33 clat (msec): min=26, max=645, avg=218.24, stdev=69.65 lat (msec): min=38, max=669, avg=234.10, stdev=71.82 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 418], 99.50th=[ 600], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 70, max= 254, per=24.89%, avg=136.42, stdev=38.05, samples=72 iops : min= 17, max= 63, avg=33.63, stdev= 9.51, samples=72 write: IOPS=35, BW=142KiB/s (145kB/s)(5192KiB/36570msec); 0 zone resets slat (usec): min=31, max=6640, avg=195.96, stdev=481.95 clat (msec): min=37, max=645, avg=219.94, stdev=56.70 lat (msec): min=38, max=645, avg=220.14, stdev=56.68 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 203], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 247], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 363], 99.50th=[ 376], 99.90th=[ 592], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 16, max= 239, per=24.63%, avg=140.62, stdev=42.46, samples=72 iops : min= 4, max= 59, avg=34.68, stdev=10.62, samples=72 lat (msec) : 50=0.16%, 100=1.17%, 250=72.85%, 500=25.27%, 750=0.55% cpu : usr=0.29%, sys=1.47%, ctx=3680, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179510: Wed Aug 26 10:38:14 2026 read: IOPS=34, BW=138KiB/s (141kB/s)(4976KiB/36127msec) slat (usec): min=39, max=75042, avg=16283.39, stdev=9601.40 clat (msec): min=25, max=553, avg=214.68, stdev=60.72 lat (msec): min=33, max=566, avg=230.97, stdev=63.18 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 131], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 288], 95.00th=[ 321], | 99.00th=[ 384], 99.50th=[ 430], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 42, max= 247, per=25.04%, avg=137.23, stdev=40.67, samples=71 iops : min= 10, max= 61, avg=33.85, stdev=10.20, samples=71 write: IOPS=36, BW=146KiB/s (149kB/s)(5264KiB/36127msec); 0 zone resets slat (usec): min=29, max=10474, avg=226.27, stdev=667.77 clat (msec): min=91, max=508, avg=217.86, stdev=56.49 lat (msec): min=91, max=508, avg=218.09, stdev=56.46 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 245], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 368], 99.50th=[ 405], 99.90th=[ 464], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 72, max= 247, per=25.52%, avg=145.70, stdev=38.60, samples=71 iops : min= 18, max= 61, avg=35.97, stdev= 9.70, samples=71 lat (msec) : 50=0.08%, 100=0.35%, 250=73.36%, 500=26.02%, 750=0.20% cpu : usr=0.25%, sys=1.45%, ctx=3705, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179511: Wed Aug 26 10:38:14 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(4984KiB/36278msec) slat (usec): min=2973, max=77048, avg=16025.51, stdev=9357.34 clat (msec): min=21, max=476, avg=214.13, stdev=58.92 lat (msec): min=31, max=509, avg=230.17, stdev=61.54 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 372], 99.50th=[ 393], 99.90th=[ 435], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 7, max= 256, per=25.01%, avg=137.06, stdev=40.18, samples=71 iops : min= 1, max= 64, avg=33.77, stdev=10.06, samples=71 write: IOPS=36, BW=145KiB/s (148kB/s)(5256KiB/36278msec); 0 zone resets slat (usec): min=30, max=21431, avg=249.35, stdev=920.37 clat (msec): min=52, max=441, avg=218.79, stdev=55.83 lat (msec): min=52, max=442, avg=219.05, stdev=55.83 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 266], 90.00th=[ 296], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 397], 99.90th=[ 426], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 28, max= 262, per=25.23%, avg=144.07, stdev=42.92, samples=71 iops : min= 7, max= 65, avg=35.58, stdev=10.75, samples=71 lat (msec) : 50=0.08%, 100=0.47%, 250=73.95%, 500=25.51% cpu : usr=0.32%, sys=1.61%, ctx=3987, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179512: Wed Aug 26 10:38:14 2026 read: IOPS=34, BW=138KiB/s (142kB/s)(5060KiB/36585msec) slat (usec): min=32, max=198890, avg=16573.84, stdev=12827.55 clat (msec): min=39, max=537, avg=215.74, stdev=61.81 lat (msec): min=52, max=545, avg=232.32, stdev=64.61 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 296], 95.00th=[ 326], | 99.00th=[ 397], 99.50th=[ 468], 99.90th=[ 527], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 48, max= 215, per=25.18%, avg=137.99, stdev=38.48, samples=72 iops : min= 12, max= 53, avg=34.04, stdev= 9.65, samples=72 write: IOPS=35, BW=142KiB/s (145kB/s)(5180KiB/36585msec); 0 zone resets slat (usec): min=33, max=15712, avg=250.87, stdev=947.27 clat (msec): min=75, max=526, avg=224.16, stdev=62.46 lat (msec): min=76, max=526, avg=224.41, stdev=62.42 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 134], 10.00th=[ 155], 20.00th=[ 176], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 230], | 70.00th=[ 249], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 414], 99.50th=[ 489], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 55, max= 238, per=24.57%, avg=140.31, stdev=34.62, samples=72 iops : min= 13, max= 59, avg=34.62, stdev= 8.64, samples=72 lat (msec) : 50=0.08%, 100=0.27%, 250=72.93%, 500=26.33%, 750=0.39% cpu : usr=0.19%, sys=1.66%, ctx=4057, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=549KiB/s (562kB/s), 137KiB/s-138KiB/s (141kB/s-142kB/s), io=19.6MiB (20.5MB), run=36127-36585msec WRITE: bw=571KiB/s (585kB/s), 142KiB/s-146KiB/s (145kB/s-149kB/s), io=20.4MiB (21.4MB), run=36127-36585msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179522: Wed Aug 26 10:38:26 2026 read: IOPS=1, BW=76.8MiB/s (80.5MB/s)(40.0MiB/521msec) slat (nsec): min=31907k, max=31907k, avg=31906639.00, stdev= 0.00 clat (nsec): min=459658k, max=459658k, avg=459657970.00, stdev= 0.00 lat (nsec): min=491569k, max=491569k, avg=491569021.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.19%, sys=2.12%, ctx=3, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.8MiB/s (80.5MB/s), 76.8MiB/s-76.8MiB/s (80.5MB/s-80.5MB/s), io=40.0MiB (41.9MB), run=521-521msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout