== sanity test 398c: run fio to test AIO ================= 11:08:50 (1788361730) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.185515 s, 226 MB/s osc.lustre-OST0000-osc-ffffa01104fab800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151478: Wed Sep 2 11:10:05 2026 write: IOPS=36, BW=145KiB/s (148kB/s)(10.0MiB/70667msec); 0 zone resets slat (usec): min=35, max=7097, avg=258.41, stdev=358.31 clat (msec): min=111, max=953, avg=440.99, stdev=148.56 lat (msec): min=111, max=953, avg=441.25, stdev=148.58 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 241], 10.00th=[ 288], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 409], 60.00th=[ 447], | 70.00th=[ 498], 80.00th=[ 558], 90.00th=[ 651], 95.00th=[ 735], | 99.00th=[ 852], 99.50th=[ 894], 99.90th=[ 927], 99.95th=[ 944], | 99.99th=[ 953] bw ( KiB/s): min= 55, max= 370, per=24.91%, avg=144.25, stdev=48.81, samples=140 iops : min= 13, max= 92, avg=35.78, stdev=12.25, samples=140 lat (msec) : 250=5.51%, 500=64.77%, 750=25.51%, 1000=4.22% cpu : usr=0.07%, sys=0.61%, ctx=3084, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151479: Wed Sep 2 11:10:05 2026 write: IOPS=36, BW=145KiB/s (148kB/s)(10.0MiB/70711msec); 0 zone resets slat (usec): min=36, max=4898, avg=253.35, stdev=301.27 clat (msec): min=108, max=959, avg=441.26, stdev=147.99 lat (msec): min=108, max=959, avg=441.51, stdev=148.01 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 245], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 384], 50.00th=[ 414], 60.00th=[ 447], | 70.00th=[ 498], 80.00th=[ 558], 90.00th=[ 651], 95.00th=[ 743], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 919], 99.95th=[ 927], | 99.99th=[ 961] bw ( KiB/s): min= 55, max= 350, per=24.89%, avg=144.11, stdev=49.72, samples=140 iops : min= 13, max= 87, avg=35.74, stdev=12.49, samples=140 lat (msec) : 250=5.20%, 500=65.70%, 750=24.49%, 1000=4.61% cpu : usr=0.09%, sys=0.60%, ctx=3076, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151480: Wed Sep 2 11:10:05 2026 write: IOPS=36, BW=145KiB/s (149kB/s)(10.0MiB/70539msec); 0 zone resets slat (usec): min=38, max=4152, avg=242.37, stdev=287.07 clat (msec): min=104, max=953, avg=440.13, stdev=149.19 lat (msec): min=104, max=954, avg=440.37, stdev=149.20 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 239], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 380], 50.00th=[ 414], 60.00th=[ 443], | 70.00th=[ 498], 80.00th=[ 558], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 936], 99.95th=[ 944], | 99.99th=[ 953] bw ( KiB/s): min= 48, max= 415, per=24.93%, avg=144.34, stdev=52.07, samples=140 iops : min= 12, max= 103, avg=35.80, stdev=13.05, samples=140 lat (msec) : 250=5.62%, 500=64.88%, 750=25.59%, 1000=3.91% cpu : usr=0.09%, sys=0.55%, ctx=2883, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151481: Wed Sep 2 11:10:05 2026 write: IOPS=36, BW=145KiB/s (148kB/s)(10.0MiB/70688msec); 0 zone resets slat (usec): min=41, max=5738, avg=245.39, stdev=314.94 clat (msec): min=42, max=986, avg=441.32, stdev=148.48 lat (msec): min=43, max=986, avg=441.57, stdev=148.48 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 241], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 384], 50.00th=[ 414], 60.00th=[ 456], | 70.00th=[ 498], 80.00th=[ 558], 90.00th=[ 651], 95.00th=[ 735], | 99.00th=[ 860], 99.50th=[ 894], 99.90th=[ 961], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 63, max= 396, per=24.89%, avg=144.09, stdev=50.22, samples=140 iops : min= 15, max= 99, avg=35.74, stdev=12.61, samples=140 lat (msec) : 50=0.08%, 100=0.39%, 250=4.96%, 500=65.59%, 750=24.84% lat (msec) : 1000=4.14% cpu : usr=0.08%, sys=0.59%, ctx=3008, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=579KiB/s (593kB/s), 145KiB/s-145KiB/s (148kB/s-149kB/s), io=40.0MiB (41.9MB), run=70539-70711msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151501: Wed Sep 2 11:10:52 2026 read: IOPS=28, BW=116KiB/s (119kB/s)(5048KiB/43588msec) slat (usec): min=1945, max=15832, avg=5117.48, stdev=1953.68 clat (msec): min=7, max=319, avg=171.67, stdev=36.20 lat (msec): min=10, max=323, avg=176.79, stdev=36.51 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 118], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 220], 95.00th=[ 232], | 99.00th=[ 251], 99.50th=[ 262], 99.90th=[ 305], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 39, max= 199, per=25.12%, avg=115.57, stdev=31.86, samples=86 iops : min= 9, max= 49, avg=28.60, stdev= 8.02, samples=86 write: IOPS=29, BW=119KiB/s (122kB/s)(5192KiB/43588msec); 0 zone resets slat (usec): min=54, max=4534, avg=267.76, stdev=298.58 clat (msec): min=120, max=677, avg=364.36, stdev=87.30 lat (msec): min=120, max=677, avg=364.63, stdev=87.30 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 222], 10.00th=[ 253], 20.00th=[ 292], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 363], 60.00th=[ 384], | 70.00th=[ 414], 80.00th=[ 439], 90.00th=[ 468], 95.00th=[ 506], | 99.00th=[ 592], 99.50th=[ 625], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 64, max= 176, per=24.75%, avg=118.57, stdev=26.04, samples=86 iops : min= 16, max= 44, avg=29.35, stdev= 6.51, samples=86 lat (msec) : 10=0.04%, 50=0.35%, 100=0.62%, 250=52.30%, 500=43.98% lat (msec) : 750=2.70% cpu : usr=0.15%, sys=1.28%, ctx=5173, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151502: Wed Sep 2 11:10:52 2026 read: IOPS=28, BW=114KiB/s (117kB/s)(4976KiB/43564msec) slat (usec): min=1965, max=16631, avg=5131.25, stdev=2004.72 clat (msec): min=6, max=314, avg=172.02, stdev=37.50 lat (msec): min=10, max=321, avg=177.16, stdev=37.91 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 234], | 99.00th=[ 264], 99.50th=[ 279], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 40, max= 215, per=24.81%, avg=114.12, stdev=33.82, samples=86 iops : min= 10, max= 53, avg=28.23, stdev= 8.52, samples=86 write: IOPS=30, BW=121KiB/s (124kB/s)(5264KiB/43564msec); 0 zone resets slat (usec): min=56, max=3399, avg=257.75, stdev=255.47 clat (msec): min=138, max=710, avg=361.14, stdev=88.42 lat (msec): min=138, max=710, avg=361.40, stdev=88.43 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 215], 10.00th=[ 249], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 384], | 70.00th=[ 409], 80.00th=[ 435], 90.00th=[ 468], 95.00th=[ 502], | 99.00th=[ 600], 99.50th=[ 617], 99.90th=[ 701], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 72, max= 199, per=25.15%, avg=120.45, stdev=28.30, samples=86 iops : min= 18, max= 49, avg=29.83, stdev= 7.07, samples=86 lat (msec) : 10=0.12%, 20=0.04%, 50=0.23%, 100=0.59%, 250=52.19% lat (msec) : 500=44.14%, 750=2.70% cpu : usr=0.17%, sys=1.27%, ctx=5157, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151503: Wed Sep 2 11:10:52 2026 read: IOPS=28, BW=114KiB/s (117kB/s)(4984KiB/43563msec) slat (usec): min=63, max=17521, avg=5138.15, stdev=2004.72 clat (msec): min=5, max=557, avg=170.55, stdev=42.63 lat (msec): min=11, max=560, avg=175.70, stdev=43.09 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 112], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 234], | 99.00th=[ 264], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 56, max= 216, per=25.00%, avg=114.99, stdev=36.60, samples=86 iops : min= 14, max= 54, avg=28.45, stdev= 9.19, samples=86 write: IOPS=30, BW=121KiB/s (124kB/s)(5256KiB/43563msec); 0 zone resets slat (usec): min=56, max=4938, avg=253.10, stdev=292.28 clat (msec): min=68, max=951, avg=363.03, stdev=95.85 lat (msec): min=68, max=951, avg=363.28, stdev=95.86 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 215], 10.00th=[ 251], 20.00th=[ 288], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 384], | 70.00th=[ 405], 80.00th=[ 435], 90.00th=[ 468], 95.00th=[ 506], | 99.00th=[ 617], 99.50th=[ 776], 99.90th=[ 877], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 48, max= 216, per=25.05%, avg=119.99, stdev=29.14, samples=86 iops : min= 12, max= 54, avg=29.71, stdev= 7.31, samples=86 lat (msec) : 10=0.16%, 20=0.23%, 50=0.62%, 100=1.09%, 250=50.70% lat (msec) : 500=44.18%, 750=2.70%, 1000=0.31% cpu : usr=0.16%, sys=1.26%, ctx=5038, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151504: Wed Sep 2 11:10:52 2026 read: IOPS=29, BW=116KiB/s (119kB/s)(5060KiB/43562msec) slat (usec): min=1799, max=14600, avg=5132.99, stdev=1914.74 clat (msec): min=5, max=431, avg=173.23, stdev=40.20 lat (msec): min=9, max=436, avg=178.37, stdev=40.53 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 116], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 236], | 99.00th=[ 271], 99.50th=[ 305], 99.90th=[ 393], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 39, max= 199, per=25.27%, avg=116.24, stdev=33.98, samples=86 iops : min= 9, max= 49, avg=28.77, stdev= 8.54, samples=86 write: IOPS=29, BW=119KiB/s (122kB/s)(5180KiB/43562msec); 0 zone resets slat (usec): min=56, max=6867, avg=270.03, stdev=315.28 clat (msec): min=67, max=716, avg=362.79, stdev=84.17 lat (msec): min=68, max=716, avg=363.07, stdev=84.18 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 228], 10.00th=[ 257], 20.00th=[ 296], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 363], 60.00th=[ 384], | 70.00th=[ 409], 80.00th=[ 430], 90.00th=[ 460], 95.00th=[ 498], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 684], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 72, max= 191, per=24.73%, avg=118.48, stdev=26.54, samples=86 iops : min= 18, max= 47, avg=29.33, stdev= 6.63, samples=86 lat (msec) : 10=0.04%, 20=0.16%, 50=0.27%, 100=0.66%, 250=51.33% lat (msec) : 500=45.23%, 750=2.30% cpu : usr=0.15%, sys=1.27%, ctx=5061, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=460KiB/s (471kB/s), 114KiB/s-116KiB/s (117kB/s-119kB/s), io=19.6MiB (20.5MB), run=43562-43588msec WRITE: bw=479KiB/s (491kB/s), 119KiB/s-121KiB/s (122kB/s-124kB/s), io=20.4MiB (21.4MB), run=43562-43588msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151514: Wed Sep 2 11:10:57 2026 read: IOPS=4, BW=187MiB/s (196MB/s)(40.0MiB/214msec) slat (nsec): min=14215k, max=14215k, avg=14214731.00, stdev= 0.00 clat (nsec): min=182091k, max=182091k, avg=182091262.00, stdev= 0.00 lat (nsec): min=196311k, max=196311k, avg=196310988.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.10%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=187MiB/s (196MB/s), 187MiB/s-187MiB/s (196MB/s-196MB/s), io=40.0MiB (41.9MB), run=214-214msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout