== sanity test 398c: run fio to test AIO ================= 16:32:13 (1788467533) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.433434 s, 96.8 MB/s osc.lustre-OST0000-osc-ffff8a8b599bb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179430: Thu Sep 3 16:32:43 2026 write: IOPS=116, BW=466KiB/s (477kB/s)(10.0MiB/21982msec); 0 zone resets slat (usec): min=22, max=12035, avg=204.08, stdev=678.13 clat (msec): min=23, max=275, avg=136.73, stdev=35.32 lat (msec): min=23, max=275, avg=136.93, stdev=35.29 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 90], 10.00th=[ 103], 20.00th=[ 110], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 163], 90.00th=[ 184], 95.00th=[ 211], | 99.00th=[ 247], 99.50th=[ 268], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 256, max= 664, per=25.00%, avg=465.53, stdev=87.66, samples=43 iops : min= 64, max= 166, avg=116.12, stdev=22.01, samples=43 lat (msec) : 50=0.35%, 100=7.50%, 250=91.33%, 500=0.82% cpu : usr=0.20%, sys=1.14%, ctx=2557, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179431: Thu Sep 3 16:32:43 2026 write: IOPS=116, BW=467KiB/s (478kB/s)(10.0MiB/21941msec); 0 zone resets slat (usec): min=22, max=11153, avg=168.87, stdev=556.08 clat (msec): min=16, max=268, avg=136.84, stdev=35.79 lat (msec): min=16, max=268, avg=137.01, stdev=35.80 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 92], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 116], 40.00th=[ 124], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 163], 90.00th=[ 182], 95.00th=[ 218], | 99.00th=[ 249], 99.50th=[ 253], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 271] bw ( KiB/s): min= 256, max= 640, per=25.00%, avg=465.47, stdev=90.20, samples=43 iops : min= 64, max= 160, avg=116.14, stdev=22.64, samples=43 lat (msec) : 20=0.04%, 50=0.08%, 100=10.23%, 250=88.67%, 500=0.98% cpu : usr=0.20%, sys=1.09%, ctx=1993, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179432: Thu Sep 3 16:32:43 2026 write: IOPS=116, BW=466KiB/s (477kB/s)(10.0MiB/21992msec); 0 zone resets slat (usec): min=22, max=23488, avg=194.63, stdev=688.43 clat (msec): min=17, max=276, avg=136.69, stdev=35.87 lat (msec): min=19, max=276, avg=136.88, stdev=35.86 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 110], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 186], 95.00th=[ 209], | 99.00th=[ 251], 99.50th=[ 255], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 256, max= 664, per=24.83%, avg=462.37, stdev=85.90, samples=43 iops : min= 64, max= 166, avg=115.33, stdev=21.59, samples=43 lat (msec) : 20=0.04%, 50=0.31%, 100=8.98%, 250=89.53%, 500=1.13% cpu : usr=0.19%, sys=1.34%, ctx=2985, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179433: Thu Sep 3 16:32:43 2026 write: IOPS=116, BW=466KiB/s (478kB/s)(10.0MiB/21958msec); 0 zone resets slat (usec): min=21, max=11991, avg=215.13, stdev=687.24 clat (msec): min=18, max=271, avg=136.68, stdev=36.07 lat (msec): min=18, max=271, avg=136.90, stdev=36.09 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 110], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 163], 90.00th=[ 184], 95.00th=[ 211], | 99.00th=[ 247], 99.50th=[ 257], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 280, max= 688, per=25.05%, avg=466.49, stdev=83.53, samples=43 iops : min= 70, max= 172, avg=116.35, stdev=21.00, samples=43 lat (msec) : 20=0.04%, 50=0.35%, 100=10.86%, 250=87.93%, 500=0.82% cpu : usr=0.29%, sys=1.34%, ctx=2757, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1862KiB/s (1907kB/s), 466KiB/s-467KiB/s (477kB/s-478kB/s), io=40.0MiB (41.9MB), run=21941-21992msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179446: Thu Sep 3 16:33:13 2026 read: IOPS=53, BW=215KiB/s (220kB/s)(5048KiB/23502msec) slat (usec): min=2285, max=52877, avg=10252.45, stdev=5856.92 clat (msec): min=36, max=318, avg=138.24, stdev=40.44 lat (msec): min=51, max=330, avg=148.50, stdev=42.28 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 87], 10.00th=[ 92], 20.00th=[ 104], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 133], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 167], 90.00th=[ 188], 95.00th=[ 209], | 99.00th=[ 275], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 111, max= 327, per=25.08%, avg=213.89, stdev=58.04, samples=46 iops : min= 27, max= 81, avg=53.00, stdev=14.60, samples=46 write: IOPS=55, BW=221KiB/s (226kB/s)(5192KiB/23502msec); 0 zone resets slat (usec): min=23, max=11377, avg=219.67, stdev=655.32 clat (msec): min=36, max=361, avg=143.05, stdev=42.97 lat (msec): min=36, max=362, avg=143.27, stdev=42.96 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 91], 10.00th=[ 96], 20.00th=[ 108], | 30.00th=[ 118], 40.00th=[ 128], 50.00th=[ 136], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 197], 95.00th=[ 226], | 99.00th=[ 288], 99.50th=[ 309], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 103, max= 403, per=24.88%, avg=220.93, stdev=54.99, samples=46 iops : min= 25, max= 100, avg=54.76, stdev=13.82, samples=46 lat (msec) : 50=0.08%, 100=14.80%, 250=83.01%, 500=2.11% cpu : usr=0.31%, sys=2.25%, ctx=4378, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179447: Thu Sep 3 16:33:13 2026 read: IOPS=53, BW=212KiB/s (218kB/s)(4976KiB/23427msec) slat (usec): min=2107, max=84527, avg=10374.19, stdev=6178.35 clat (msec): min=15, max=494, avg=138.93, stdev=45.52 lat (msec): min=26, max=515, avg=149.31, stdev=47.64 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 103], | 30.00th=[ 113], 40.00th=[ 123], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 194], 95.00th=[ 213], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 472], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 111, max= 326, per=24.92%, avg=212.61, stdev=52.41, samples=46 iops : min= 27, max= 81, avg=52.70, stdev=13.14, samples=46 write: IOPS=56, BW=225KiB/s (230kB/s)(5264KiB/23427msec); 0 zone resets slat (usec): min=26, max=16969, avg=211.89, stdev=664.39 clat (msec): min=23, max=319, avg=142.15, stdev=40.38 lat (msec): min=23, max=319, avg=142.37, stdev=40.37 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 125], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 222], | 99.00th=[ 264], 99.50th=[ 284], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 103, max= 351, per=25.44%, avg=225.87, stdev=58.25, samples=46 iops : min= 25, max= 87, avg=56.00, stdev=14.71, samples=46 lat (msec) : 20=0.04%, 50=0.23%, 100=14.26%, 250=83.40%, 500=2.07% cpu : usr=0.30%, sys=2.21%, ctx=4460, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179448: Thu Sep 3 16:33:13 2026 read: IOPS=53, BW=212KiB/s (217kB/s)(4984KiB/23470msec) slat (usec): min=2127, max=86885, avg=10249.68, stdev=6284.53 clat (msec): min=58, max=299, avg=137.93, stdev=40.21 lat (msec): min=64, max=317, avg=148.19, stdev=42.36 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 85], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 192], 95.00th=[ 211], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 87, max= 335, per=24.95%, avg=212.83, stdev=61.22, samples=46 iops : min= 21, max= 83, avg=52.74, stdev=15.40, samples=46 write: IOPS=55, BW=224KiB/s (229kB/s)(5256KiB/23470msec); 0 zone resets slat (usec): min=25, max=11803, avg=202.66, stdev=570.58 clat (msec): min=46, max=328, avg=143.56, stdev=39.91 lat (msec): min=46, max=329, avg=143.76, stdev=39.91 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 220], | 99.00th=[ 259], 99.50th=[ 279], 99.90th=[ 317], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 134, max= 340, per=25.14%, avg=223.28, stdev=48.81, samples=46 iops : min= 33, max= 85, avg=55.37, stdev=12.33, samples=46 lat (msec) : 50=0.04%, 100=14.06%, 250=84.38%, 500=1.52% cpu : usr=0.28%, sys=2.31%, ctx=4543, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179449: Thu Sep 3 16:33:13 2026 read: IOPS=53, BW=216KiB/s (221kB/s)(5060KiB/23428msec) slat (usec): min=52, max=82992, avg=10405.39, stdev=6204.87 clat (msec): min=17, max=338, avg=136.63, stdev=39.85 lat (msec): min=25, max=350, avg=147.04, stdev=41.77 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 104], | 30.00th=[ 112], 40.00th=[ 123], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 155], 80.00th=[ 167], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 119, max= 348, per=25.42%, avg=216.80, stdev=55.26, samples=46 iops : min= 29, max= 87, avg=53.74, stdev=13.99, samples=46 write: IOPS=55, BW=221KiB/s (226kB/s)(5180KiB/23428msec); 0 zone resets slat (usec): min=26, max=6480, avg=202.29, stdev=475.78 clat (msec): min=16, max=426, avg=145.28, stdev=42.68 lat (msec): min=16, max=427, avg=145.48, stdev=42.67 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 110], | 30.00th=[ 120], 40.00th=[ 129], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 194], 95.00th=[ 228], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 388], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 136, max= 366, per=24.85%, avg=220.65, stdev=49.10, samples=46 iops : min= 34, max= 91, avg=54.72, stdev=12.33, samples=46 lat (msec) : 20=0.08%, 50=0.12%, 100=12.97%, 250=84.69%, 500=2.15% cpu : usr=0.32%, sys=2.41%, ctx=4735, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=854KiB/s (874kB/s), 212KiB/s-216KiB/s (217kB/s-221kB/s), io=19.6MiB (20.5MB), run=23427-23502msec WRITE: bw=889KiB/s (910kB/s), 221KiB/s-225KiB/s (226kB/s-230kB/s), io=20.4MiB (21.4MB), run=23427-23502msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179457: Thu Sep 3 16:33:23 2026 read: IOPS=2, BW=92.6MiB/s (97.1MB/s)(40.0MiB/432msec) slat (nsec): min=50112k, max=50112k, avg=50111685.00, stdev= 0.00 clat (nsec): min=345670k, max=345670k, avg=345669893.00, stdev= 0.00 lat (nsec): min=395816k, max=395816k, avg=395815520.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.39%, ctx=5, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=92.6MiB/s (97.1MB/s), 92.6MiB/s-92.6MiB/s (97.1MB/s-97.1MB/s), io=40.0MiB (41.9MB), run=432-432msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout