== sanity test 398c: run fio to test AIO ================= 21:57:18 (1767322638) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.395013 s, 106 MB/s osc.lustre-OST0000-osc-ffff9eb3450fc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=186273: Thu Jan 1 21:57:47 2026 write: IOPS=117, BW=470KiB/s (481kB/s)(10.0MiB/21807msec); 0 zone resets slat (usec): min=22, max=8025, avg=160.02, stdev=413.34 clat (msec): min=16, max=249, avg=135.99, stdev=32.26 lat (msec): min=17, max=249, avg=136.15, stdev=32.24 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 90], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 117], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 163], 90.00th=[ 178], 95.00th=[ 199], | 99.00th=[ 234], 99.50th=[ 236], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 277, max= 640, per=24.91%, avg=466.86, stdev=88.22, samples=43 iops : min= 69, max= 160, avg=116.35, stdev=22.05, samples=43 lat (msec) : 20=0.04%, 50=0.08%, 100=9.80%, 250=90.08% cpu : usr=0.19%, sys=1.10%, ctx=2246, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186274: Thu Jan 1 21:57:47 2026 write: IOPS=117, BW=469KiB/s (480kB/s)(10.0MiB/21849msec); 0 zone resets slat (usec): min=21, max=16784, avg=151.12, stdev=554.88 clat (msec): min=19, max=251, avg=135.71, stdev=32.98 lat (msec): min=19, max=251, avg=135.86, stdev=32.99 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 111], | 30.00th=[ 118], 40.00th=[ 125], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 197], | 99.00th=[ 245], 99.50th=[ 247], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 251] bw ( KiB/s): min= 256, max= 638, per=24.64%, avg=461.84, stdev=88.50, samples=43 iops : min= 64, max= 159, avg=115.14, stdev=22.09, samples=43 lat (msec) : 20=0.04%, 50=0.90%, 100=10.31%, 250=88.71%, 500=0.04% cpu : usr=0.16%, sys=1.02%, ctx=2233, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186275: Thu Jan 1 21:57:47 2026 write: IOPS=117, BW=469KiB/s (480kB/s)(10.0MiB/21852msec); 0 zone resets slat (usec): min=21, max=7527, avg=170.70, stdev=489.35 clat (msec): min=18, max=269, avg=135.73, stdev=32.46 lat (msec): min=18, max=269, avg=135.90, stdev=32.45 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 89], 10.00th=[ 101], 20.00th=[ 109], | 30.00th=[ 118], 40.00th=[ 127], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 182], 95.00th=[ 194], | 99.00th=[ 236], 99.50th=[ 239], 99.90th=[ 243], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 264, max= 632, per=24.70%, avg=462.88, stdev=85.86, samples=43 iops : min= 66, max= 158, avg=115.40, stdev=21.44, samples=43 lat (msec) : 20=0.04%, 50=0.43%, 100=9.14%, 250=90.31%, 500=0.08% cpu : usr=0.18%, sys=1.09%, ctx=2582, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186276: Thu Jan 1 21:57:47 2026 write: IOPS=117, BW=469KiB/s (480kB/s)(10.0MiB/21846msec); 0 zone resets slat (usec): min=22, max=8837, avg=164.08, stdev=441.85 clat (msec): min=17, max=269, avg=135.63, stdev=33.84 lat (msec): min=18, max=270, avg=135.80, stdev=33.86 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 90], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 194], | 99.00th=[ 243], 99.50th=[ 257], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 278, max= 637, per=24.71%, avg=463.12, stdev=85.44, samples=43 iops : min= 69, max= 159, avg=115.40, stdev=21.38, samples=43 lat (msec) : 20=0.04%, 50=0.98%, 100=8.75%, 250=89.61%, 500=0.62% cpu : usr=0.22%, sys=1.08%, ctx=2847, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1874KiB/s (1919kB/s), 469KiB/s-470KiB/s (480kB/s-481kB/s), io=40.0MiB (41.9MB), run=21807-21852msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=186289: Thu Jan 1 21:58:12 2026 read: IOPS=66, BW=264KiB/s (271kB/s)(5048KiB/19088msec) slat (usec): min=1545, max=152809, avg=8564.95, stdev=5912.25 clat (msec): min=31, max=250, avg=112.45, stdev=28.61 lat (msec): min=53, max=264, avg=121.02, stdev=29.70 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 75], 10.00th=[ 81], 20.00th=[ 90], | 30.00th=[ 96], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 114], | 70.00th=[ 123], 80.00th=[ 133], 90.00th=[ 153], 95.00th=[ 167], | 99.00th=[ 199], 99.50th=[ 218], 99.90th=[ 241], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 81, max= 352, per=25.04%, avg=263.14, stdev=53.18, samples=37 iops : min= 20, max= 88, avg=65.43, stdev=13.36, samples=37 write: IOPS=68, BW=272KiB/s (279kB/s)(5192KiB/19088msec); 0 zone resets slat (usec): min=24, max=7416, avg=166.63, stdev=376.34 clat (msec): min=40, max=245, avg=115.20, stdev=27.13 lat (msec): min=40, max=245, avg=115.37, stdev=27.14 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 83], 10.00th=[ 87], 20.00th=[ 93], | 30.00th=[ 99], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 134], 90.00th=[ 155], 95.00th=[ 171], | 99.00th=[ 201], 99.50th=[ 211], 99.90th=[ 234], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 94, max= 352, per=24.72%, avg=270.43, stdev=54.29, samples=37 iops : min= 23, max= 88, avg=67.24, stdev=13.59, samples=37 lat (msec) : 50=0.12%, 100=34.45%, 250=65.39%, 500=0.04% cpu : usr=0.28%, sys=2.18%, ctx=4397, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186290: Thu Jan 1 21:58:12 2026 read: IOPS=65, BW=262KiB/s (269kB/s)(4976KiB/18977msec) slat (usec): min=44, max=35707, avg=8464.22, stdev=4478.07 clat (msec): min=21, max=230, avg=111.46, stdev=26.58 lat (msec): min=46, max=238, avg=119.93, stdev=27.53 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 77], 10.00th=[ 82], 20.00th=[ 90], | 30.00th=[ 96], 40.00th=[ 103], 50.00th=[ 107], 60.00th=[ 114], | 70.00th=[ 122], 80.00th=[ 131], 90.00th=[ 148], 95.00th=[ 163], | 99.00th=[ 192], 99.50th=[ 197], 99.90th=[ 213], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 135, max= 383, per=24.88%, avg=261.51, stdev=57.33, samples=37 iops : min= 33, max= 95, avg=65.03, stdev=14.45, samples=37 write: IOPS=69, BW=277KiB/s (284kB/s)(5264KiB/18977msec); 0 zone resets slat (usec): min=23, max=8369, avg=210.21, stdev=634.24 clat (msec): min=45, max=213, avg=115.43, stdev=25.61 lat (msec): min=45, max=213, avg=115.65, stdev=25.64 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 82], 10.00th=[ 88], 20.00th=[ 95], | 30.00th=[ 101], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 116], | 70.00th=[ 124], 80.00th=[ 136], 90.00th=[ 153], 95.00th=[ 165], | 99.00th=[ 194], 99.50th=[ 203], 99.90th=[ 211], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 190, max= 382, per=25.32%, avg=276.97, stdev=49.16, samples=37 iops : min= 47, max= 95, avg=68.89, stdev=12.31, samples=37 lat (msec) : 50=0.16%, 100=32.11%, 250=67.73% cpu : usr=0.34%, sys=2.34%, ctx=4238, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186291: Thu Jan 1 21:58:12 2026 read: IOPS=65, BW=261KiB/s (267kB/s)(4984KiB/19080msec) slat (usec): min=52, max=44978, avg=8607.32, stdev=4825.50 clat (msec): min=13, max=218, avg=112.10, stdev=26.80 lat (msec): min=14, max=227, avg=120.71, stdev=27.87 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 74], 10.00th=[ 81], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 103], 50.00th=[ 109], 60.00th=[ 114], | 70.00th=[ 123], 80.00th=[ 134], 90.00th=[ 148], 95.00th=[ 165], | 99.00th=[ 184], 99.50th=[ 190], 99.90th=[ 213], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 151, max= 357, per=24.81%, avg=260.73, stdev=52.46, samples=37 iops : min= 37, max= 89, avg=64.84, stdev=13.18, samples=37 write: IOPS=68, BW=275KiB/s (282kB/s)(5256KiB/19080msec); 0 zone resets slat (usec): min=25, max=8493, avg=186.67, stdev=502.43 clat (msec): min=37, max=210, avg=117.32, stdev=26.27 lat (msec): min=37, max=213, avg=117.50, stdev=26.32 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 83], 10.00th=[ 88], 20.00th=[ 96], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 114], 60.00th=[ 120], | 70.00th=[ 128], 80.00th=[ 138], 90.00th=[ 157], 95.00th=[ 169], | 99.00th=[ 188], 99.50th=[ 199], 99.90th=[ 207], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 190, max= 364, per=24.90%, avg=272.35, stdev=52.95, samples=37 iops : min= 47, max= 91, avg=67.73, stdev=13.31, samples=37 lat (msec) : 20=0.08%, 50=0.12%, 100=31.17%, 250=68.63% cpu : usr=0.21%, sys=2.45%, ctx=4756, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186292: Thu Jan 1 21:58:12 2026 read: IOPS=66, BW=265KiB/s (272kB/s)(5060KiB/19082msec) slat (usec): min=1932, max=35671, avg=8621.08, stdev=4717.98 clat (msec): min=16, max=207, avg=112.50, stdev=27.35 lat (msec): min=22, max=221, avg=121.13, stdev=28.57 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 77], 10.00th=[ 83], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 114], | 70.00th=[ 123], 80.00th=[ 132], 90.00th=[ 150], 95.00th=[ 169], | 99.00th=[ 190], 99.50th=[ 194], 99.90th=[ 203], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 135, max= 333, per=25.08%, avg=263.62, stdev=44.34, samples=37 iops : min= 33, max= 83, avg=65.54, stdev=11.18, samples=37 write: IOPS=67, BW=271KiB/s (278kB/s)(5180KiB/19082msec); 0 zone resets slat (usec): min=23, max=18350, avg=183.33, stdev=654.84 clat (msec): min=26, max=210, avg=115.67, stdev=25.35 lat (msec): min=26, max=210, avg=115.86, stdev=25.38 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 82], 10.00th=[ 87], 20.00th=[ 95], | 30.00th=[ 101], 40.00th=[ 106], 50.00th=[ 112], 60.00th=[ 117], | 70.00th=[ 126], 80.00th=[ 136], 90.00th=[ 153], 95.00th=[ 167], | 99.00th=[ 184], 99.50th=[ 192], 99.90th=[ 205], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 162, max= 399, per=24.60%, avg=269.16, stdev=50.88, samples=37 iops : min= 40, max= 99, avg=66.92, stdev=12.78, samples=37 lat (msec) : 20=0.04%, 50=0.12%, 100=32.11%, 250=67.73% cpu : usr=0.28%, sys=2.29%, ctx=4391, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1051KiB/s (1077kB/s), 261KiB/s-265KiB/s (267kB/s-272kB/s), io=19.6MiB (20.5MB), run=18977-19088msec WRITE: bw=1095KiB/s (1121kB/s), 271KiB/s-277KiB/s (278kB/s-284kB/s), io=20.4MiB (21.4MB), run=18977-19088msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=186299: Thu Jan 1 21:58:18 2026 read: IOPS=4, BW=160MiB/s (168MB/s)(40.0MiB/250msec) slat (nsec): min=14049k, max=14049k, avg=14048796.00, stdev= 0.00 clat (nsec): min=226479k, max=226479k, avg=226478823.00, stdev= 0.00 lat (nsec): min=240532k, max=240532k, avg=240531629.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.82%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=40.0MiB (41.9MB), run=250-250msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout