== sanity test 398c: run fio to test AIO ================= 06:24:33 (1768562673) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.436289 s, 96.1 MB/s osc.lustre-OST0000-osc-ffff8fe291cf3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182446: Fri Jan 16 06:25:06 2026 write: IOPS=106, BW=424KiB/s (435kB/s)(10.0MiB/24132msec); 0 zone resets slat (usec): min=21, max=10730, avg=162.49, stdev=542.05 clat (msec): min=26, max=309, avg=149.87, stdev=41.42 lat (msec): min=26, max=309, avg=150.03, stdev=41.42 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 96], 10.00th=[ 109], 20.00th=[ 118], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 163], 80.00th=[ 182], 90.00th=[ 213], 95.00th=[ 232], | 99.00th=[ 268], 99.50th=[ 292], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 277, max= 631, per=24.87%, avg=421.98, stdev=89.86, samples=48 iops : min= 69, max= 157, avg=105.17, stdev=22.43, samples=48 lat (msec) : 50=0.16%, 100=5.78%, 250=91.84%, 500=2.23% cpu : usr=0.17%, sys=1.03%, ctx=2776, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182447: Fri Jan 16 06:25:06 2026 write: IOPS=106, BW=425KiB/s (435kB/s)(10.0MiB/24120msec); 0 zone resets slat (usec): min=21, max=14815, avg=169.42, stdev=657.46 clat (msec): min=13, max=305, avg=149.73, stdev=40.84 lat (msec): min=14, max=306, avg=149.90, stdev=40.83 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 94], 10.00th=[ 108], 20.00th=[ 118], | 30.00th=[ 128], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 163], 80.00th=[ 186], 90.00th=[ 209], 95.00th=[ 228], | 99.00th=[ 262], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 305] bw ( KiB/s): min= 167, max= 734, per=24.95%, avg=423.43, stdev=100.03, samples=47 iops : min= 41, max= 183, avg=105.53, stdev=25.07, samples=47 lat (msec) : 20=0.16%, 50=0.20%, 100=6.37%, 250=91.64%, 500=1.64% cpu : usr=0.22%, sys=1.05%, ctx=2645, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182448: Fri Jan 16 06:25:06 2026 write: IOPS=106, BW=425KiB/s (435kB/s)(10.0MiB/24117msec); 0 zone resets slat (usec): min=27, max=13018, avg=195.10, stdev=701.07 clat (msec): min=13, max=342, avg=150.24, stdev=41.69 lat (msec): min=14, max=343, avg=150.44, stdev=41.69 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 97], 10.00th=[ 109], 20.00th=[ 118], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 163], 80.00th=[ 182], 90.00th=[ 207], 95.00th=[ 230], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 342] bw ( KiB/s): min= 254, max= 640, per=25.09%, avg=425.85, stdev=93.57, samples=47 iops : min= 63, max= 160, avg=106.11, stdev=23.35, samples=47 lat (msec) : 20=0.04%, 50=0.12%, 100=6.84%, 250=90.23%, 500=2.77% cpu : usr=0.28%, sys=1.05%, ctx=2891, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182449: Fri Jan 16 06:25:06 2026 write: IOPS=106, BW=425KiB/s (435kB/s)(10.0MiB/24115msec); 0 zone resets slat (usec): min=21, max=20375, avg=171.89, stdev=907.03 clat (msec): min=22, max=355, avg=149.76, stdev=40.90 lat (msec): min=23, max=355, avg=149.93, stdev=40.91 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 118], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 165], 80.00th=[ 182], 90.00th=[ 209], 95.00th=[ 226], | 99.00th=[ 247], 99.50th=[ 326], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 355] bw ( KiB/s): min= 239, max= 638, per=24.99%, avg=424.04, stdev=98.60, samples=47 iops : min= 59, max= 159, avg=105.70, stdev=24.71, samples=47 lat (msec) : 50=0.12%, 100=6.41%, 250=92.66%, 500=0.82% cpu : usr=0.27%, sys=0.92%, ctx=2009, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1697KiB/s (1738kB/s), 424KiB/s-425KiB/s (435kB/s-435kB/s), io=40.0MiB (41.9MB), run=24115-24132msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182463: Fri Jan 16 06:25:41 2026 read: IOPS=48, BW=196KiB/s (200kB/s)(5048KiB/25814msec) slat (usec): min=2713, max=51234, avg=12157.73, stdev=7057.32 clat (msec): min=50, max=297, avg=152.42, stdev=37.76 lat (msec): min=64, max=319, avg=164.58, stdev=39.23 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 100], 10.00th=[ 108], 20.00th=[ 121], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 184], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 75, max= 287, per=25.19%, avg=194.75, stdev=40.67, samples=51 iops : min= 18, max= 71, avg=48.25, stdev=10.20, samples=51 write: IOPS=50, BW=201KiB/s (206kB/s)(5192KiB/25814msec); 0 zone resets slat (usec): min=28, max=11883, avg=190.19, stdev=654.95 clat (msec): min=71, max=277, avg=155.05, stdev=35.17 lat (msec): min=71, max=277, avg=155.25, stdev=35.16 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 186], 90.00th=[ 203], 95.00th=[ 220], | 99.00th=[ 247], 99.50th=[ 262], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 45, max= 285, per=24.95%, avg=200.86, stdev=48.38, samples=51 iops : min= 11, max= 71, avg=49.80, stdev=12.10, samples=51 lat (msec) : 100=4.84%, 250=93.95%, 500=1.21% cpu : usr=0.29%, sys=1.80%, ctx=4187, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182464: Fri Jan 16 06:25:41 2026 read: IOPS=48, BW=194KiB/s (199kB/s)(4976KiB/25651msec) slat (usec): min=40, max=56787, avg=12368.01, stdev=7249.27 clat (msec): min=34, max=358, avg=152.06, stdev=39.78 lat (msec): min=36, max=373, avg=164.44, stdev=41.56 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 93], 10.00th=[ 105], 20.00th=[ 118], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 161], | 70.00th=[ 174], 80.00th=[ 184], 90.00th=[ 201], 95.00th=[ 220], | 99.00th=[ 266], 99.50th=[ 288], 99.90th=[ 342], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 87, max= 287, per=25.07%, avg=193.82, stdev=44.13, samples=50 iops : min= 21, max= 71, avg=48.02, stdev=11.08, samples=50 write: IOPS=51, BW=205KiB/s (210kB/s)(5264KiB/25651msec); 0 zone resets slat (usec): min=31, max=16451, avg=200.23, stdev=781.43 clat (msec): min=58, max=400, avg=155.59, stdev=38.68 lat (msec): min=58, max=400, avg=155.80, stdev=38.67 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 124], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 186], 90.00th=[ 205], 95.00th=[ 220], | 99.00th=[ 257], 99.50th=[ 309], 99.90th=[ 376], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 143, max= 335, per=25.66%, avg=206.60, stdev=43.57, samples=50 iops : min= 35, max= 83, avg=51.24, stdev=10.94, samples=50 lat (msec) : 50=0.12%, 100=6.68%, 250=91.95%, 500=1.25% cpu : usr=0.29%, sys=1.85%, ctx=4339, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182465: Fri Jan 16 06:25:41 2026 read: IOPS=48, BW=192KiB/s (197kB/s)(4984KiB/25931msec) slat (usec): min=2714, max=52771, avg=12194.89, stdev=6931.21 clat (msec): min=55, max=265, avg=150.92, stdev=37.25 lat (msec): min=69, max=283, avg=163.12, stdev=38.90 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 95], 10.00th=[ 105], 20.00th=[ 118], | 30.00th=[ 129], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 182], 90.00th=[ 203], 95.00th=[ 220], | 99.00th=[ 247], 99.50th=[ 257], 99.90th=[ 262], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 60, max= 359, per=24.94%, avg=192.80, stdev=48.48, samples=51 iops : min= 15, max= 89, avg=47.78, stdev=12.09, samples=51 write: IOPS=50, BW=203KiB/s (208kB/s)(5256KiB/25931msec); 0 zone resets slat (usec): min=23, max=11585, avg=188.74, stdev=615.53 clat (msec): min=64, max=288, avg=157.83, stdev=36.17 lat (msec): min=64, max=288, avg=158.02, stdev=36.17 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 105], 10.00th=[ 114], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 207], 95.00th=[ 224], | 99.00th=[ 249], 99.50th=[ 268], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 75, max= 299, per=25.10%, avg=202.06, stdev=41.88, samples=51 iops : min= 18, max= 74, avg=50.08, stdev=10.46, samples=51 lat (msec) : 100=5.55%, 250=93.63%, 500=0.82% cpu : usr=0.22%, sys=1.94%, ctx=4471, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182466: Fri Jan 16 06:25:41 2026 read: IOPS=48, BW=195KiB/s (200kB/s)(5060KiB/25929msec) slat (usec): min=2620, max=64233, avg=12338.56, stdev=7363.71 clat (msec): min=9, max=294, avg=150.93, stdev=38.67 lat (msec): min=17, max=303, avg=163.28, stdev=40.52 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 118], | 30.00th=[ 129], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 157], | 70.00th=[ 169], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 222], | 99.00th=[ 247], 99.50th=[ 253], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 22, max= 325, per=25.15%, avg=194.39, stdev=46.55, samples=51 iops : min= 5, max= 81, avg=48.16, stdev=11.64, samples=51 write: IOPS=49, BW=200KiB/s (205kB/s)(5180KiB/25929msec); 0 zone resets slat (usec): min=24, max=13142, avg=166.89, stdev=537.44 clat (msec): min=37, max=280, avg=156.82, stdev=38.71 lat (msec): min=37, max=280, avg=156.99, stdev=38.70 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 99], 10.00th=[ 112], 20.00th=[ 124], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 211], 95.00th=[ 228], | 99.00th=[ 253], 99.50th=[ 264], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 45, max= 295, per=24.64%, avg=198.39, stdev=44.44, samples=51 iops : min= 11, max= 73, avg=49.16, stdev=11.11, samples=51 lat (msec) : 10=0.04%, 20=0.04%, 50=0.23%, 100=6.29%, 250=92.42% lat (msec) : 500=0.98% cpu : usr=0.26%, sys=1.92%, ctx=4245, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=774KiB/s (792kB/s), 192KiB/s-196KiB/s (197kB/s-200kB/s), io=19.6MiB (20.5MB), run=25651-25931msec WRITE: bw=806KiB/s (825kB/s), 200KiB/s-205KiB/s (205kB/s-210kB/s), io=20.4MiB (21.4MB), run=25651-25931msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182474: Fri Jan 16 06:25:52 2026 read: IOPS=1, BW=68.3MiB/s (71.6MB/s)(40.0MiB/586msec) slat (nsec): min=45342k, max=45342k, avg=45341877.00, stdev= 0.00 clat (nsec): min=520420k, max=520420k, avg=520419600.00, stdev= 0.00 lat (nsec): min=565766k, max=565766k, avg=565765942.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.05%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.3MiB/s (71.6MB/s), 68.3MiB/s-68.3MiB/s (71.6MB/s-71.6MB/s), io=40.0MiB (41.9MB), run=586-586msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout