== sanity test 398c: run fio to test AIO ================= 06:04:20 (1768129460) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.150361 s, 279 MB/s osc.lustre-OST0000-osc-ffff9aac80dca000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=158915: Sun Jan 11 06:04:57 2026 write: IOPS=76, BW=307KiB/s (315kB/s)(10.0MiB/33312msec); 0 zone resets slat (usec): min=24, max=601, avg=141.47, stdev=66.25 clat (msec): min=64, max=353, avg=207.75, stdev=35.80 lat (msec): min=64, max=353, avg=207.89, stdev=35.80 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 209], 60.00th=[ 218], | 70.00th=[ 226], 80.00th=[ 239], 90.00th=[ 253], 95.00th=[ 266], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 305], 99.95th=[ 321], | 99.99th=[ 355] bw ( KiB/s): min= 232, max= 428, per=24.97%, avg=306.85, stdev=41.18, samples=66 iops : min= 58, max= 107, avg=76.71, stdev=10.29, samples=66 lat (msec) : 100=0.70%, 250=87.46%, 500=11.84% cpu : usr=0.15%, sys=1.01%, ctx=2781, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158916: Sun Jan 11 06:04:57 2026 write: IOPS=76, BW=307KiB/s (315kB/s)(10.0MiB/33313msec); 0 zone resets slat (usec): min=24, max=613, avg=135.93, stdev=70.55 clat (msec): min=46, max=368, avg=207.86, stdev=37.57 lat (msec): min=46, max=368, avg=208.00, stdev=37.58 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 192], 40.00th=[ 201], 50.00th=[ 209], 60.00th=[ 218], | 70.00th=[ 228], 80.00th=[ 239], 90.00th=[ 251], 95.00th=[ 266], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 248, max= 511, per=24.94%, avg=306.53, stdev=45.14, samples=66 iops : min= 62, max= 127, avg=76.62, stdev=11.23, samples=66 lat (msec) : 50=0.62%, 100=0.04%, 250=87.97%, 500=11.37% cpu : usr=0.14%, sys=0.97%, ctx=2708, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158917: Sun Jan 11 06:04:57 2026 write: IOPS=76, BW=308KiB/s (315kB/s)(10.0MiB/33255msec); 0 zone resets slat (usec): min=25, max=660, avg=133.67, stdev=69.92 clat (msec): min=37, max=427, avg=207.58, stdev=38.36 lat (msec): min=37, max=427, avg=207.72, stdev=38.37 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 144], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 207], 60.00th=[ 215], | 70.00th=[ 228], 80.00th=[ 241], 90.00th=[ 255], 95.00th=[ 266], | 99.00th=[ 284], 99.50th=[ 296], 99.90th=[ 372], 99.95th=[ 414], | 99.99th=[ 426] bw ( KiB/s): min= 240, max= 511, per=24.94%, avg=306.53, stdev=49.15, samples=66 iops : min= 60, max= 127, avg=76.62, stdev=12.24, samples=66 lat (msec) : 50=0.62%, 100=0.66%, 250=85.82%, 500=12.89% cpu : usr=0.16%, sys=0.96%, ctx=2479, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=158918: Sun Jan 11 06:04:57 2026 write: IOPS=76, BW=307KiB/s (315kB/s)(10.0MiB/33313msec); 0 zone resets slat (usec): min=37, max=437, avg=140.84, stdev=65.85 clat (msec): min=44, max=367, avg=207.81, stdev=36.68 lat (msec): min=44, max=368, avg=207.95, stdev=36.69 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 190], 40.00th=[ 201], 50.00th=[ 209], 60.00th=[ 218], | 70.00th=[ 226], 80.00th=[ 239], 90.00th=[ 253], 95.00th=[ 268], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 355], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 240, max= 472, per=24.95%, avg=306.67, stdev=43.58, samples=66 iops : min= 60, max= 118, avg=76.67, stdev=10.89, samples=66 lat (msec) : 50=0.04%, 100=0.94%, 250=87.38%, 500=11.64% cpu : usr=0.14%, sys=1.00%, ctx=2840, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1230KiB/s (1259kB/s), 307KiB/s-308KiB/s (315kB/s-315kB/s), io=40.0MiB (41.9MB), run=33255-33313msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=158932: Sun Jan 11 06:05:20 2026 read: IOPS=62, BW=251KiB/s (257kB/s)(5048KiB/20121msec) slat (usec): min=918, max=8476, avg=2560.04, stdev=924.98 clat (msec): min=4, max=127, avg=79.74, stdev=15.68 lat (msec): min=6, max=130, avg=82.30, stdev=15.81 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 105], | 99.00th=[ 118], 99.50th=[ 122], 99.90th=[ 127], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 144, max= 368, per=25.15%, avg=250.77, stdev=53.68, samples=39 iops : min= 36, max= 92, avg=62.64, stdev=13.43, samples=39 write: IOPS=64, BW=258KiB/s (264kB/s)(5192KiB/20121msec); 0 zone resets slat (usec): min=34, max=1427, avg=109.25, stdev=78.96 clat (msec): min=60, max=277, avg=167.28, stdev=41.32 lat (msec): min=60, max=277, avg=167.39, stdev=41.32 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 205], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 259], 99.50th=[ 266], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 200, max= 360, per=24.79%, avg=257.31, stdev=39.89, samples=39 iops : min= 50, max= 90, avg=64.28, stdev= 9.98, samples=39 lat (msec) : 10=0.04%, 20=0.47%, 50=0.47%, 100=45.90%, 250=51.76% lat (msec) : 500=1.37% cpu : usr=0.30%, sys=1.62%, ctx=4927, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158933: Sun Jan 11 06:05:20 2026 read: IOPS=61, BW=247KiB/s (253kB/s)(4976KiB/20109msec) slat (usec): min=26, max=8166, avg=2530.12, stdev=957.98 clat (msec): min=3, max=146, avg=79.92, stdev=15.28 lat (msec): min=5, max=149, avg=82.45, stdev=15.40 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 116], 99.50th=[ 120], 99.90th=[ 144], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 128, max= 368, per=24.91%, avg=248.36, stdev=59.84, samples=39 iops : min= 32, max= 92, avg=62.05, stdev=14.94, samples=39 write: IOPS=65, BW=262KiB/s (268kB/s)(5264KiB/20109msec); 0 zone resets slat (usec): min=25, max=1572, avg=110.04, stdev=74.12 clat (msec): min=30, max=297, avg=166.23, stdev=42.30 lat (msec): min=30, max=297, avg=166.34, stdev=42.30 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 100], 10.00th=[ 114], 20.00th=[ 130], | 30.00th=[ 144], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 191, max= 336, per=25.19%, avg=261.51, stdev=33.70, samples=39 iops : min= 47, max= 84, avg=65.33, stdev= 8.45, samples=39 lat (msec) : 4=0.04%, 10=0.04%, 20=0.04%, 50=1.25%, 100=44.88% lat (msec) : 250=52.66%, 500=1.09% cpu : usr=0.28%, sys=1.64%, ctx=4802, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158934: Sun Jan 11 06:05:20 2026 read: IOPS=61, BW=248KiB/s (254kB/s)(4984KiB/20109msec) slat (usec): min=979, max=8254, avg=2566.79, stdev=919.10 clat (msec): min=2, max=125, avg=79.53, stdev=16.02 lat (msec): min=4, max=129, avg=82.10, stdev=16.13 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 107], | 99.00th=[ 118], 99.50th=[ 124], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 144, max= 392, per=25.08%, avg=250.00, stdev=52.38, samples=39 iops : min= 36, max= 98, avg=62.46, stdev=13.07, samples=39 write: IOPS=65, BW=261KiB/s (268kB/s)(5256KiB/20109msec); 0 zone resets slat (usec): min=34, max=1940, avg=113.64, stdev=85.72 clat (msec): min=36, max=282, avg=166.40, stdev=40.81 lat (msec): min=36, max=282, avg=166.51, stdev=40.81 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 103], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 239], | 99.00th=[ 259], 99.50th=[ 266], 99.90th=[ 275], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 208, max= 376, per=25.09%, avg=260.44, stdev=37.91, samples=39 iops : min= 52, max= 94, avg=65.05, stdev= 9.51, samples=39 lat (msec) : 4=0.08%, 10=0.12%, 20=0.35%, 50=0.66%, 100=44.92% lat (msec) : 250=52.89%, 500=0.98% cpu : usr=0.23%, sys=1.71%, ctx=4812, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=158935: Sun Jan 11 06:05:20 2026 read: IOPS=62, BW=252KiB/s (258kB/s)(5060KiB/20099msec) slat (usec): min=987, max=8265, avg=2563.57, stdev=932.83 clat (msec): min=2, max=194, avg=80.10, stdev=16.39 lat (msec): min=5, max=196, avg=82.66, stdev=16.55 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 107], | 99.00th=[ 121], 99.50th=[ 123], 99.90th=[ 176], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 144, max= 336, per=25.45%, avg=253.72, stdev=48.74, samples=39 iops : min= 36, max= 84, avg=63.38, stdev=12.19, samples=39 write: IOPS=64, BW=258KiB/s (264kB/s)(5180KiB/20099msec); 0 zone resets slat (usec): min=24, max=3933, avg=114.29, stdev=164.69 clat (msec): min=51, max=285, avg=166.81, stdev=42.13 lat (msec): min=52, max=285, avg=166.92, stdev=42.13 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 259], 99.50th=[ 264], 99.90th=[ 271], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 176, max= 344, per=24.79%, avg=257.31, stdev=41.82, samples=39 iops : min= 44, max= 86, avg=64.28, stdev=10.46, samples=39 lat (msec) : 4=0.08%, 20=0.23%, 50=0.70%, 100=46.05%, 250=51.41% lat (msec) : 500=1.52% cpu : usr=0.27%, sys=1.66%, ctx=4886, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=997KiB/s (1021kB/s), 247KiB/s-252KiB/s (253kB/s-258kB/s), io=19.6MiB (20.5MB), run=20099-20121msec WRITE: bw=1038KiB/s (1063kB/s), 258KiB/s-262KiB/s (264kB/s-268kB/s), io=20.4MiB (21.4MB), run=20099-20121msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=158941: Sun Jan 11 06:05:24 2026 read: IOPS=9, BW=370MiB/s (388MB/s)(40.0MiB/108msec) slat (nsec): min=9291.8k, max=9291.8k, avg=9291811.00, stdev= 0.00 clat (nsec): min=94246k, max=94246k, avg=94245587.00, stdev= 0.00 lat (nsec): min=103542k, max=103542k, avg=103542121.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[93848], 5.00th=[93848], 10.00th=[93848], 20.00th=[93848], | 30.00th=[93848], 40.00th=[93848], 50.00th=[93848], 60.00th=[93848], | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.48%, ctx=35, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=370MiB/s (388MB/s), 370MiB/s-370MiB/s (388MB/s-388MB/s), io=40.0MiB (41.9MB), run=108-108msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout