== sanity test 398c: run fio to test AIO ================= 09:37:15 (1775223435) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.726428 s, 57.7 MB/s osc.lustre-OST0000-osc-ffff943110f5f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182342: Fri Apr 3 09:37:59 2026 write: IOPS=78, BW=316KiB/s (323kB/s)(10.0MiB/32426msec); 0 zone resets slat (usec): min=34, max=29294, avg=348.52, stdev=1213.50 clat (msec): min=58, max=471, avg=201.87, stdev=64.55 lat (msec): min=59, max=471, avg=202.23, stdev=64.62 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 89], 10.00th=[ 122], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 184], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 372], 99.50th=[ 376], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 142, max= 659, per=24.89%, avg=314.14, stdev=86.95, samples=64 iops : min= 35, max= 164, avg=78.06, stdev=21.73, samples=64 lat (msec) : 100=6.29%, 250=73.16%, 500=20.55% cpu : usr=0.17%, sys=1.21%, ctx=2663, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182343: Fri Apr 3 09:37:59 2026 write: IOPS=79, BW=316KiB/s (324kB/s)(10.0MiB/32355msec); 0 zone resets slat (usec): min=36, max=12028, avg=294.69, stdev=879.03 clat (msec): min=46, max=547, avg=201.69, stdev=67.05 lat (msec): min=47, max=549, avg=201.99, stdev=67.07 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 85], 10.00th=[ 125], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 124, max= 761, per=24.87%, avg=313.81, stdev=98.84, samples=64 iops : min= 31, max= 190, avg=78.00, stdev=24.74, samples=64 lat (msec) : 50=0.04%, 100=6.95%, 250=72.11%, 500=20.62%, 750=0.27% cpu : usr=0.17%, sys=0.98%, ctx=2599, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182344: Fri Apr 3 09:37:59 2026 write: IOPS=78, BW=316KiB/s (323kB/s)(10.0MiB/32423msec); 0 zone resets slat (usec): min=35, max=14619, avg=322.79, stdev=947.97 clat (msec): min=24, max=505, avg=201.77, stdev=66.39 lat (msec): min=24, max=506, avg=202.10, stdev=66.39 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 87], 10.00th=[ 124], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 184], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 253], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 384], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 166, max= 654, per=24.89%, avg=314.11, stdev=90.71, samples=64 iops : min= 41, max= 163, avg=78.06, stdev=22.73, samples=64 lat (msec) : 50=0.08%, 100=7.30%, 250=71.68%, 500=20.78%, 750=0.16% cpu : usr=0.16%, sys=1.11%, ctx=2606, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182345: Fri Apr 3 09:37:59 2026 write: IOPS=78, BW=316KiB/s (323kB/s)(10.0MiB/32436msec); 0 zone resets slat (usec): min=37, max=13596, avg=293.67, stdev=827.23 clat (msec): min=13, max=413, avg=201.73, stdev=64.24 lat (msec): min=13, max=413, avg=202.03, stdev=64.24 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 92], 10.00th=[ 122], 20.00th=[ 150], | 30.00th=[ 169], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 232], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 414] bw ( KiB/s): min= 159, max= 597, per=24.85%, avg=313.58, stdev=79.94, samples=64 iops : min= 39, max= 149, avg=77.92, stdev=20.02, samples=64 lat (msec) : 20=0.08%, 50=0.16%, 100=6.17%, 250=71.88%, 500=21.72% cpu : usr=0.13%, sys=1.07%, ctx=2884, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1263KiB/s (1293kB/s), 316KiB/s-316KiB/s (323kB/s-324kB/s), io=40.0MiB (41.9MB), run=32355-32436msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182360: Fri Apr 3 09:38:38 2026 read: IOPS=42, BW=170KiB/s (174kB/s)(5048KiB/29727msec) slat (usec): min=2440, max=65744, avg=12761.88, stdev=7181.61 clat (msec): min=21, max=375, avg=174.69, stdev=42.20 lat (msec): min=29, max=381, avg=187.46, stdev=43.84 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 114], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 363], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 86, max= 247, per=25.16%, avg=169.60, stdev=40.06, samples=58 iops : min= 21, max= 61, avg=41.90, stdev=10.07, samples=58 write: IOPS=43, BW=175KiB/s (179kB/s)(5192KiB/29727msec); 0 zone resets slat (usec): min=43, max=13945, avg=325.31, stdev=890.18 clat (msec): min=52, max=389, avg=180.85, stdev=41.90 lat (msec): min=53, max=390, avg=181.18, stdev=41.90 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 236], 95.00th=[ 255], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 72, max= 296, per=25.00%, avg=175.28, stdev=41.25, samples=58 iops : min= 18, max= 74, avg=43.29, stdev=10.30, samples=58 lat (msec) : 50=0.12%, 100=0.94%, 250=94.10%, 500=4.84% cpu : usr=0.26%, sys=2.08%, ctx=4773, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182361: Fri Apr 3 09:38:38 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(4976KiB/29765msec) slat (usec): min=2706, max=65061, avg=12837.29, stdev=6930.91 clat (msec): min=18, max=401, avg=176.06, stdev=43.87 lat (msec): min=25, max=411, avg=188.90, stdev=45.38 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 140], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 36, max= 271, per=24.73%, avg=166.69, stdev=40.51, samples=58 iops : min= 9, max= 67, avg=41.16, stdev=10.09, samples=58 write: IOPS=44, BW=177KiB/s (181kB/s)(5264KiB/29765msec); 0 zone resets slat (usec): min=37, max=24166, avg=281.04, stdev=974.21 clat (msec): min=32, max=382, avg=179.89, stdev=41.42 lat (msec): min=32, max=383, avg=180.17, stdev=41.42 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 296], 99.50th=[ 313], 99.90th=[ 355], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 101, max= 287, per=25.32%, avg=177.47, stdev=38.49, samples=58 iops : min= 25, max= 71, avg=43.86, stdev= 9.67, samples=58 lat (msec) : 20=0.04%, 50=0.12%, 100=1.52%, 250=93.36%, 500=4.96% cpu : usr=0.25%, sys=2.21%, ctx=4689, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182362: Fri Apr 3 09:38:38 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(4984KiB/29763msec) slat (usec): min=2851, max=81608, avg=12981.13, stdev=7378.67 clat (msec): min=22, max=367, avg=175.73, stdev=42.85 lat (msec): min=49, max=388, avg=188.71, stdev=44.28 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 251], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 351], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 65, max= 252, per=24.81%, avg=167.21, stdev=40.10, samples=58 iops : min= 16, max= 63, avg=41.31, stdev=10.01, samples=58 write: IOPS=44, BW=177KiB/s (181kB/s)(5256KiB/29763msec); 0 zone resets slat (usec): min=40, max=16132, avg=313.80, stdev=863.60 clat (msec): min=74, max=323, avg=179.79, stdev=40.27 lat (msec): min=74, max=324, avg=180.11, stdev=40.28 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 251], | 99.00th=[ 288], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 80, max= 286, per=25.02%, avg=175.41, stdev=44.10, samples=58 iops : min= 20, max= 71, avg=43.34, stdev=11.06, samples=58 lat (msec) : 50=0.08%, 100=1.45%, 250=93.12%, 500=5.35% cpu : usr=0.27%, sys=2.18%, ctx=4524, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182363: Fri Apr 3 09:38:38 2026 read: IOPS=42, BW=172KiB/s (176kB/s)(5060KiB/29435msec) slat (usec): min=61, max=73426, avg=12580.15, stdev=6885.27 clat (msec): min=82, max=354, avg=176.15, stdev=43.89 lat (msec): min=97, max=373, avg=188.74, stdev=45.21 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 138], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 334], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 108, max= 296, per=25.46%, avg=171.62, stdev=38.66, samples=58 iops : min= 27, max= 74, avg=42.40, stdev= 9.71, samples=58 write: IOPS=43, BW=176KiB/s (180kB/s)(5180KiB/29435msec); 0 zone resets slat (usec): min=37, max=16211, avg=314.14, stdev=828.85 clat (msec): min=25, max=375, avg=178.14, stdev=40.12 lat (msec): min=30, max=375, avg=178.46, stdev=40.09 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 251], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 334], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 80, max= 279, per=25.08%, avg=175.84, stdev=40.85, samples=58 iops : min= 20, max= 69, avg=43.47, stdev=10.17, samples=58 lat (msec) : 50=0.04%, 100=0.86%, 250=93.95%, 500=5.16% cpu : usr=0.24%, sys=1.96%, ctx=4488, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=674KiB/s (690kB/s), 167KiB/s-172KiB/s (171kB/s-176kB/s), io=19.6MiB (20.5MB), run=29435-29765msec WRITE: bw=702KiB/s (719kB/s), 175KiB/s-177KiB/s (179kB/s-181kB/s), io=20.4MiB (21.4MB), run=29435-29765msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182372: Fri Apr 3 09:38:48 2026 read: IOPS=2, BW=82.1MiB/s (86.1MB/s)(40.0MiB/487msec) slat (nsec): min=36677k, max=36677k, avg=36677188.00, stdev= 0.00 clat (nsec): min=427757k, max=427757k, avg=427757340.00, stdev= 0.00 lat (nsec): min=464440k, max=464440k, avg=464439918.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.62%, sys=3.29%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.1MiB/s (86.1MB/s), 82.1MiB/s-82.1MiB/s (86.1MB/s-86.1MB/s), io=40.0MiB (41.9MB), run=487-487msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout