== sanity test 398c: run fio to test AIO ================= 10:04:34 (1788876274) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.554575 s, 75.6 MB/s osc.lustre-OST0000-osc-ffff9d6745193000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179827: Tue Sep 8 10:05:12 2026 write: IOPS=97, BW=392KiB/s (401kB/s)(10.0MiB/26127msec); 0 zone resets slat (usec): min=50, max=16611, avg=270.97, stdev=687.16 clat (msec): min=14, max=268, avg=162.26, stdev=33.32 lat (msec): min=14, max=268, avg=162.54, stdev=33.29 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 110], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 205], 95.00th=[ 220], | 99.00th=[ 251], 99.50th=[ 257], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 271] bw ( KiB/s): min= 232, max= 547, per=24.93%, avg=390.71, stdev=62.84, samples=51 iops : min= 58, max= 136, avg=97.24, stdev=15.70, samples=51 lat (msec) : 20=0.04%, 50=0.16%, 100=3.24%, 250=95.39%, 500=1.17% cpu : usr=0.18%, sys=1.27%, ctx=2782, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179828: Tue Sep 8 10:05:12 2026 write: IOPS=98, BW=392KiB/s (402kB/s)(10.0MiB/26103msec); 0 zone resets slat (usec): min=50, max=21580, avg=252.48, stdev=779.56 clat (msec): min=32, max=265, avg=162.26, stdev=34.34 lat (msec): min=33, max=266, avg=162.51, stdev=34.35 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 207], 95.00th=[ 220], | 99.00th=[ 255], 99.50th=[ 259], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 255, max= 515, per=24.91%, avg=390.39, stdev=62.38, samples=51 iops : min= 63, max= 128, avg=97.20, stdev=15.59, samples=51 lat (msec) : 50=0.12%, 100=2.50%, 250=95.98%, 500=1.41% cpu : usr=0.15%, sys=1.15%, ctx=1747, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179829: Tue Sep 8 10:05:12 2026 write: IOPS=98, BW=394KiB/s (404kB/s)(10.0MiB/25959msec); 0 zone resets slat (usec): min=51, max=11703, avg=279.89, stdev=685.84 clat (msec): min=22, max=267, avg=161.78, stdev=33.32 lat (msec): min=23, max=268, avg=162.06, stdev=33.32 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 107], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 203], 95.00th=[ 222], | 99.00th=[ 251], 99.50th=[ 257], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 248, max= 547, per=25.13%, avg=393.75, stdev=66.07, samples=51 iops : min= 62, max= 136, avg=98.02, stdev=16.47, samples=51 lat (msec) : 50=0.23%, 100=2.97%, 250=95.74%, 500=1.05% cpu : usr=0.17%, sys=1.31%, ctx=2674, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179830: Tue Sep 8 10:05:12 2026 write: IOPS=98, BW=393KiB/s (402kB/s)(10.0MiB/26075msec); 0 zone resets slat (usec): min=52, max=14949, avg=304.25, stdev=759.65 clat (msec): min=18, max=264, avg=161.85, stdev=33.47 lat (msec): min=18, max=264, avg=162.16, stdev=33.45 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 107], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 205], 95.00th=[ 218], | 99.00th=[ 247], 99.50th=[ 253], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 255, max= 580, per=25.01%, avg=391.98, stdev=64.42, samples=51 iops : min= 63, max= 145, avg=97.59, stdev=16.09, samples=51 lat (msec) : 20=0.04%, 50=0.12%, 100=2.50%, 250=96.45%, 500=0.90% cpu : usr=0.20%, sys=1.31%, ctx=2983, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1568KiB/s (1605kB/s), 392KiB/s-394KiB/s (401kB/s-404kB/s), io=40.0MiB (41.9MB), run=25959-26127msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179845: Tue Sep 8 10:05:50 2026 read: IOPS=43, BW=174KiB/s (178kB/s)(5048KiB/28997msec) slat (usec): min=2870, max=56738, avg=12573.87, stdev=6594.84 clat (msec): min=28, max=319, avg=171.31, stdev=40.48 lat (msec): min=56, max=333, avg=183.89, stdev=42.03 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 111], 10.00th=[ 122], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 78, max= 264, per=25.13%, avg=173.63, stdev=35.32, samples=57 iops : min= 19, max= 66, avg=42.95, stdev= 8.90, samples=57 write: IOPS=44, BW=179KiB/s (183kB/s)(5192KiB/28997msec); 0 zone resets slat (usec): min=57, max=14791, avg=324.60, stdev=918.92 clat (msec): min=86, max=305, avg=175.48, stdev=38.50 lat (msec): min=86, max=305, avg=175.81, stdev=38.54 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 121], 10.00th=[ 131], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 86, max= 279, per=24.86%, avg=178.72, stdev=42.20, samples=57 iops : min= 21, max= 69, avg=44.21, stdev=10.56, samples=57 lat (msec) : 50=0.04%, 100=1.09%, 250=94.77%, 500=4.10% cpu : usr=0.29%, sys=1.98%, ctx=4378, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179846: Tue Sep 8 10:05:50 2026 read: IOPS=42, BW=172KiB/s (176kB/s)(4976KiB/28939msec) slat (usec): min=69, max=52286, avg=12801.51, stdev=6852.84 clat (msec): min=15, max=323, avg=171.34, stdev=42.61 lat (msec): min=26, max=345, avg=184.15, stdev=44.30 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 109], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 249], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 317], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 95, max= 256, per=24.79%, avg=171.28, stdev=38.47, samples=57 iops : min= 23, max= 64, avg=42.37, stdev= 9.69, samples=57 write: IOPS=45, BW=182KiB/s (186kB/s)(5264KiB/28939msec); 0 zone resets slat (usec): min=55, max=9677, avg=309.66, stdev=749.55 clat (msec): min=44, max=303, avg=176.89, stdev=39.24 lat (msec): min=53, max=303, avg=177.20, stdev=39.21 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 251], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 103, max= 288, per=25.27%, avg=181.68, stdev=43.65, samples=57 iops : min= 25, max= 72, avg=44.98, stdev=10.95, samples=57 lat (msec) : 20=0.04%, 50=0.12%, 100=1.21%, 250=93.91%, 500=4.73% cpu : usr=0.27%, sys=2.04%, ctx=4419, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179847: Tue Sep 8 10:05:50 2026 read: IOPS=42, BW=172KiB/s (176kB/s)(4984KiB/29025msec) slat (usec): min=2917, max=53931, avg=12750.75, stdev=6811.34 clat (msec): min=33, max=346, avg=171.72, stdev=41.73 lat (msec): min=50, max=362, avg=184.47, stdev=43.09 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 205], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 330], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 78, max= 240, per=24.76%, avg=171.12, stdev=39.14, samples=57 iops : min= 19, max= 60, avg=42.32, stdev= 9.81, samples=57 write: IOPS=45, BW=181KiB/s (185kB/s)(5256KiB/29025msec); 0 zone resets slat (usec): min=55, max=15073, avg=350.25, stdev=922.50 clat (msec): min=66, max=317, avg=175.97, stdev=37.68 lat (msec): min=66, max=317, avg=176.32, stdev=37.67 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 92, max= 288, per=25.01%, avg=179.82, stdev=44.19, samples=57 iops : min= 23, max= 72, avg=44.53, stdev=11.10, samples=57 lat (msec) : 50=0.04%, 100=0.98%, 250=94.77%, 500=4.22% cpu : usr=0.24%, sys=1.99%, ctx=4404, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179848: Tue Sep 8 10:05:50 2026 read: IOPS=43, BW=174KiB/s (179kB/s)(5060KiB/29020msec) slat (usec): min=2697, max=49806, avg=12704.83, stdev=6421.18 clat (msec): min=29, max=323, avg=169.71, stdev=41.53 lat (msec): min=39, max=345, avg=182.42, stdev=43.01 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 111], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 243], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 317], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 93, max= 264, per=25.16%, avg=173.84, stdev=41.00, samples=57 iops : min= 23, max= 66, avg=43.00, stdev=10.24, samples=57 write: IOPS=44, BW=178KiB/s (183kB/s)(5180KiB/29020msec); 0 zone resets slat (usec): min=56, max=11386, avg=313.64, stdev=753.63 clat (msec): min=89, max=318, avg=177.75, stdev=38.36 lat (msec): min=90, max=318, avg=178.07, stdev=38.35 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 123], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 284], 99.50th=[ 288], 99.90th=[ 305], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 107, max= 248, per=24.70%, avg=177.56, stdev=33.99, samples=57 iops : min= 26, max= 62, avg=43.95, stdev= 8.61, samples=57 lat (msec) : 50=0.08%, 100=1.25%, 250=94.73%, 500=3.95% cpu : usr=0.21%, sys=1.98%, ctx=4407, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=691KiB/s (708kB/s), 172KiB/s-174KiB/s (176kB/s-179kB/s), io=19.6MiB (20.5MB), run=28939-29025msec WRITE: bw=720KiB/s (737kB/s), 178KiB/s-182KiB/s (183kB/s-186kB/s), io=20.4MiB (21.4MB), run=28939-29025msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179856: Tue Sep 8 10:05:59 2026 read: IOPS=2, BW=95.9MiB/s (101MB/s)(40.0MiB/417msec) slat (nsec): min=27353k, max=27353k, avg=27353463.00, stdev= 0.00 clat (nsec): min=363521k, max=363521k, avg=363521098.00, stdev= 0.00 lat (nsec): min=390879k, max=390879k, avg=390878829.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.12%, ctx=12, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=95.9MiB/s (101MB/s), 95.9MiB/s-95.9MiB/s (101MB/s-101MB/s), io=40.0MiB (41.9MB), run=417-417msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout