== sanity test 398c: run fio to test AIO ================= 06:07:41 (1788516461) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.752137 s, 55.8 MB/s osc.lustre-OST0000-osc-ffff999f82ea2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180126: Fri Sep 4 06:08:21 2026 write: IOPS=89, BW=357KiB/s (365kB/s)(10.0MiB/28716msec); 0 zone resets slat (usec): min=35, max=16420, avg=207.17, stdev=654.51 clat (msec): min=39, max=383, avg=179.08, stdev=61.82 lat (msec): min=39, max=383, avg=179.29, stdev=61.83 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 88], 10.00th=[ 96], 20.00th=[ 128], | 30.00th=[ 144], 40.00th=[ 159], 50.00th=[ 174], 60.00th=[ 188], | 70.00th=[ 205], 80.00th=[ 228], 90.00th=[ 264], 95.00th=[ 300], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 144, max= 664, per=25.17%, avg=354.44, stdev=112.18, samples=57 iops : min= 36, max= 166, avg=88.32, stdev=28.13, samples=57 lat (msec) : 50=0.12%, 100=11.13%, 250=75.00%, 500=13.75% cpu : usr=0.14%, sys=0.99%, ctx=2003, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180127: Fri Sep 4 06:08:21 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29065msec); 0 zone resets slat (usec): min=35, max=11577, avg=223.56, stdev=619.54 clat (msec): min=21, max=402, avg=180.25, stdev=61.65 lat (msec): min=21, max=403, avg=180.47, stdev=61.67 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 128], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 228], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 368], 99.90th=[ 372], 99.95th=[ 393], | 99.99th=[ 405] bw ( KiB/s): min= 143, max= 776, per=25.08%, avg=353.19, stdev=116.80, samples=57 iops : min= 35, max= 194, avg=88.00, stdev=29.30, samples=57 lat (msec) : 50=0.12%, 100=11.05%, 250=75.70%, 500=13.12% cpu : usr=0.11%, sys=0.94%, ctx=1824, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180128: Fri Sep 4 06:08:21 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29074msec); 0 zone resets slat (usec): min=36, max=12549, avg=239.66, stdev=703.71 clat (msec): min=15, max=375, avg=180.17, stdev=61.03 lat (msec): min=16, max=375, avg=180.42, stdev=61.05 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 87], 10.00th=[ 100], 20.00th=[ 131], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 176], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 228], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 160, max= 640, per=24.91%, avg=350.68, stdev=107.15, samples=57 iops : min= 40, max= 160, avg=87.39, stdev=26.87, samples=57 lat (msec) : 20=0.04%, 50=0.12%, 100=10.20%, 250=75.82%, 500=13.83% cpu : usr=0.14%, sys=1.01%, ctx=2575, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180129: Fri Sep 4 06:08:21 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29051msec); 0 zone resets slat (usec): min=35, max=18515, avg=270.72, stdev=887.48 clat (msec): min=23, max=401, avg=180.25, stdev=63.39 lat (msec): min=25, max=402, avg=180.52, stdev=63.42 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 130], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 205], 80.00th=[ 228], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 401] bw ( KiB/s): min= 175, max= 648, per=25.10%, avg=353.39, stdev=107.03, samples=57 iops : min= 43, max= 162, avg=88.04, stdev=26.85, samples=57 lat (msec) : 50=0.12%, 100=10.74%, 250=75.23%, 500=13.91% cpu : usr=0.17%, sys=1.19%, ctx=2788, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1409KiB/s (1443kB/s), 352KiB/s-357KiB/s (361kB/s-365kB/s), io=40.0MiB (41.9MB), run=28716-29074msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180144: Fri Sep 4 06:08:57 2026 read: IOPS=44, BW=179KiB/s (183kB/s)(5048KiB/28211msec) slat (usec): min=2670, max=54016, avg=12704.64, stdev=6978.68 clat (msec): min=72, max=326, avg=165.45, stdev=41.88 lat (msec): min=80, max=357, avg=178.16, stdev=43.66 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 271], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 74, max= 270, per=25.11%, avg=178.53, stdev=43.72, samples=55 iops : min= 18, max= 67, avg=44.18, stdev=10.96, samples=55 write: IOPS=46, BW=184KiB/s (188kB/s)(5192KiB/28211msec); 0 zone resets slat (usec): min=39, max=9945, avg=315.57, stdev=735.94 clat (msec): min=37, max=345, avg=171.03, stdev=40.08 lat (msec): min=37, max=346, avg=171.34, stdev=40.10 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 66, max= 293, per=24.93%, avg=184.51, stdev=47.64, samples=55 iops : min= 16, max= 73, avg=45.69, stdev=11.95, samples=55 lat (msec) : 50=0.04%, 100=2.30%, 250=93.79%, 500=3.87% cpu : usr=0.34%, sys=2.23%, ctx=4271, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180145: Fri Sep 4 06:08:57 2026 read: IOPS=44, BW=179KiB/s (183kB/s)(4976KiB/27856msec) slat (usec): min=61, max=68807, avg=12490.39, stdev=7225.70 clat (msec): min=15, max=321, avg=165.62, stdev=41.76 lat (msec): min=25, max=340, avg=178.11, stdev=43.38 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 224], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 309], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 87, max= 269, per=25.06%, avg=178.15, stdev=41.46, samples=55 iops : min= 21, max= 67, avg=44.11, stdev=10.40, samples=55 write: IOPS=47, BW=189KiB/s (194kB/s)(5264KiB/27856msec); 0 zone resets slat (usec): min=39, max=10266, avg=336.29, stdev=786.19 clat (msec): min=76, max=339, avg=169.54, stdev=39.26 lat (msec): min=76, max=339, avg=169.88, stdev=39.25 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 111], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 245], | 99.00th=[ 279], 99.50th=[ 284], 99.90th=[ 317], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 95, max= 301, per=25.53%, avg=188.91, stdev=42.53, samples=55 iops : min= 23, max= 75, avg=46.80, stdev=10.72, samples=55 lat (msec) : 20=0.08%, 100=2.34%, 250=93.83%, 500=3.75% cpu : usr=0.35%, sys=2.28%, ctx=4362, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180146: Fri Sep 4 06:08:57 2026 read: IOPS=44, BW=177KiB/s (181kB/s)(4984KiB/28209msec) slat (usec): min=2730, max=48856, avg=12701.36, stdev=6796.87 clat (msec): min=39, max=367, avg=166.55, stdev=41.36 lat (msec): min=46, max=389, avg=179.26, stdev=42.97 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 106], 10.00th=[ 118], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 81, max= 262, per=24.91%, avg=177.15, stdev=42.73, samples=55 iops : min= 20, max= 65, avg=43.84, stdev=10.71, samples=55 write: IOPS=46, BW=186KiB/s (191kB/s)(5256KiB/28209msec); 0 zone resets slat (usec): min=41, max=13247, avg=340.77, stdev=876.38 clat (msec): min=83, max=337, avg=170.91, stdev=40.47 lat (msec): min=83, max=337, avg=171.25, stdev=40.48 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 245], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 87, max= 304, per=25.12%, avg=185.89, stdev=49.32, samples=55 iops : min= 21, max= 76, avg=46.04, stdev=12.41, samples=55 lat (msec) : 50=0.04%, 100=2.50%, 250=93.55%, 500=3.91% cpu : usr=0.27%, sys=2.36%, ctx=4324, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180147: Fri Sep 4 06:08:57 2026 read: IOPS=44, BW=179KiB/s (184kB/s)(5060KiB/28217msec) slat (usec): min=2050, max=60461, avg=12467.95, stdev=7039.42 clat (msec): min=44, max=336, avg=166.63, stdev=40.50 lat (msec): min=54, max=359, avg=179.10, stdev=42.27 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 108], 10.00th=[ 117], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 220], 95.00th=[ 241], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 317], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 88, max= 317, per=25.06%, avg=178.18, stdev=42.24, samples=56 iops : min= 22, max= 79, avg=44.11, stdev=10.54, samples=56 write: IOPS=45, BW=184KiB/s (188kB/s)(5180KiB/28217msec); 0 zone resets slat (usec): min=39, max=20155, avg=338.99, stdev=942.31 clat (msec): min=40, max=342, avg=170.35, stdev=40.24 lat (msec): min=40, max=342, avg=170.69, stdev=40.25 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 111], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 247], | 99.00th=[ 275], 99.50th=[ 292], 99.90th=[ 317], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 63, max= 277, per=24.67%, avg=182.59, stdev=45.37, samples=56 iops : min= 15, max= 69, avg=45.21, stdev=11.39, samples=56 lat (msec) : 50=0.08%, 100=1.80%, 250=94.26%, 500=3.87% cpu : usr=0.30%, sys=2.24%, ctx=4181, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=711KiB/s (728kB/s), 177KiB/s-179KiB/s (181kB/s-184kB/s), io=19.6MiB (20.5MB), run=27856-28217msec WRITE: bw=740KiB/s (758kB/s), 184KiB/s-189KiB/s (188kB/s-194kB/s), io=20.4MiB (21.4MB), run=27856-28217msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180156: Fri Sep 4 06:09:08 2026 read: IOPS=1, BW=70.7MiB/s (74.1MB/s)(40.0MiB/566msec) slat (nsec): min=32996k, max=32996k, avg=32995901.00, stdev= 0.00 clat (nsec): min=509123k, max=509123k, avg=509123204.00, stdev= 0.00 lat (nsec): min=542125k, max=542125k, avg=542125048.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.59%, ctx=10, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.7MiB/s (74.1MB/s), 70.7MiB/s-70.7MiB/s (74.1MB/s-74.1MB/s), io=40.0MiB (41.9MB), run=566-566msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout