== sanity test 398c: run fio to test AIO ================= 04:24:27 (1788942267) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.556266 s, 75.4 MB/s osc.lustre-OST0000-osc-ffff982047d07800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179920: Wed Sep 9 04:25:07 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29054msec); 0 zone resets slat (usec): min=29, max=12760, avg=143.92, stdev=482.34 clat (msec): min=53, max=362, avg=180.60, stdev=42.67 lat (msec): min=53, max=362, avg=180.75, stdev=42.67 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 351], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 222, max= 510, per=24.87%, avg=350.39, stdev=65.00, samples=57 iops : min= 55, max= 127, avg=87.18, stdev=16.27, samples=57 lat (msec) : 100=1.21%, 250=92.70%, 500=6.09% cpu : usr=0.12%, sys=0.68%, ctx=1501, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179921: Wed Sep 9 04:25:07 2026 write: IOPS=88, BW=353KiB/s (362kB/s)(10.0MiB/28977msec); 0 zone resets slat (usec): min=29, max=14930, avg=219.45, stdev=794.22 clat (msec): min=12, max=354, avg=180.70, stdev=43.73 lat (msec): min=12, max=355, avg=180.92, stdev=43.73 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 115], 10.00th=[ 129], 20.00th=[ 148], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 296], 99.50th=[ 317], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 355] bw ( KiB/s): min= 216, max= 542, per=25.08%, avg=353.39, stdev=75.87, samples=57 iops : min= 54, max= 135, avg=87.93, stdev=18.98, samples=57 lat (msec) : 20=0.04%, 50=0.16%, 100=1.84%, 250=91.37%, 500=6.60% cpu : usr=0.14%, sys=0.85%, ctx=2802, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179922: Wed Sep 9 04:25:07 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29063msec); 0 zone resets slat (usec): min=25, max=11485, avg=176.75, stdev=599.60 clat (msec): min=36, max=359, avg=180.51, stdev=43.29 lat (msec): min=36, max=359, avg=180.69, stdev=43.29 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 292], 99.50th=[ 321], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 222, max= 510, per=24.82%, avg=349.68, stdev=68.11, samples=57 iops : min= 55, max= 127, avg=87.00, stdev=17.04, samples=57 lat (msec) : 50=0.04%, 100=1.72%, 250=91.60%, 500=6.64% cpu : usr=0.20%, sys=0.83%, ctx=2487, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179923: Wed Sep 9 04:25:07 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29068msec); 0 zone resets slat (usec): min=30, max=13054, avg=179.05, stdev=623.53 clat (msec): min=15, max=349, avg=180.51, stdev=42.70 lat (msec): min=15, max=354, avg=180.69, stdev=42.69 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 115], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 253], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 351] bw ( KiB/s): min= 254, max= 502, per=24.86%, avg=350.30, stdev=65.69, samples=57 iops : min= 63, max= 125, avg=87.14, stdev=16.41, samples=57 lat (msec) : 20=0.04%, 50=0.04%, 100=1.64%, 250=92.50%, 500=5.78% cpu : usr=0.15%, sys=0.81%, ctx=2478, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1409KiB/s (1443kB/s), 352KiB/s-353KiB/s (361kB/s-362kB/s), io=40.0MiB (41.9MB), run=28977-29068msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179937: Wed Sep 9 04:25:44 2026 read: IOPS=44, BW=177KiB/s (181kB/s)(5048KiB/28573msec) slat (usec): min=2769, max=54284, avg=12359.73, stdev=6692.34 clat (msec): min=21, max=319, avg=167.84, stdev=41.51 lat (msec): min=31, max=332, avg=180.21, stdev=43.51 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 105], 10.00th=[ 117], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 88, max= 272, per=25.11%, avg=176.25, stdev=40.78, samples=56 iops : min= 22, max= 68, avg=43.63, stdev=10.13, samples=56 write: IOPS=45, BW=182KiB/s (186kB/s)(5192KiB/28573msec); 0 zone resets slat (usec): min=34, max=22956, avg=263.28, stdev=1044.86 clat (msec): min=87, max=339, avg=174.65, stdev=38.60 lat (msec): min=87, max=348, avg=174.91, stdev=38.69 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 95, max= 296, per=24.89%, avg=181.95, stdev=41.42, samples=56 iops : min= 23, max= 74, avg=45.05, stdev=10.37, samples=56 lat (msec) : 50=0.04%, 100=2.30%, 250=94.38%, 500=3.28% cpu : usr=0.24%, sys=1.91%, ctx=4655, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179938: Wed Sep 9 04:25:44 2026 read: IOPS=43, BW=174KiB/s (178kB/s)(4976KiB/28574msec) slat (msec): min=2, max=170, avg=12.71, stdev= 9.14 clat (msec): min=5, max=543, avg=169.86, stdev=49.13 lat (msec): min=28, max=564, avg=182.57, stdev=51.68 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 106], 10.00th=[ 116], 20.00th=[ 133], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 226], 95.00th=[ 251], | 99.00th=[ 300], 99.50th=[ 342], 99.90th=[ 531], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 23, max= 272, per=24.65%, avg=173.02, stdev=41.96, samples=56 iops : min= 5, max= 68, avg=42.82, stdev=10.50, samples=56 write: IOPS=46, BW=184KiB/s (189kB/s)(5264KiB/28574msec); 0 zone resets slat (usec): min=27, max=26278, avg=250.87, stdev=973.27 clat (msec): min=21, max=539, avg=172.94, stdev=48.17 lat (msec): min=21, max=539, avg=173.19, stdev=48.17 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 114], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 205], 90.00th=[ 222], 95.00th=[ 247], | 99.00th=[ 309], 99.50th=[ 502], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 87, max= 295, per=25.18%, avg=184.05, stdev=47.59, samples=56 iops : min= 21, max= 73, avg=45.59, stdev=11.90, samples=56 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=2.50%, 250=92.54% lat (msec) : 500=4.26%, 750=0.47% cpu : usr=0.30%, sys=1.61%, ctx=4674, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179939: Wed Sep 9 04:25:44 2026 read: IOPS=43, BW=175KiB/s (179kB/s)(4984KiB/28501msec) slat (msec): min=2, max=112, avg=12.36, stdev= 7.41 clat (msec): min=41, max=375, avg=169.94, stdev=41.13 lat (msec): min=47, max=383, avg=182.31, stdev=42.60 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 107], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 334], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 71, max= 270, per=24.94%, avg=175.07, stdev=41.45, samples=56 iops : min= 17, max= 67, avg=43.32, stdev=10.38, samples=56 write: IOPS=46, BW=184KiB/s (189kB/s)(5256KiB/28501msec); 0 zone resets slat (usec): min=27, max=9526, avg=213.30, stdev=650.48 clat (msec): min=87, max=350, avg=172.03, stdev=39.47 lat (msec): min=87, max=350, avg=172.25, stdev=39.48 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 120], 10.00th=[ 128], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 96, max= 255, per=25.18%, avg=184.09, stdev=38.61, samples=56 iops : min= 24, max= 63, avg=45.59, stdev= 9.62, samples=56 lat (msec) : 50=0.12%, 100=1.29%, 250=94.80%, 500=3.79% cpu : usr=0.26%, sys=1.68%, ctx=4372, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179940: Wed Sep 9 04:25:44 2026 read: IOPS=44, BW=178KiB/s (182kB/s)(5060KiB/28470msec) slat (usec): min=57, max=50632, avg=12134.52, stdev=6705.26 clat (msec): min=16, max=343, avg=168.60, stdev=41.72 lat (msec): min=16, max=362, avg=180.74, stdev=43.76 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 108], 10.00th=[ 117], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 330], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 88, max= 271, per=25.35%, avg=177.98, stdev=41.46, samples=56 iops : min= 22, max= 67, avg=44.04, stdev=10.35, samples=56 write: IOPS=45, BW=182KiB/s (186kB/s)(5180KiB/28470msec); 0 zone resets slat (usec): min=29, max=7979, avg=192.58, stdev=513.39 clat (msec): min=42, max=376, avg=174.55, stdev=42.05 lat (msec): min=42, max=377, avg=174.75, stdev=42.04 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 117], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 253], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 112, max= 272, per=24.79%, avg=181.20, stdev=39.28, samples=56 iops : min= 28, max= 68, avg=44.88, stdev= 9.84, samples=56 lat (msec) : 20=0.04%, 50=0.08%, 100=1.60%, 250=93.91%, 500=4.38% cpu : usr=0.18%, sys=1.68%, ctx=4205, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=702KiB/s (719kB/s), 174KiB/s-178KiB/s (178kB/s-182kB/s), io=19.6MiB (20.5MB), run=28470-28574msec WRITE: bw=731KiB/s (749kB/s), 182KiB/s-184KiB/s (186kB/s-189kB/s), io=20.4MiB (21.4MB), run=28470-28574msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179948: Wed Sep 9 04:25:53 2026 read: IOPS=1, BW=79.1MiB/s (82.9MB/s)(40.0MiB/506msec) slat (nsec): min=19618k, max=19618k, avg=19618448.00, stdev= 0.00 clat (nsec): min=458236k, max=458236k, avg=458235761.00, stdev= 0.00 lat (nsec): min=477858k, max=477858k, avg=477858186.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.18%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.1MiB/s (82.9MB/s), 79.1MiB/s-79.1MiB/s (82.9MB/s-82.9MB/s), io=40.0MiB (41.9MB), run=506-506msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout