== sanity test 398c: run fio to test AIO ================= 07:07:48 (1788692868) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.512559 s, 81.8 MB/s osc.lustre-OST0000-osc-ffff90c573160800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179408: Sun Sep 6 07:08:22 2026 write: IOPS=107, BW=432KiB/s (442kB/s)(10.0MiB/23707msec); 0 zone resets slat (usec): min=21, max=11927, avg=147.88, stdev=562.14 clat (msec): min=29, max=280, avg=147.41, stdev=35.96 lat (msec): min=29, max=280, avg=147.56, stdev=35.96 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 92], 10.00th=[ 105], 20.00th=[ 117], | 30.00th=[ 126], 40.00th=[ 136], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 211], | 99.00th=[ 247], 99.50th=[ 262], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 279] bw ( KiB/s): min= 254, max= 766, per=24.86%, avg=429.26, stdev=91.70, samples=47 iops : min= 63, max= 191, avg=106.91, stdev=22.88, samples=47 lat (msec) : 50=0.12%, 100=8.52%, 250=90.66%, 500=0.70% cpu : usr=0.18%, sys=0.89%, ctx=2174, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179409: Sun Sep 6 07:08:22 2026 write: IOPS=108, BW=433KiB/s (443kB/s)(10.0MiB/23661msec); 0 zone resets slat (usec): min=20, max=9356, avg=130.66, stdev=532.73 clat (msec): min=68, max=270, avg=147.59, stdev=36.00 lat (msec): min=68, max=270, avg=147.72, stdev=35.99 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 99], 10.00th=[ 105], 20.00th=[ 117], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 199], 95.00th=[ 213], | 99.00th=[ 245], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 253, max= 677, per=24.89%, avg=429.93, stdev=85.03, samples=46 iops : min= 63, max= 169, avg=107.11, stdev=21.28, samples=46 lat (msec) : 100=7.81%, 250=91.56%, 500=0.62% cpu : usr=0.13%, sys=0.71%, ctx=1032, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179410: Sun Sep 6 07:08:22 2026 write: IOPS=107, BW=432KiB/s (442kB/s)(10.0MiB/23711msec); 0 zone resets slat (usec): min=23, max=26478, avg=205.71, stdev=859.64 clat (msec): min=18, max=287, avg=147.24, stdev=36.93 lat (msec): min=18, max=287, avg=147.45, stdev=36.93 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 95], 10.00th=[ 106], 20.00th=[ 117], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 157], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 194], 95.00th=[ 218], | 99.00th=[ 257], 99.50th=[ 279], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 264, max= 702, per=24.87%, avg=429.49, stdev=80.19, samples=47 iops : min= 66, max= 175, avg=106.96, stdev=20.00, samples=47 lat (msec) : 20=0.04%, 50=0.16%, 100=7.97%, 250=90.55%, 500=1.29% cpu : usr=0.20%, sys=1.08%, ctx=2968, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179411: Sun Sep 6 07:08:22 2026 write: IOPS=108, BW=432KiB/s (443kB/s)(10.0MiB/23688msec); 0 zone resets slat (usec): min=21, max=13686, avg=185.82, stdev=653.44 clat (msec): min=14, max=262, avg=147.58, stdev=36.91 lat (msec): min=14, max=262, avg=147.76, stdev=36.88 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 117], | 30.00th=[ 129], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 222], | 99.00th=[ 255], 99.50th=[ 259], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 253, max= 710, per=24.90%, avg=429.96, stdev=87.22, samples=46 iops : min= 63, max= 177, avg=107.11, stdev=21.78, samples=46 lat (msec) : 20=0.12%, 50=0.23%, 100=7.58%, 250=90.86%, 500=1.21% cpu : usr=0.26%, sys=1.03%, ctx=2781, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1727KiB/s (1769kB/s), 432KiB/s-433KiB/s (442kB/s-443kB/s), io=40.0MiB (41.9MB), run=23661-23711msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179425: Sun Sep 6 07:08:58 2026 read: IOPS=46, BW=184KiB/s (189kB/s)(5048KiB/27410msec) slat (usec): min=28, max=58103, avg=11899.39, stdev=6727.41 clat (msec): min=70, max=345, avg=160.75, stdev=40.20 lat (msec): min=76, max=353, avg=172.66, stdev=42.26 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 178], 80.00th=[ 194], 90.00th=[ 215], 95.00th=[ 234], | 99.00th=[ 266], 99.50th=[ 279], 99.90th=[ 330], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 103, max= 270, per=25.25%, avg=183.56, stdev=43.47, samples=54 iops : min= 25, max= 67, avg=45.35, stdev=10.81, samples=54 write: IOPS=47, BW=189KiB/s (194kB/s)(5192KiB/27410msec); 0 zone resets slat (usec): min=28, max=17419, avg=211.66, stdev=816.52 clat (msec): min=74, max=324, avg=168.25, stdev=42.15 lat (msec): min=74, max=324, avg=168.46, stdev=42.17 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 249], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 95, max= 288, per=25.01%, avg=189.31, stdev=48.36, samples=54 iops : min= 23, max= 72, avg=46.80, stdev=12.11, samples=54 lat (msec) : 100=3.36%, 250=93.16%, 500=3.48% cpu : usr=0.32%, sys=1.91%, ctx=4626, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179426: Sun Sep 6 07:08:58 2026 read: IOPS=45, BW=181KiB/s (185kB/s)(4976KiB/27544msec) slat (usec): min=35, max=69480, avg=12268.73, stdev=7346.15 clat (msec): min=19, max=384, avg=162.49, stdev=43.27 lat (msec): min=35, max=394, avg=174.77, stdev=45.66 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 292], 99.90th=[ 338], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 87, max= 285, per=24.66%, avg=179.28, stdev=46.07, samples=54 iops : min= 21, max= 71, avg=44.28, stdev=11.53, samples=54 write: IOPS=47, BW=191KiB/s (196kB/s)(5264KiB/27544msec); 0 zone resets slat (usec): min=21, max=11650, avg=176.38, stdev=485.42 clat (msec): min=19, max=346, avg=167.86, stdev=42.18 lat (msec): min=19, max=347, avg=168.04, stdev=42.19 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 108], 10.00th=[ 121], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 111, max= 335, per=25.06%, avg=189.70, stdev=47.18, samples=54 iops : min= 27, max= 83, avg=46.89, stdev=11.76, samples=54 lat (msec) : 20=0.08%, 50=0.12%, 100=4.06%, 250=92.42%, 500=3.32% cpu : usr=0.32%, sys=1.87%, ctx=4422, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179427: Sun Sep 6 07:08:58 2026 read: IOPS=45, BW=182KiB/s (186kB/s)(4984KiB/27377msec) slat (usec): min=34, max=63846, avg=12359.41, stdev=7234.67 clat (msec): min=17, max=371, avg=162.61, stdev=44.76 lat (msec): min=18, max=391, avg=174.98, stdev=46.80 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 110], 20.00th=[ 127], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 245], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 363], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 103, max= 303, per=25.01%, avg=181.81, stdev=39.89, samples=54 iops : min= 25, max= 75, avg=44.91, stdev= 9.95, samples=54 write: IOPS=47, BW=192KiB/s (197kB/s)(5256KiB/27377msec); 0 zone resets slat (usec): min=23, max=17776, avg=213.38, stdev=770.20 clat (msec): min=25, max=358, avg=166.75, stdev=42.53 lat (msec): min=31, max=358, avg=166.97, stdev=42.57 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 321], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 80, max= 294, per=25.24%, avg=191.07, stdev=51.31, samples=54 iops : min= 20, max= 73, avg=47.24, stdev=12.85, samples=54 lat (msec) : 20=0.08%, 50=0.27%, 100=3.98%, 250=91.68%, 500=3.98% cpu : usr=0.29%, sys=1.90%, ctx=4507, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179428: Sun Sep 6 07:08:58 2026 read: IOPS=45, BW=184KiB/s (188kB/s)(5060KiB/27566msec) slat (usec): min=38, max=64542, avg=12191.95, stdev=7071.60 clat (msec): min=16, max=360, avg=160.41, stdev=42.54 lat (msec): min=20, max=368, avg=172.61, stdev=44.51 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 180], 80.00th=[ 194], 90.00th=[ 218], 95.00th=[ 239], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 72, max= 319, per=25.02%, avg=181.91, stdev=46.76, samples=54 iops : min= 18, max= 79, avg=44.93, stdev=11.64, samples=54 write: IOPS=46, BW=188KiB/s (192kB/s)(5180KiB/27566msec); 0 zone resets slat (usec): min=24, max=22250, avg=236.35, stdev=896.69 clat (msec): min=20, max=344, avg=169.88, stdev=44.03 lat (msec): min=20, max=345, avg=170.12, stdev=44.05 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 205], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 79, max= 320, per=24.50%, avg=185.44, stdev=42.10, samples=54 iops : min= 19, max= 80, avg=45.81, stdev=10.57, samples=54 lat (msec) : 20=0.04%, 50=0.23%, 100=3.59%, 250=92.19%, 500=3.95% cpu : usr=0.32%, sys=1.97%, ctx=4861, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=728KiB/s (745kB/s), 181KiB/s-184KiB/s (185kB/s-189kB/s), io=19.6MiB (20.5MB), run=27377-27566msec WRITE: bw=758KiB/s (776kB/s), 188KiB/s-192KiB/s (192kB/s-197kB/s), io=20.4MiB (21.4MB), run=27377-27566msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179437: Sun Sep 6 07:09:07 2026 read: IOPS=2, BW=87.7MiB/s (91.0MB/s)(40.0MiB/456msec) slat (nsec): min=44518k, max=44518k, avg=44518101.00, stdev= 0.00 clat (nsec): min=370122k, max=370122k, avg=370122480.00, stdev= 0.00 lat (nsec): min=414645k, max=414645k, avg=414645036.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.86%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=87.7MiB/s (91.0MB/s), 87.7MiB/s-87.7MiB/s (91.0MB/s-91.0MB/s), io=40.0MiB (41.9MB), run=456-456msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout