== sanity test 398c: run fio to test AIO ================= 10:20:04 (1784125204) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.21208 s, 198 MB/s osc.lustre-OST0000-osc-ffff9fb846f6c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=189615: Wed Jul 15 10:20:25 2026 write: IOPS=163, BW=653KiB/s (669kB/s)(10.0MiB/15674msec); 0 zone resets slat (usec): min=37, max=62218, avg=259.20, stdev=1286.18 clat (msec): min=9, max=216, avg=96.92, stdev=20.37 lat (msec): min=10, max=216, avg=97.18, stdev=20.33 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 82], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 96], 60.00th=[ 101], | 70.00th=[ 107], 80.00th=[ 113], 90.00th=[ 121], 95.00th=[ 128], | 99.00th=[ 167], 99.50th=[ 169], 99.90th=[ 207], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 464, max= 896, per=24.92%, avg=650.58, stdev=100.43, samples=31 iops : min= 116, max= 224, avg=162.45, stdev=25.14, samples=31 lat (msec) : 10=0.04%, 20=0.08%, 50=0.55%, 100=58.24%, 250=41.09% cpu : usr=0.20%, sys=2.53%, ctx=2826, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189616: Wed Jul 15 10:20:25 2026 write: IOPS=164, BW=656KiB/s (672kB/s)(10.0MiB/15600msec); 0 zone resets slat (usec): min=37, max=10528, avg=249.04, stdev=478.28 clat (msec): min=8, max=228, avg=97.12, stdev=20.62 lat (msec): min=8, max=228, avg=97.37, stdev=20.63 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 107], 80.00th=[ 113], 90.00th=[ 121], 95.00th=[ 129], | 99.00th=[ 159], 99.50th=[ 167], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 510, max= 904, per=25.19%, avg=657.63, stdev=94.67, samples=30 iops : min= 127, max= 226, avg=164.20, stdev=23.70, samples=30 lat (msec) : 10=0.04%, 20=0.04%, 50=0.70%, 100=57.27%, 250=41.95% cpu : usr=0.35%, sys=2.30%, ctx=2939, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189617: Wed Jul 15 10:20:25 2026 write: IOPS=163, BW=653KiB/s (669kB/s)(10.0MiB/15672msec); 0 zone resets slat (usec): min=36, max=66598, avg=257.63, stdev=1374.24 clat (msec): min=9, max=207, avg=96.92, stdev=20.00 lat (msec): min=9, max=207, avg=97.18, stdev=20.00 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 83], | 30.00th=[ 87], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 101], | 70.00th=[ 107], 80.00th=[ 112], 90.00th=[ 121], 95.00th=[ 127], | 99.00th=[ 159], 99.50th=[ 169], 99.90th=[ 194], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 479, max= 896, per=24.94%, avg=651.23, stdev=96.35, samples=31 iops : min= 119, max= 224, avg=162.61, stdev=24.10, samples=31 lat (msec) : 10=0.04%, 20=0.04%, 50=0.51%, 100=58.55%, 250=40.86% cpu : usr=0.27%, sys=2.37%, ctx=2904, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189618: Wed Jul 15 10:20:25 2026 write: IOPS=163, BW=653KiB/s (669kB/s)(10.0MiB/15685msec); 0 zone resets slat (usec): min=35, max=3497, avg=223.69, stdev=323.66 clat (msec): min=38, max=170, avg=97.01, stdev=19.57 lat (msec): min=38, max=170, avg=97.23, stdev=19.57 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 66], 10.00th=[ 75], 20.00th=[ 82], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 103], | 70.00th=[ 108], 80.00th=[ 113], 90.00th=[ 122], 95.00th=[ 128], | 99.00th=[ 153], 99.50th=[ 155], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 502, max= 904, per=24.93%, avg=650.81, stdev=95.04, samples=31 iops : min= 125, max= 226, avg=162.52, stdev=23.80, samples=31 lat (msec) : 50=0.70%, 100=55.59%, 250=43.71% cpu : usr=0.34%, sys=2.25%, ctx=2881, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2611KiB/s (2674kB/s), 653KiB/s-656KiB/s (669kB/s-672kB/s), io=40.0MiB (41.9MB), run=15600-15685msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=189630: Wed Jul 15 10:20:43 2026 read: IOPS=90, BW=362KiB/s (371kB/s)(5048KiB/13943msec) slat (usec): min=1465, max=19977, avg=5943.82, stdev=2873.93 clat (msec): min=11, max=160, avg=82.01, stdev=19.06 lat (msec): min=17, max=167, avg=87.96, stdev=19.65 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 97], 90.00th=[ 108], 95.00th=[ 116], | 99.00th=[ 138], 99.50th=[ 150], 99.90th=[ 155], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 256, max= 468, per=25.15%, avg=361.85, stdev=56.22, samples=27 iops : min= 64, max= 117, avg=90.26, stdev=14.12, samples=27 write: IOPS=93, BW=372KiB/s (381kB/s)(5192KiB/13943msec); 0 zone resets slat (usec): min=56, max=3960, avg=210.07, stdev=265.17 clat (msec): min=12, max=154, avg=85.04, stdev=17.80 lat (msec): min=12, max=155, avg=85.25, stdev=17.79 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 108], 95.00th=[ 118], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 153], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 215, max= 552, per=24.93%, avg=373.44, stdev=68.73, samples=27 iops : min= 53, max= 138, avg=93.19, stdev=17.32, samples=27 lat (msec) : 20=0.08%, 50=1.21%, 100=81.68%, 250=17.03% cpu : usr=0.36%, sys=3.12%, ctx=4531, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189631: Wed Jul 15 10:20:43 2026 read: IOPS=89, BW=358KiB/s (367kB/s)(4976KiB/13900msec) slat (usec): min=1537, max=26758, avg=5817.75, stdev=2791.09 clat (msec): min=16, max=164, avg=81.64, stdev=18.88 lat (msec): min=25, max=176, avg=87.46, stdev=19.43 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 55], 10.00th=[ 61], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 106], 95.00th=[ 115], | 99.00th=[ 140], 99.50th=[ 146], 99.90th=[ 161], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 255, max= 476, per=24.84%, avg=357.44, stdev=63.78, samples=27 iops : min= 63, max= 119, avg=89.19, stdev=16.08, samples=27 write: IOPS=94, BW=379KiB/s (388kB/s)(5264KiB/13900msec); 0 zone resets slat (usec): min=36, max=4600, avg=211.67, stdev=276.51 clat (msec): min=40, max=153, avg=85.51, stdev=18.84 lat (msec): min=40, max=154, avg=85.72, stdev=18.86 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 95], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 120], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 153], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 286, max= 488, per=25.26%, avg=378.37, stdev=49.88, samples=27 iops : min= 71, max= 122, avg=94.41, stdev=12.58, samples=27 lat (msec) : 20=0.04%, 50=1.99%, 100=80.23%, 250=17.73% cpu : usr=0.40%, sys=3.14%, ctx=4578, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189632: Wed Jul 15 10:20:43 2026 read: IOPS=90, BW=361KiB/s (370kB/s)(4984KiB/13812msec) slat (usec): min=61, max=24930, avg=5766.77, stdev=2762.12 clat (msec): min=11, max=154, avg=81.79, stdev=19.82 lat (msec): min=11, max=164, avg=87.56, stdev=20.52 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 71], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 97], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 136], 99.50th=[ 144], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 272, max= 464, per=25.15%, avg=361.89, stdev=56.01, samples=27 iops : min= 68, max= 116, avg=90.30, stdev=14.03, samples=27 write: IOPS=95, BW=381KiB/s (390kB/s)(5256KiB/13812msec); 0 zone resets slat (usec): min=40, max=6671, avg=223.82, stdev=361.35 clat (msec): min=17, max=160, avg=84.68, stdev=19.04 lat (msec): min=17, max=160, avg=84.91, stdev=19.06 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 93], 80.00th=[ 100], 90.00th=[ 108], 95.00th=[ 120], | 99.00th=[ 142], 99.50th=[ 148], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 215, max= 512, per=25.29%, avg=378.85, stdev=78.28, samples=27 iops : min= 53, max= 128, avg=94.52, stdev=19.73, samples=27 lat (msec) : 20=0.23%, 50=2.46%, 100=79.92%, 250=17.38% cpu : usr=0.31%, sys=3.13%, ctx=4545, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189633: Wed Jul 15 10:20:43 2026 read: IOPS=90, BW=363KiB/s (372kB/s)(5060KiB/13943msec) slat (usec): min=1515, max=22537, avg=5842.07, stdev=2779.44 clat (msec): min=31, max=160, avg=82.41, stdev=19.39 lat (msec): min=33, max=164, avg=88.26, stdev=19.91 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 56], 10.00th=[ 60], 20.00th=[ 66], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 99], 90.00th=[ 108], 95.00th=[ 117], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 157], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 279, max= 512, per=25.28%, avg=363.78, stdev=60.02, samples=27 iops : min= 69, max= 128, avg=90.78, stdev=15.06, samples=27 write: IOPS=92, BW=372KiB/s (380kB/s)(5180KiB/13943msec); 0 zone resets slat (usec): min=45, max=3201, avg=215.87, stdev=258.25 clat (msec): min=36, max=155, avg=85.04, stdev=18.92 lat (msec): min=36, max=155, avg=85.25, stdev=18.92 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 74], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 100], 90.00th=[ 110], 95.00th=[ 121], | 99.00th=[ 142], 99.50th=[ 146], 99.90th=[ 153], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 255, max= 478, per=24.75%, avg=370.78, stdev=60.71, samples=27 iops : min= 63, max= 119, avg=92.52, stdev=15.32, samples=27 lat (msec) : 50=1.52%, 100=80.23%, 250=18.24% cpu : usr=0.35%, sys=3.11%, ctx=4595, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1439KiB/s (1474kB/s), 358KiB/s-363KiB/s (367kB/s-372kB/s), io=19.6MiB (20.5MB), run=13812-13943msec WRITE: bw=1498KiB/s (1534kB/s), 372KiB/s-381KiB/s (380kB/s-390kB/s), io=20.4MiB (21.4MB), run=13812-13943msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=189639: Wed Jul 15 10:20:47 2026 read: IOPS=5, BW=211MiB/s (221MB/s)(40.0MiB/190msec) slat (nsec): min=13766k, max=13766k, avg=13765851.00, stdev= 0.00 clat (nsec): min=165985k, max=165985k, avg=165985283.00, stdev= 0.00 lat (nsec): min=179755k, max=179755k, avg=179755473.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.29%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=40.0MiB (41.9MB), run=190-190msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout