== sanity test 398c: run fio to test AIO ================= 14:53:36 (1776192816) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.785127 s, 53.4 MB/s osc.lustre-OST0000-osc-ffff9f49e1a86000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148904: Tue Apr 14 14:54:14 2026 write: IOPS=112, BW=452KiB/s (463kB/s)(10.0MiB/22658msec); 0 zone resets slat (usec): min=20, max=18607, avg=247.05, stdev=1010.62 clat (msec): min=13, max=330, avg=140.75, stdev=48.36 lat (msec): min=14, max=330, avg=140.99, stdev=48.36 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 74], 10.00th=[ 88], 20.00th=[ 102], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 133], 60.00th=[ 144], | 70.00th=[ 157], 80.00th=[ 176], 90.00th=[ 209], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 229, max= 768, per=24.78%, avg=447.86, stdev=133.14, samples=44 iops : min= 57, max= 192, avg=111.61, stdev=33.25, samples=44 lat (msec) : 20=0.04%, 100=19.10%, 250=76.88%, 500=3.98% cpu : usr=0.30%, sys=1.28%, ctx=955, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148905: Tue Apr 14 14:54:14 2026 write: IOPS=113, BW=453KiB/s (463kB/s)(10.0MiB/22627msec); 0 zone resets slat (usec): min=20, max=19111, avg=209.64, stdev=818.44 clat (msec): min=35, max=322, avg=140.61, stdev=48.63 lat (msec): min=36, max=323, avg=140.82, stdev=48.66 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 73], 10.00th=[ 86], 20.00th=[ 102], | 30.00th=[ 114], 40.00th=[ 123], 50.00th=[ 132], 60.00th=[ 144], | 70.00th=[ 159], 80.00th=[ 178], 90.00th=[ 209], 95.00th=[ 228], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 205, max= 763, per=24.80%, avg=448.14, stdev=130.02, samples=44 iops : min= 51, max= 190, avg=111.68, stdev=32.50, samples=44 lat (msec) : 50=0.08%, 100=18.40%, 250=78.12%, 500=3.40% cpu : usr=0.21%, sys=1.36%, ctx=1093, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148906: Tue Apr 14 14:54:14 2026 write: IOPS=113, BW=454KiB/s (465kB/s)(10.0MiB/22551msec); 0 zone resets slat (usec): min=20, max=15060, avg=270.75, stdev=947.69 clat (msec): min=15, max=339, avg=140.24, stdev=48.22 lat (msec): min=15, max=339, avg=140.52, stdev=48.22 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 73], 10.00th=[ 87], 20.00th=[ 102], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 133], 60.00th=[ 144], | 70.00th=[ 157], 80.00th=[ 180], 90.00th=[ 203], 95.00th=[ 228], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 253, max= 779, per=25.10%, avg=453.61, stdev=127.47, samples=44 iops : min= 63, max= 194, avg=113.09, stdev=31.88, samples=44 lat (msec) : 20=0.12%, 50=0.12%, 100=19.22%, 250=77.50%, 500=3.05% cpu : usr=0.31%, sys=1.47%, ctx=1083, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148907: Tue Apr 14 14:54:14 2026 write: IOPS=113, BW=453KiB/s (464kB/s)(10.0MiB/22581msec); 0 zone resets slat (usec): min=21, max=17948, avg=260.77, stdev=912.05 clat (msec): min=24, max=336, avg=140.73, stdev=48.84 lat (msec): min=25, max=336, avg=140.99, stdev=48.89 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 72], 10.00th=[ 87], 20.00th=[ 104], | 30.00th=[ 112], 40.00th=[ 123], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 161], 80.00th=[ 178], 90.00th=[ 207], 95.00th=[ 228], | 99.00th=[ 292], 99.50th=[ 326], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 221, max= 763, per=25.01%, avg=451.89, stdev=126.37, samples=44 iops : min= 55, max= 190, avg=112.64, stdev=31.59, samples=44 lat (msec) : 50=0.16%, 100=18.01%, 250=78.59%, 500=3.24% cpu : usr=0.31%, sys=1.43%, ctx=1359, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1808KiB/s (1851kB/s), 452KiB/s-454KiB/s (463kB/s-465kB/s), io=40.0MiB (41.9MB), run=22551-22658msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148920: Tue Apr 14 14:54:51 2026 read: IOPS=44, BW=178KiB/s (183kB/s)(5048KiB/28317msec) slat (usec): min=2343, max=63723, avg=12661.09, stdev=8229.54 clat (msec): min=4, max=430, avg=167.18, stdev=52.03 lat (msec): min=7, max=444, avg=179.85, stdev=53.85 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 93], 10.00th=[ 107], 20.00th=[ 125], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 207], 90.00th=[ 234], 95.00th=[ 255], | 99.00th=[ 326], 99.50th=[ 355], 99.90th=[ 422], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 78, max= 293, per=25.11%, avg=177.27, stdev=48.24, samples=56 iops : min= 19, max= 73, avg=43.75, stdev=12.05, samples=56 write: IOPS=45, BW=183KiB/s (188kB/s)(5192KiB/28317msec); 0 zone resets slat (usec): min=22, max=12895, avg=270.10, stdev=935.44 clat (msec): min=65, max=408, avg=170.89, stdev=51.60 lat (msec): min=65, max=408, avg=171.16, stdev=51.60 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 97], 10.00th=[ 109], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 232], 95.00th=[ 264], | 99.00th=[ 342], 99.50th=[ 372], 99.90th=[ 397], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 72, max= 351, per=24.86%, avg=182.71, stdev=51.83, samples=56 iops : min= 18, max= 87, avg=45.14, stdev=12.92, samples=56 lat (msec) : 10=0.04%, 50=0.04%, 100=6.76%, 250=87.03%, 500=6.13% cpu : usr=0.33%, sys=2.17%, ctx=2543, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148921: Tue Apr 14 14:54:51 2026 read: IOPS=44, BW=176KiB/s (181kB/s)(4976KiB/28205msec) slat (usec): min=39, max=58335, avg=12687.91, stdev=8042.11 clat (msec): min=9, max=396, avg=166.86, stdev=52.16 lat (msec): min=21, max=412, avg=179.56, stdev=54.27 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 93], 10.00th=[ 105], 20.00th=[ 123], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 163], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 321], 99.50th=[ 355], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 78, max= 294, per=24.94%, avg=176.11, stdev=45.52, samples=55 iops : min= 19, max= 73, avg=43.49, stdev=11.36, samples=55 write: IOPS=46, BW=187KiB/s (191kB/s)(5264KiB/28205msec); 0 zone resets slat (usec): min=22, max=16232, avg=242.53, stdev=948.17 clat (msec): min=37, max=396, avg=172.44, stdev=48.87 lat (msec): min=38, max=396, avg=172.68, stdev=48.87 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 157], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 255], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 376], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 70, max= 351, per=25.51%, avg=187.49, stdev=51.78, samples=55 iops : min= 17, max= 87, avg=46.33, stdev=12.91, samples=55 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=6.37%, 250=87.30% lat (msec) : 500=6.09% cpu : usr=0.37%, sys=2.08%, ctx=2553, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148922: Tue Apr 14 14:54:51 2026 read: IOPS=43, BW=176KiB/s (180kB/s)(4984KiB/28372msec) slat (usec): min=2222, max=47716, avg=12842.22, stdev=7552.11 clat (msec): min=18, max=408, avg=166.93, stdev=50.74 lat (msec): min=35, max=414, avg=179.78, stdev=52.61 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 126], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 259], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 347], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 86, max= 318, per=24.88%, avg=175.66, stdev=49.60, samples=56 iops : min= 21, max= 79, avg=43.37, stdev=12.38, samples=56 write: IOPS=46, BW=185KiB/s (190kB/s)(5256KiB/28372msec); 0 zone resets slat (usec): min=22, max=23743, avg=301.73, stdev=1323.17 clat (msec): min=28, max=417, avg=172.11, stdev=49.50 lat (msec): min=28, max=417, avg=172.41, stdev=49.52 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 99], 10.00th=[ 113], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 157], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 313], 99.50th=[ 388], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 78, max= 295, per=25.06%, avg=184.18, stdev=48.68, samples=56 iops : min= 19, max= 73, avg=45.52, stdev=12.16, samples=56 lat (msec) : 20=0.04%, 50=0.23%, 100=6.52%, 250=87.27%, 500=5.94% cpu : usr=0.36%, sys=2.24%, ctx=2686, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148923: Tue Apr 14 14:54:51 2026 read: IOPS=44, BW=178KiB/s (183kB/s)(5060KiB/28386msec) slat (usec): min=2133, max=58306, avg=12699.74, stdev=7767.87 clat (msec): min=12, max=397, avg=166.33, stdev=48.55 lat (msec): min=35, max=413, avg=179.03, stdev=50.41 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 97], 10.00th=[ 108], 20.00th=[ 124], | 30.00th=[ 138], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 249], | 99.00th=[ 300], 99.50th=[ 321], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 78, max= 279, per=25.18%, avg=177.79, stdev=45.70, samples=56 iops : min= 19, max= 69, avg=43.91, stdev=11.40, samples=56 write: IOPS=45, BW=182KiB/s (187kB/s)(5180KiB/28386msec); 0 zone resets slat (usec): min=22, max=29809, avg=289.82, stdev=1200.56 clat (msec): min=62, max=393, avg=173.19, stdev=49.80 lat (msec): min=63, max=394, avg=173.49, stdev=49.77 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 129], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 326], 99.50th=[ 342], 99.90th=[ 380], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 102, max= 316, per=24.78%, avg=182.16, stdev=46.15, samples=56 iops : min= 25, max= 79, avg=45.00, stdev=11.55, samples=56 lat (msec) : 20=0.04%, 100=5.86%, 250=88.48%, 500=5.62% cpu : usr=0.35%, sys=2.22%, ctx=2726, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=707KiB/s (724kB/s), 176KiB/s-178KiB/s (180kB/s-183kB/s), io=19.6MiB (20.5MB), run=28205-28386msec WRITE: bw=736KiB/s (754kB/s), 182KiB/s-187KiB/s (187kB/s-191kB/s), io=20.4MiB (21.4MB), run=28205-28386msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148932: Tue Apr 14 14:55:02 2026 read: IOPS=1, BW=61.7MiB/s (64.7MB/s)(40.0MiB/648msec) slat (nsec): min=44748k, max=44748k, avg=44748040.00, stdev= 0.00 clat (nsec): min=589466k, max=589466k, avg=589466098.00, stdev= 0.00 lat (nsec): min=634219k, max=634219k, avg=634219103.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] lat (msec) : 750=100.00% cpu : usr=0.31%, sys=3.09%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=61.7MiB/s (64.7MB/s), 61.7MiB/s-61.7MiB/s (64.7MB/s-64.7MB/s), io=40.0MiB (41.9MB), run=648-648msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck