== sanity test 398c: run fio to test AIO ================= 10:44:58 (1760453098) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.106494 s, 394 MB/s osc.lustre-OST0000-osc-ffff94c8d836a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=122198: Tue Oct 14 10:45:27 2025 write: IOPS=99, BW=396KiB/s (406kB/s)(10.0MiB/25851msec); 0 zone resets slat (usec): min=27, max=463, avg=103.43, stdev=47.98 clat (msec): min=61, max=271, avg=161.35, stdev=32.17 lat (msec): min=61, max=271, avg=161.46, stdev=32.18 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 105], 10.00th=[ 121], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 201], 95.00th=[ 211], | 99.00th=[ 234], 99.50th=[ 239], 99.90th=[ 253], 99.95th=[ 262], | 99.99th=[ 271] bw ( KiB/s): min= 304, max= 648, per=24.93%, avg=394.82, stdev=59.86, samples=51 iops : min= 76, max= 162, avg=98.71, stdev=14.97, samples=51 lat (msec) : 100=3.52%, 250=96.29%, 500=0.20% cpu : usr=0.15%, sys=1.02%, ctx=1977, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122199: Tue Oct 14 10:45:27 2025 write: IOPS=99, BW=396KiB/s (406kB/s)(10.0MiB/25852msec); 0 zone resets slat (usec): min=25, max=468, avg=105.28, stdev=49.99 clat (msec): min=58, max=256, avg=161.37, stdev=32.25 lat (msec): min=58, max=256, avg=161.47, stdev=32.26 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 105], 10.00th=[ 121], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 201], 95.00th=[ 211], | 99.00th=[ 232], 99.50th=[ 234], 99.90th=[ 249], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 288, max= 654, per=24.92%, avg=394.78, stdev=62.84, samples=51 iops : min= 72, max= 163, avg=98.69, stdev=15.67, samples=51 lat (msec) : 100=3.16%, 250=96.76%, 500=0.08% cpu : usr=0.18%, sys=1.00%, ctx=2057, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122200: Tue Oct 14 10:45:27 2025 write: IOPS=99, BW=397KiB/s (406kB/s)(10.0MiB/25802msec); 0 zone resets slat (usec): min=23, max=557, avg=105.10, stdev=50.95 clat (msec): min=17, max=293, avg=161.07, stdev=33.44 lat (msec): min=17, max=294, avg=161.18, stdev=33.44 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 106], 10.00th=[ 121], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 201], 95.00th=[ 211], | 99.00th=[ 234], 99.50th=[ 241], 99.90th=[ 268], 99.95th=[ 284], | 99.99th=[ 296] bw ( KiB/s): min= 304, max= 719, per=24.98%, avg=395.75, stdev=66.61, samples=51 iops : min= 76, max= 179, avg=98.92, stdev=16.58, samples=51 lat (msec) : 20=0.04%, 50=0.12%, 100=4.06%, 250=95.51%, 500=0.27% cpu : usr=0.15%, sys=1.03%, ctx=1927, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122201: Tue Oct 14 10:45:27 2025 write: IOPS=99, BW=396KiB/s (406kB/s)(10.0MiB/25851msec); 0 zone resets slat (usec): min=26, max=877, avg=104.75, stdev=52.05 clat (msec): min=16, max=268, avg=161.37, stdev=32.48 lat (msec): min=16, max=268, avg=161.47, stdev=32.48 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 105], 10.00th=[ 120], 20.00th=[ 133], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 203], 95.00th=[ 213], | 99.00th=[ 234], 99.50th=[ 243], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 268] bw ( KiB/s): min= 312, max= 642, per=24.93%, avg=394.86, stdev=61.21, samples=51 iops : min= 78, max= 160, avg=98.71, stdev=15.26, samples=51 lat (msec) : 20=0.04%, 100=3.48%, 250=96.37%, 500=0.12% cpu : usr=0.17%, sys=1.01%, ctx=1972, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1584KiB/s (1622kB/s), 396KiB/s-397KiB/s (406kB/s-406kB/s), io=40.0MiB (41.9MB), run=25802-25852msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=122214: Tue Oct 14 10:45:45 2025 read: IOPS=80, BW=324KiB/s (332kB/s)(5048KiB/15586msec) slat (usec): min=670, max=6272, avg=1239.15, stdev=482.61 clat (usec): min=1657, max=117794, avg=67011.81, stdev=12402.76 lat (msec): min=3, max=119, avg=68.25, stdev=12.42 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 83], 95.00th=[ 87], | 99.00th=[ 101], 99.50th=[ 105], 99.90th=[ 116], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 216, max= 432, per=25.22%, avg=324.60, stdev=62.83, samples=30 iops : min= 54, max= 108, avg=81.13, stdev=15.72, samples=30 write: IOPS=83, BW=333KiB/s (341kB/s)(5192KiB/15586msec); 0 zone resets slat (usec): min=26, max=1204, avg=94.52, stdev=62.91 clat (msec): min=62, max=222, avg=125.42, stdev=29.59 lat (msec): min=62, max=222, avg=125.51, stdev=29.60 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 82], 10.00th=[ 90], 20.00th=[ 100], | 30.00th=[ 108], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 130], | 70.00th=[ 140], 80.00th=[ 150], 90.00th=[ 169], 95.00th=[ 182], | 99.00th=[ 197], 99.50th=[ 205], 99.90th=[ 222], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 288, max= 408, per=24.82%, avg=332.53, stdev=30.85, samples=30 iops : min= 72, max= 102, avg=83.13, stdev= 7.71, samples=30 lat (msec) : 2=0.04%, 20=0.16%, 50=2.73%, 100=56.45%, 250=40.62% cpu : usr=0.28%, sys=2.00%, ctx=3134, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122215: Tue Oct 14 10:45:45 2025 read: IOPS=79, BW=319KiB/s (327kB/s)(4976KiB/15576msec) slat (usec): min=46, max=5666, avg=1213.23, stdev=464.92 clat (usec): min=1025, max=127878, avg=66837.12, stdev=12970.71 lat (usec): min=1548, max=128915, avg=68051.82, stdev=13008.76 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 96], 99.50th=[ 103], 99.90th=[ 106], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 200, max= 424, per=24.82%, avg=319.40, stdev=61.10, samples=30 iops : min= 50, max= 106, avg=79.83, stdev=15.27, samples=30 write: IOPS=84, BW=338KiB/s (346kB/s)(5264KiB/15576msec); 0 zone resets slat (usec): min=23, max=1063, avg=95.43, stdev=58.31 clat (msec): min=13, max=220, avg=124.82, stdev=30.62 lat (msec): min=13, max=220, avg=124.91, stdev=30.62 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 81], 10.00th=[ 90], 20.00th=[ 100], | 30.00th=[ 108], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 130], | 70.00th=[ 140], 80.00th=[ 150], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 201], 99.50th=[ 203], 99.90th=[ 220], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 272, max= 424, per=25.23%, avg=338.03, stdev=44.47, samples=30 iops : min= 68, max= 106, avg=84.50, stdev=11.12, samples=30 lat (msec) : 2=0.12%, 10=0.12%, 20=0.35%, 50=2.62%, 100=55.82% lat (msec) : 250=40.98% cpu : usr=0.27%, sys=1.99%, ctx=3100, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122216: Tue Oct 14 10:45:45 2025 read: IOPS=80, BW=320KiB/s (328kB/s)(4984KiB/15565msec) slat (usec): min=632, max=5158, avg=1254.22, stdev=498.73 clat (usec): min=867, max=128291, avg=66391.94, stdev=12115.09 lat (msec): min=2, max=129, avg=67.65, stdev=12.16 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 85], | 99.00th=[ 96], 99.50th=[ 100], 99.90th=[ 113], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 224, max= 472, per=25.08%, avg=322.73, stdev=62.11, samples=30 iops : min= 56, max= 118, avg=80.67, stdev=15.54, samples=30 write: IOPS=84, BW=338KiB/s (346kB/s)(5256KiB/15565msec); 0 zone resets slat (usec): min=26, max=457, avg=94.08, stdev=51.44 clat (msec): min=33, max=218, avg=125.08, stdev=29.60 lat (msec): min=33, max=218, avg=125.18, stdev=29.60 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 81], 10.00th=[ 90], 20.00th=[ 100], | 30.00th=[ 108], 40.00th=[ 115], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 140], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 197], 99.50th=[ 203], 99.90th=[ 205], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 256, max= 416, per=25.10%, avg=336.37, stdev=36.33, samples=30 iops : min= 64, max= 104, avg=84.07, stdev= 9.10, samples=30 lat (usec) : 1000=0.04% lat (msec) : 10=0.08%, 20=0.12%, 50=3.36%, 100=55.62%, 250=40.78% cpu : usr=0.27%, sys=2.02%, ctx=3211, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122217: Tue Oct 14 10:45:45 2025 read: IOPS=81, BW=325KiB/s (332kB/s)(5060KiB/15586msec) slat (usec): min=654, max=5003, avg=1246.83, stdev=472.72 clat (msec): min=2, max=149, avg=66.62, stdev=12.82 lat (msec): min=4, max=150, avg=67.87, stdev=12.85 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 140], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 216, max= 424, per=25.34%, avg=326.07, stdev=53.38, samples=30 iops : min= 54, max= 106, avg=81.50, stdev=13.36, samples=30 write: IOPS=83, BW=332KiB/s (340kB/s)(5180KiB/15586msec); 0 zone resets slat (usec): min=26, max=445, avg=93.41, stdev=47.91 clat (msec): min=57, max=220, avg=125.95, stdev=29.67 lat (msec): min=57, max=220, avg=126.05, stdev=29.67 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 83], 10.00th=[ 89], 20.00th=[ 101], | 30.00th=[ 109], 40.00th=[ 117], 50.00th=[ 124], 60.00th=[ 130], | 70.00th=[ 140], 80.00th=[ 153], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 201], 99.50th=[ 205], 99.90th=[ 220], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 240, max= 392, per=24.77%, avg=331.90, stdev=33.82, samples=30 iops : min= 60, max= 98, avg=82.97, stdev= 8.46, samples=30 lat (msec) : 4=0.04%, 20=0.27%, 50=2.85%, 100=55.78%, 250=41.05% cpu : usr=0.19%, sys=2.08%, ctx=3188, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1288KiB/s (1318kB/s), 319KiB/s-325KiB/s (327kB/s-332kB/s), io=19.6MiB (20.5MB), run=15565-15586msec WRITE: bw=1340KiB/s (1373kB/s), 332KiB/s-338KiB/s (340kB/s-346kB/s), io=20.4MiB (21.4MB), run=15565-15586msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=122223: Tue Oct 14 10:45:48 2025 read: IOPS=13, BW=541MiB/s (567MB/s)(40.0MiB/74msec) slat (nsec): min=10228k, max=10228k, avg=10228257.00, stdev= 0.00 clat (nsec): min=59648k, max=59648k, avg=59648472.00, stdev= 0.00 lat (nsec): min=69881k, max=69881k, avg=69880992.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[59507], 5.00th=[59507], 10.00th=[59507], 20.00th=[59507], | 30.00th=[59507], 40.00th=[59507], 50.00th=[59507], 60.00th=[59507], | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=13.70%, ctx=2, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=541MiB/s (567MB/s), 541MiB/s-541MiB/s (567MB/s-567MB/s), io=40.0MiB (41.9MB), run=74-74msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck