== sanity test 398c: run fio to test AIO ================= 10:39:47 (1761316787) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.388014 s, 108 MB/s osc.lustre-OST0000-osc-ffffa09d48239000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151847: Fri Oct 24 10:40:15 2025 write: IOPS=124, BW=498KiB/s (510kB/s)(10.0MiB/20552msec); 0 zone resets slat (usec): min=25, max=7012, avg=133.17, stdev=349.64 clat (msec): min=18, max=232, avg=127.71, stdev=26.79 lat (msec): min=18, max=232, avg=127.85, stdev=26.82 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 121], 50.00th=[ 127], 60.00th=[ 133], | 70.00th=[ 140], 80.00th=[ 146], 90.00th=[ 159], 95.00th=[ 176], | 99.00th=[ 203], 99.50th=[ 209], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 375, max= 648, per=24.94%, avg=496.77, stdev=63.72, samples=40 iops : min= 93, max= 162, avg=123.83, stdev=16.03, samples=40 lat (msec) : 20=0.04%, 50=0.16%, 100=12.70%, 250=87.11% cpu : usr=0.17%, sys=1.09%, ctx=947, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151848: Fri Oct 24 10:40:15 2025 write: IOPS=124, BW=499KiB/s (511kB/s)(10.0MiB/20507msec); 0 zone resets slat (usec): min=24, max=15136, avg=128.12, stdev=419.59 clat (msec): min=32, max=229, avg=127.89, stdev=26.87 lat (msec): min=32, max=229, avg=128.02, stdev=26.87 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 88], 10.00th=[ 95], 20.00th=[ 107], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 127], 60.00th=[ 133], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 180], | 99.00th=[ 211], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 380, max= 741, per=24.95%, avg=497.05, stdev=72.92, samples=40 iops : min= 95, max= 185, avg=123.90, stdev=18.27, samples=40 lat (msec) : 50=0.08%, 100=13.95%, 250=85.98% cpu : usr=0.18%, sys=1.17%, ctx=1022, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151849: Fri Oct 24 10:40:15 2025 write: IOPS=124, BW=499KiB/s (511kB/s)(10.0MiB/20534msec); 0 zone resets slat (usec): min=24, max=6040, avg=153.87, stdev=394.29 clat (msec): min=11, max=234, avg=127.60, stdev=26.43 lat (msec): min=12, max=235, avg=127.75, stdev=26.45 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 108], | 30.00th=[ 113], 40.00th=[ 120], 50.00th=[ 126], 60.00th=[ 132], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 163], 95.00th=[ 174], | 99.00th=[ 201], 99.50th=[ 218], 99.90th=[ 228], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 375, max= 645, per=24.90%, avg=496.05, stdev=68.89, samples=40 iops : min= 93, max= 161, avg=123.63, stdev=17.30, samples=40 lat (msec) : 20=0.04%, 100=13.28%, 250=86.68% cpu : usr=0.23%, sys=1.28%, ctx=1436, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151850: Fri Oct 24 10:40:15 2025 write: IOPS=124, BW=499KiB/s (511kB/s)(10.0MiB/20530msec); 0 zone resets slat (usec): min=24, max=4247, avg=151.95, stdev=379.79 clat (msec): min=21, max=228, avg=127.68, stdev=26.90 lat (msec): min=21, max=228, avg=127.83, stdev=26.90 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 132], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 174], | 99.00th=[ 209], 99.50th=[ 224], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 375, max= 712, per=24.96%, avg=497.25, stdev=73.77, samples=40 iops : min= 93, max= 178, avg=123.93, stdev=18.53, samples=40 lat (msec) : 50=0.12%, 100=13.52%, 250=86.37% cpu : usr=0.16%, sys=1.21%, ctx=1316, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1993KiB/s (2041kB/s), 498KiB/s-499KiB/s (510kB/s-511kB/s), io=40.0MiB (41.9MB), run=20507-20552msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151862: Fri Oct 24 10:40:41 2025 read: IOPS=64, BW=257KiB/s (264kB/s)(5048KiB/19612msec) slat (usec): min=1900, max=25257, avg=7750.56, stdev=3376.03 clat (msec): min=31, max=204, avg=115.98, stdev=23.60 lat (msec): min=38, max=218, avg=123.74, stdev=24.41 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 81], 10.00th=[ 88], 20.00th=[ 96], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 114], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 136], 90.00th=[ 148], 95.00th=[ 157], | 99.00th=[ 182], 99.50th=[ 192], 99.90th=[ 201], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 142, max= 344, per=25.05%, avg=256.24, stdev=44.22, samples=38 iops : min= 35, max= 86, avg=63.63, stdev=11.21, samples=38 write: IOPS=66, BW=265KiB/s (271kB/s)(5192KiB/19612msec); 0 zone resets slat (usec): min=25, max=5845, avg=157.85, stdev=369.83 clat (msec): min=11, max=209, avg=120.10, stdev=24.63 lat (msec): min=11, max=210, avg=120.26, stdev=24.63 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 86], 10.00th=[ 92], 20.00th=[ 100], | 30.00th=[ 106], 40.00th=[ 112], 50.00th=[ 118], 60.00th=[ 125], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 161], | 99.00th=[ 194], 99.50th=[ 199], 99.90th=[ 203], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 184, max= 384, per=24.85%, avg=264.68, stdev=48.23, samples=38 iops : min= 46, max= 96, avg=65.76, stdev=12.14, samples=38 lat (msec) : 20=0.04%, 50=0.16%, 100=23.36%, 250=76.45% cpu : usr=0.35%, sys=2.27%, ctx=2544, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151863: Fri Oct 24 10:40:41 2025 read: IOPS=63, BW=254KiB/s (260kB/s)(4976KiB/19588msec) slat (usec): min=2101, max=26402, avg=7698.85, stdev=3423.48 clat (msec): min=48, max=210, avg=116.46, stdev=23.69 lat (msec): min=56, max=219, avg=124.16, stdev=24.29 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 82], 10.00th=[ 88], 20.00th=[ 96], | 30.00th=[ 104], 40.00th=[ 108], 50.00th=[ 114], 60.00th=[ 122], | 70.00th=[ 128], 80.00th=[ 136], 90.00th=[ 148], 95.00th=[ 159], | 99.00th=[ 184], 99.50th=[ 192], 99.90th=[ 205], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 183, max= 351, per=24.83%, avg=254.03, stdev=44.33, samples=38 iops : min= 45, max= 87, avg=63.11, stdev=11.19, samples=38 write: IOPS=67, BW=269KiB/s (275kB/s)(5264KiB/19588msec); 0 zone resets slat (usec): min=27, max=10210, avg=185.02, stdev=554.84 clat (msec): min=30, max=212, avg=119.55, stdev=22.92 lat (msec): min=31, max=212, avg=119.74, stdev=22.95 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 88], 10.00th=[ 93], 20.00th=[ 101], | 30.00th=[ 106], 40.00th=[ 112], 50.00th=[ 117], 60.00th=[ 124], | 70.00th=[ 130], 80.00th=[ 138], 90.00th=[ 148], 95.00th=[ 161], | 99.00th=[ 188], 99.50th=[ 199], 99.90th=[ 205], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 142, max= 368, per=25.26%, avg=269.05, stdev=46.20, samples=38 iops : min= 35, max= 92, avg=66.84, stdev=11.63, samples=38 lat (msec) : 50=0.08%, 100=22.62%, 250=77.30% cpu : usr=0.27%, sys=2.37%, ctx=2775, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151864: Fri Oct 24 10:40:41 2025 read: IOPS=63, BW=254KiB/s (260kB/s)(4984KiB/19616msec) slat (usec): min=1934, max=27100, avg=7712.29, stdev=3526.43 clat (msec): min=29, max=212, avg=116.18, stdev=24.35 lat (msec): min=36, max=221, avg=123.90, stdev=25.16 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 83], 10.00th=[ 88], 20.00th=[ 95], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 114], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 136], 90.00th=[ 148], 95.00th=[ 161], | 99.00th=[ 190], 99.50th=[ 201], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 157, max= 384, per=24.86%, avg=254.32, stdev=47.03, samples=38 iops : min= 39, max= 96, avg=63.16, stdev=11.87, samples=38 write: IOPS=66, BW=268KiB/s (274kB/s)(5256KiB/19616msec); 0 zone resets slat (usec): min=26, max=32919, avg=221.05, stdev=1061.40 clat (msec): min=18, max=225, avg=119.98, stdev=24.14 lat (msec): min=18, max=225, avg=120.20, stdev=24.18 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 88], 10.00th=[ 93], 20.00th=[ 101], | 30.00th=[ 107], 40.00th=[ 112], 50.00th=[ 118], 60.00th=[ 124], | 70.00th=[ 130], 80.00th=[ 138], 90.00th=[ 150], 95.00th=[ 163], | 99.00th=[ 194], 99.50th=[ 209], 99.90th=[ 222], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 127, max= 360, per=24.98%, avg=266.03, stdev=52.50, samples=38 iops : min= 31, max= 90, avg=66.08, stdev=13.21, samples=38 lat (msec) : 20=0.04%, 50=0.12%, 100=22.46%, 250=77.38% cpu : usr=0.32%, sys=2.31%, ctx=2659, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151865: Fri Oct 24 10:40:41 2025 read: IOPS=65, BW=260KiB/s (267kB/s)(5060KiB/19436msec) slat (usec): min=33, max=23968, avg=7542.69, stdev=3377.38 clat (msec): min=10, max=213, avg=114.49, stdev=24.75 lat (msec): min=11, max=221, avg=122.04, stdev=25.57 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 80], 10.00th=[ 86], 20.00th=[ 94], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 113], 60.00th=[ 120], | 70.00th=[ 127], 80.00th=[ 134], 90.00th=[ 146], 95.00th=[ 157], | 99.00th=[ 180], 99.50th=[ 188], 99.90th=[ 209], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 149, max= 384, per=25.45%, avg=260.39, stdev=49.81, samples=38 iops : min= 37, max= 96, avg=64.68, stdev=12.54, samples=38 write: IOPS=66, BW=267KiB/s (273kB/s)(5180KiB/19436msec); 0 zone resets slat (usec): min=27, max=6483, avg=169.23, stdev=432.95 clat (msec): min=35, max=231, avg=120.39, stdev=24.33 lat (msec): min=35, max=231, avg=120.56, stdev=24.35 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 101], | 30.00th=[ 107], 40.00th=[ 113], 50.00th=[ 118], 60.00th=[ 125], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 161], | 99.00th=[ 188], 99.50th=[ 203], 99.90th=[ 226], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 192, max= 360, per=24.95%, avg=265.74, stdev=40.49, samples=38 iops : min= 48, max= 90, avg=66.03, stdev=10.23, samples=38 lat (msec) : 20=0.12%, 50=0.27%, 100=23.67%, 250=75.94% cpu : usr=0.29%, sys=2.22%, ctx=2668, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1023KiB/s (1048kB/s), 254KiB/s-260KiB/s (260kB/s-267kB/s), io=19.6MiB (20.5MB), run=19436-19616msec WRITE: bw=1065KiB/s (1091kB/s), 265KiB/s-269KiB/s (271kB/s-275kB/s), io=20.4MiB (21.4MB), run=19436-19616msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151874: Fri Oct 24 10:40:49 2025 read: IOPS=3, BW=125MiB/s (131MB/s)(40.0MiB/320msec) slat (nsec): min=29960k, max=29960k, avg=29960078.00, stdev= 0.00 clat (nsec): min=260081k, max=260081k, avg=260080886.00, stdev= 0.00 lat (nsec): min=290045k, max=290045k, avg=290045277.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 262], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.45%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=40.0MiB (41.9MB), run=320-320msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck