== sanity test 398c: run fio to test AIO ================= 10:18:00 (1761315480) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.220902 s, 190 MB/s osc.lustre-OST0000-osc-ffff98a1ca61c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149606: Fri Oct 24 10:18:24 2025 write: IOPS=144, BW=576KiB/s (590kB/s)(10.0MiB/17764msec); 0 zone resets slat (usec): min=22, max=7068, avg=154.51, stdev=440.62 clat (msec): min=10, max=193, avg=110.48, stdev=25.58 lat (msec): min=10, max=193, avg=110.64, stdev=25.59 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 77], 10.00th=[ 81], 20.00th=[ 89], | 30.00th=[ 96], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 116], | 70.00th=[ 123], 80.00th=[ 132], 90.00th=[ 144], 95.00th=[ 155], | 99.00th=[ 178], 99.50th=[ 182], 99.90th=[ 186], 99.95th=[ 188], | 99.99th=[ 194] bw ( KiB/s): min= 431, max= 782, per=24.86%, avg=573.03, stdev=86.12, samples=35 iops : min= 107, max= 195, avg=142.97, stdev=21.52, samples=35 lat (msec) : 20=0.16%, 50=0.86%, 100=35.66%, 250=63.32% cpu : usr=0.31%, sys=1.55%, ctx=1298, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149607: Fri Oct 24 10:18:24 2025 write: IOPS=144, BW=579KiB/s (593kB/s)(10.0MiB/17675msec); 0 zone resets slat (usec): min=25, max=13709, avg=172.04, stdev=520.82 clat (msec): min=12, max=192, avg=110.03, stdev=27.08 lat (msec): min=13, max=192, avg=110.21, stdev=27.07 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 89], | 30.00th=[ 95], 40.00th=[ 102], 50.00th=[ 109], 60.00th=[ 116], | 70.00th=[ 123], 80.00th=[ 131], 90.00th=[ 146], 95.00th=[ 159], | 99.00th=[ 180], 99.50th=[ 188], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 406, max= 774, per=25.01%, avg=576.37, stdev=105.28, samples=35 iops : min= 101, max= 193, avg=143.80, stdev=26.31, samples=35 lat (msec) : 20=0.16%, 50=1.52%, 100=36.13%, 250=62.19% cpu : usr=0.27%, sys=1.59%, ctx=1483, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149608: Fri Oct 24 10:18:24 2025 write: IOPS=144, BW=578KiB/s (592kB/s)(10.0MiB/17725msec); 0 zone resets slat (usec): min=24, max=4995, avg=133.88, stdev=335.48 clat (msec): min=6, max=212, avg=110.54, stdev=26.64 lat (msec): min=6, max=212, avg=110.68, stdev=26.65 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 71], 10.00th=[ 81], 20.00th=[ 89], | 30.00th=[ 95], 40.00th=[ 102], 50.00th=[ 109], 60.00th=[ 116], | 70.00th=[ 124], 80.00th=[ 132], 90.00th=[ 146], 95.00th=[ 159], | 99.00th=[ 182], 99.50th=[ 184], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 213] bw ( KiB/s): min= 455, max= 768, per=25.01%, avg=576.37, stdev=91.80, samples=35 iops : min= 113, max= 192, avg=143.80, stdev=22.94, samples=35 lat (msec) : 10=0.04%, 20=0.12%, 50=1.29%, 100=36.76%, 250=61.80% cpu : usr=0.25%, sys=1.42%, ctx=1141, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149609: Fri Oct 24 10:18:24 2025 write: IOPS=144, BW=577KiB/s (591kB/s)(10.0MiB/17740msec); 0 zone resets slat (usec): min=24, max=4653, avg=116.28, stdev=295.89 clat (msec): min=13, max=190, avg=110.41, stdev=25.67 lat (msec): min=13, max=191, avg=110.53, stdev=25.69 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 74], 10.00th=[ 80], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 104], 50.00th=[ 108], 60.00th=[ 114], | 70.00th=[ 121], 80.00th=[ 131], 90.00th=[ 146], 95.00th=[ 161], | 99.00th=[ 176], 99.50th=[ 180], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 407, max= 894, per=24.93%, avg=574.54, stdev=101.20, samples=35 iops : min= 101, max= 223, avg=143.37, stdev=25.31, samples=35 lat (msec) : 20=0.04%, 50=0.86%, 100=34.41%, 250=64.69% cpu : usr=0.21%, sys=1.08%, ctx=625, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2306KiB/s (2361kB/s), 576KiB/s-579KiB/s (590kB/s-593kB/s), io=40.0MiB (41.9MB), run=17675-17764msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149622: Fri Oct 24 10:18:49 2025 read: IOPS=70, BW=282KiB/s (289kB/s)(5048KiB/17908msec) slat (usec): min=1958, max=42682, avg=7236.16, stdev=3755.19 clat (msec): min=5, max=239, avg=105.55, stdev=24.28 lat (msec): min=9, max=253, avg=112.79, stdev=25.17 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 86], | 30.00th=[ 93], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 126], 90.00th=[ 136], 95.00th=[ 146], | 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 203], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 168, max= 415, per=25.11%, avg=280.74, stdev=47.66, samples=35 iops : min= 42, max= 103, avg=69.86, stdev=11.88, samples=35 write: IOPS=72, BW=290KiB/s (297kB/s)(5192KiB/17908msec); 0 zone resets slat (usec): min=26, max=13202, avg=175.75, stdev=531.90 clat (msec): min=20, max=229, avg=109.51, stdev=24.10 lat (msec): min=20, max=229, avg=109.69, stdev=24.09 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 71], 10.00th=[ 80], 20.00th=[ 90], | 30.00th=[ 97], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 115], | 70.00th=[ 122], 80.00th=[ 128], 90.00th=[ 140], 95.00th=[ 153], | 99.00th=[ 174], 99.50th=[ 190], 99.90th=[ 209], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 199, max= 454, per=24.90%, avg=289.86, stdev=57.98, samples=35 iops : min= 49, max= 113, avg=72.11, stdev=14.47, samples=35 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=38.52%, 250=61.25% cpu : usr=0.37%, sys=2.41%, ctx=2676, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149623: Fri Oct 24 10:18:49 2025 read: IOPS=69, BW=277KiB/s (284kB/s)(4976KiB/17937msec) slat (usec): min=1755, max=41689, avg=7008.78, stdev=3572.60 clat (msec): min=7, max=194, avg=106.71, stdev=24.43 lat (msec): min=17, max=201, avg=113.73, stdev=25.11 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 77], 20.00th=[ 87], | 30.00th=[ 94], 40.00th=[ 100], 50.00th=[ 106], 60.00th=[ 112], | 70.00th=[ 120], 80.00th=[ 128], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 174], 99.50th=[ 182], 99.90th=[ 190], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 200, max= 424, per=24.69%, avg=276.06, stdev=49.81, samples=35 iops : min= 50, max= 106, avg=68.69, stdev=12.45, samples=35 write: IOPS=73, BW=293KiB/s (301kB/s)(5264KiB/17937msec); 0 zone resets slat (usec): min=25, max=3759, avg=154.30, stdev=305.80 clat (msec): min=12, max=208, avg=109.46, stdev=23.50 lat (msec): min=12, max=208, avg=109.61, stdev=23.51 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 90], | 30.00th=[ 99], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 115], | 70.00th=[ 121], 80.00th=[ 128], 90.00th=[ 140], 95.00th=[ 153], | 99.00th=[ 171], 99.50th=[ 180], 99.90th=[ 199], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 160, max= 470, per=25.14%, avg=292.57, stdev=65.32, samples=35 iops : min= 40, max= 117, avg=72.80, stdev=16.25, samples=35 lat (msec) : 10=0.04%, 20=0.08%, 50=0.16%, 100=37.38%, 250=62.34% cpu : usr=0.30%, sys=2.31%, ctx=2651, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149624: Fri Oct 24 10:18:49 2025 read: IOPS=69, BW=279KiB/s (286kB/s)(4984KiB/17868msec) slat (usec): min=30, max=31560, avg=6982.64, stdev=3550.72 clat (msec): min=10, max=191, avg=105.17, stdev=23.81 lat (msec): min=19, max=199, avg=112.16, stdev=24.67 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 85], | 30.00th=[ 92], 40.00th=[ 99], 50.00th=[ 104], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 125], 90.00th=[ 136], 95.00th=[ 146], | 99.00th=[ 169], 99.50th=[ 182], 99.90th=[ 190], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 184, max= 431, per=25.00%, avg=279.49, stdev=56.99, samples=35 iops : min= 46, max= 107, avg=69.54, stdev=14.21, samples=35 write: IOPS=73, BW=294KiB/s (301kB/s)(5256KiB/17868msec); 0 zone resets slat (usec): min=26, max=9842, avg=174.98, stdev=507.37 clat (msec): min=10, max=211, avg=110.38, stdev=24.22 lat (msec): min=10, max=211, avg=110.56, stdev=24.20 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 71], 10.00th=[ 81], 20.00th=[ 92], | 30.00th=[ 99], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 115], | 70.00th=[ 122], 80.00th=[ 130], 90.00th=[ 140], 95.00th=[ 150], | 99.00th=[ 180], 99.50th=[ 197], 99.90th=[ 205], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 192, max= 400, per=25.17%, avg=293.00, stdev=56.81, samples=35 iops : min= 48, max= 100, avg=72.91, stdev=14.21, samples=35 lat (msec) : 20=0.08%, 100=38.36%, 250=61.56% cpu : usr=0.29%, sys=2.29%, ctx=2544, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149625: Fri Oct 24 10:18:49 2025 read: IOPS=70, BW=284KiB/s (290kB/s)(5060KiB/17843msec) slat (usec): min=50, max=32418, avg=6906.33, stdev=3496.16 clat (msec): min=6, max=235, avg=105.47, stdev=25.51 lat (msec): min=6, max=245, avg=112.39, stdev=26.50 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 85], | 30.00th=[ 92], 40.00th=[ 99], 50.00th=[ 106], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 127], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 174], 99.50th=[ 184], 99.90th=[ 224], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 190, max= 407, per=25.43%, avg=284.26, stdev=54.38, samples=35 iops : min= 47, max= 101, avg=70.71, stdev=13.59, samples=35 write: IOPS=72, BW=290KiB/s (297kB/s)(5180KiB/17843msec); 0 zone resets slat (usec): min=24, max=4160, avg=143.57, stdev=276.44 clat (msec): min=25, max=217, avg=110.20, stdev=24.10 lat (msec): min=25, max=217, avg=110.34, stdev=24.11 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 90], | 30.00th=[ 99], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 114], | 70.00th=[ 121], 80.00th=[ 129], 90.00th=[ 142], 95.00th=[ 153], | 99.00th=[ 180], 99.50th=[ 197], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 152, max= 408, per=24.88%, avg=289.60, stdev=57.52, samples=35 iops : min= 38, max= 102, avg=72.06, stdev=14.36, samples=35 lat (msec) : 10=0.04%, 50=0.16%, 100=37.77%, 250=62.03% cpu : usr=0.35%, sys=2.30%, ctx=2684, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1119KiB/s (1146kB/s), 277KiB/s-284KiB/s (284kB/s-290kB/s), io=19.6MiB (20.5MB), run=17843-17937msec WRITE: bw=1165KiB/s (1193kB/s), 290KiB/s-294KiB/s (297kB/s-301kB/s), io=20.4MiB (21.4MB), run=17843-17937msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149632: Fri Oct 24 10:18:58 2025 read: IOPS=2, BW=118MiB/s (124MB/s)(40.0MiB/338msec) slat (nsec): min=35116k, max=35116k, avg=35115759.00, stdev= 0.00 clat (nsec): min=290793k, max=290793k, avg=290792943.00, stdev= 0.00 lat (nsec): min=325914k, max=325914k, avg=325913827.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 500=100.00% cpu : usr=0.59%, sys=5.64%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=40.0MiB (41.9MB), run=338-338msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck