== sanity test 398c: run fio to test AIO ================= 17:23:56 (1776201836) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.417301 s, 101 MB/s osc.lustre-OST0000-osc-ffff952851abe000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149555: Tue Apr 14 17:24:37 2026 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30512msec); 0 zone resets slat (usec): min=21, max=19813, avg=244.39, stdev=842.00 clat (msec): min=16, max=333, avg=189.86, stdev=42.54 lat (msec): min=17, max=333, avg=190.11, stdev=42.54 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 268], | 99.00th=[ 300], 99.50th=[ 330], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 222, max= 494, per=24.92%, avg=334.13, stdev=53.36, samples=60 iops : min= 55, max= 123, avg=83.10, stdev=13.26, samples=60 lat (msec) : 20=0.04%, 50=0.12%, 100=0.70%, 250=91.84%, 500=7.30% cpu : usr=0.23%, sys=0.84%, ctx=1130, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149556: Tue Apr 14 17:24:37 2026 write: IOPS=84, BW=336KiB/s (344kB/s)(10.0MiB/30466msec); 0 zone resets slat (usec): min=21, max=14608, avg=269.55, stdev=872.13 clat (msec): min=34, max=327, avg=189.92, stdev=40.66 lat (msec): min=35, max=331, avg=190.20, stdev=40.66 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 220], 90.00th=[ 245], 95.00th=[ 264], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 200, max= 510, per=24.97%, avg=334.82, stdev=65.62, samples=60 iops : min= 50, max= 127, avg=83.30, stdev=16.35, samples=60 lat (msec) : 50=0.04%, 100=0.31%, 250=90.59%, 500=9.06% cpu : usr=0.21%, sys=1.09%, ctx=1347, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149557: Tue Apr 14 17:24:37 2026 write: IOPS=83, BW=336KiB/s (344kB/s)(10.0MiB/30485msec); 0 zone resets slat (usec): min=24, max=11680, avg=236.75, stdev=763.79 clat (msec): min=25, max=336, avg=189.47, stdev=43.49 lat (msec): min=25, max=338, avg=189.71, stdev=43.47 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 249], 95.00th=[ 264], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 334], | 99.99th=[ 338] bw ( KiB/s): min= 176, max= 509, per=24.88%, avg=333.68, stdev=64.17, samples=60 iops : min= 44, max= 127, avg=83.03, stdev=16.00, samples=60 lat (msec) : 50=0.31%, 100=0.43%, 250=89.69%, 500=9.57% cpu : usr=0.20%, sys=0.97%, ctx=1101, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149558: Tue Apr 14 17:24:37 2026 write: IOPS=83, BW=335KiB/s (343kB/s)(10.0MiB/30534msec); 0 zone resets slat (usec): min=27, max=18399, avg=325.79, stdev=1088.48 clat (msec): min=28, max=330, avg=189.73, stdev=41.68 lat (msec): min=28, max=337, avg=190.06, stdev=41.72 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 245], 95.00th=[ 264], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 246, max= 462, per=24.86%, avg=333.35, stdev=59.53, samples=60 iops : min= 61, max= 115, avg=82.93, stdev=14.78, samples=60 lat (msec) : 50=0.08%, 100=0.98%, 250=90.59%, 500=8.36% cpu : usr=0.25%, sys=1.26%, ctx=1676, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1341KiB/s (1374kB/s), 335KiB/s-336KiB/s (343kB/s-344kB/s), io=40.0MiB (41.9MB), run=30466-30534msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149573: Tue Apr 14 17:25:18 2026 read: IOPS=40, BW=162KiB/s (166kB/s)(5048KiB/31190msec) slat (usec): min=2255, max=66030, avg=13166.60, stdev=7796.39 clat (msec): min=74, max=326, avg=183.37, stdev=45.65 lat (msec): min=82, max=343, avg=196.55, stdev=46.96 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 115], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 63, max= 270, per=25.05%, avg=161.10, stdev=40.66, samples=61 iops : min= 15, max= 67, avg=39.77, stdev=10.22, samples=61 write: IOPS=41, BW=166KiB/s (170kB/s)(5192KiB/31190msec); 0 zone resets slat (usec): min=26, max=37744, avg=343.41, stdev=1563.86 clat (msec): min=71, max=358, avg=191.06, stdev=43.76 lat (msec): min=71, max=358, avg=191.41, stdev=43.83 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 75, max= 264, per=24.86%, avg=166.33, stdev=38.01, samples=61 iops : min= 18, max= 66, avg=41.08, stdev= 9.56, samples=61 lat (msec) : 100=1.41%, 250=89.69%, 500=8.91% cpu : usr=0.27%, sys=1.98%, ctx=2844, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149574: Tue Apr 14 17:25:18 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(4976KiB/30932msec) slat (usec): min=34, max=58076, avg=13327.14, stdev=7746.47 clat (msec): min=30, max=332, avg=183.29, stdev=44.06 lat (msec): min=45, max=343, avg=196.62, stdev=45.48 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 115], 10.00th=[ 129], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 257], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 326], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 79, max= 294, per=24.90%, avg=160.11, stdev=41.28, samples=61 iops : min= 19, max= 73, avg=39.51, stdev=10.37, samples=61 write: IOPS=42, BW=170KiB/s (174kB/s)(5264KiB/30932msec); 0 zone resets slat (usec): min=22, max=12770, avg=269.33, stdev=886.33 clat (msec): min=37, max=358, avg=189.43, stdev=45.23 lat (msec): min=37, max=359, avg=189.70, stdev=45.23 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 226], 90.00th=[ 247], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 94, max= 304, per=25.43%, avg=170.11, stdev=38.56, samples=61 iops : min= 23, max= 76, avg=42.03, stdev= 9.72, samples=61 lat (msec) : 50=0.08%, 100=1.13%, 250=90.59%, 500=8.20% cpu : usr=0.32%, sys=1.95%, ctx=2798, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149575: Tue Apr 14 17:25:18 2026 read: IOPS=39, BW=160KiB/s (164kB/s)(4984KiB/31157msec) slat (usec): min=2627, max=52203, avg=13484.18, stdev=7487.32 clat (msec): min=29, max=355, avg=182.14, stdev=46.94 lat (msec): min=34, max=365, avg=195.63, stdev=48.28 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 112], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 266], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 342], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 91, max= 272, per=24.92%, avg=160.25, stdev=39.44, samples=61 iops : min= 22, max= 68, avg=39.54, stdev= 9.92, samples=61 write: IOPS=42, BW=169KiB/s (173kB/s)(5256KiB/31157msec); 0 zone resets slat (usec): min=27, max=9312, avg=276.69, stdev=775.22 clat (msec): min=41, max=381, avg=191.83, stdev=44.50 lat (msec): min=41, max=381, avg=192.11, stdev=44.50 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 127], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 199], | 70.00th=[ 213], 80.00th=[ 228], 90.00th=[ 251], 95.00th=[ 268], | 99.00th=[ 300], 99.50th=[ 326], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 83, max= 271, per=25.10%, avg=167.95, stdev=45.00, samples=61 iops : min= 20, max= 67, avg=41.48, stdev=11.30, samples=61 lat (msec) : 50=0.12%, 100=1.52%, 250=88.75%, 500=9.61% cpu : usr=0.37%, sys=2.06%, ctx=2902, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149576: Tue Apr 14 17:25:18 2026 read: IOPS=40, BW=163KiB/s (166kB/s)(5060KiB/31125msec) slat (usec): min=36, max=56742, avg=13147.78, stdev=7694.11 clat (msec): min=66, max=337, avg=185.03, stdev=44.45 lat (msec): min=76, max=363, avg=198.18, stdev=45.94 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 122], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 245], 95.00th=[ 262], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 78, max= 312, per=25.24%, avg=162.31, stdev=37.59, samples=61 iops : min= 19, max= 78, avg=40.07, stdev= 9.44, samples=61 write: IOPS=41, BW=166KiB/s (170kB/s)(5180KiB/31125msec); 0 zone resets slat (usec): min=27, max=13060, avg=287.11, stdev=910.11 clat (msec): min=84, max=341, avg=189.96, stdev=42.67 lat (msec): min=84, max=342, avg=190.25, stdev=42.70 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 197], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 90, max= 271, per=24.84%, avg=166.15, stdev=40.39, samples=61 iops : min= 22, max= 67, avg=41.03, stdev=10.13, samples=61 lat (msec) : 100=1.05%, 250=90.31%, 500=8.63% cpu : usr=0.33%, sys=1.90%, ctx=2829, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=643KiB/s (659kB/s), 160KiB/s-163KiB/s (164kB/s-166kB/s), io=19.6MiB (20.5MB), run=30932-31190msec WRITE: bw=670KiB/s (686kB/s), 166KiB/s-170KiB/s (170kB/s-174kB/s), io=20.4MiB (21.4MB), run=30932-31190msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149584: Tue Apr 14 17:25:28 2026 read: IOPS=2, BW=96.2MiB/s (101MB/s)(40.0MiB/416msec) slat (nsec): min=28915k, max=28915k, avg=28915444.00, stdev= 0.00 clat (nsec): min=373038k, max=373038k, avg=373037990.00, stdev= 0.00 lat (nsec): min=401959k, max=401959k, avg=401958802.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 500=100.00% cpu : usr=0.24%, sys=2.89%, ctx=3, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.2MiB/s (101MB/s), 96.2MiB/s-96.2MiB/s (101MB/s-101MB/s), io=40.0MiB (41.9MB), run=416-416msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck