== sanity test 398c: run fio to test AIO ================= 09:49:52 (1761313792) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.19751 s, 212 MB/s osc.lustre-OST0000-osc-ffff9f6606e34000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148976: Fri Oct 24 09:50:15 2025 write: IOPS=138, BW=552KiB/s (566kB/s)(10.0MiB/18539msec); 0 zone resets slat (usec): min=20, max=5467, avg=153.03, stdev=371.10 clat (msec): min=16, max=223, avg=115.64, stdev=26.15 lat (msec): min=16, max=223, avg=115.79, stdev=26.17 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 79], 10.00th=[ 83], 20.00th=[ 93], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 113], 60.00th=[ 121], | 70.00th=[ 129], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 157], | 99.00th=[ 184], 99.50th=[ 201], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 224] bw ( KiB/s): min= 399, max= 760, per=25.08%, avg=551.17, stdev=93.78, samples=36 iops : min= 99, max= 190, avg=137.39, stdev=23.47, samples=36 lat (msec) : 20=0.12%, 50=0.08%, 100=28.59%, 250=71.21% cpu : usr=0.23%, sys=1.46%, ctx=1527, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148977: Fri Oct 24 09:50:15 2025 write: IOPS=137, BW=550KiB/s (564kB/s)(10.0MiB/18607msec); 0 zone resets slat (usec): min=19, max=35051, avg=153.45, stdev=789.62 clat (msec): min=10, max=210, avg=115.75, stdev=26.03 lat (msec): min=11, max=210, avg=115.91, stdev=26.00 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 94], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 114], 60.00th=[ 121], | 70.00th=[ 128], 80.00th=[ 138], 90.00th=[ 148], 95.00th=[ 159], | 99.00th=[ 186], 99.50th=[ 199], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 402, max= 768, per=24.96%, avg=548.56, stdev=95.88, samples=36 iops : min= 100, max= 192, avg=136.75, stdev=24.01, samples=36 lat (msec) : 20=0.16%, 50=0.23%, 100=28.91%, 250=70.70% cpu : usr=0.23%, sys=1.40%, ctx=1387, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148978: Fri Oct 24 09:50:15 2025 write: IOPS=137, BW=550KiB/s (563kB/s)(10.0MiB/18630msec); 0 zone resets slat (usec): min=21, max=35554, avg=140.52, stdev=768.43 clat (msec): min=8, max=213, avg=115.88, stdev=25.32 lat (msec): min=8, max=214, avg=116.02, stdev=25.30 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 80], 10.00th=[ 85], 20.00th=[ 95], | 30.00th=[ 103], 40.00th=[ 108], 50.00th=[ 113], 60.00th=[ 120], | 70.00th=[ 129], 80.00th=[ 140], 90.00th=[ 148], 95.00th=[ 155], | 99.00th=[ 182], 99.50th=[ 203], 99.90th=[ 213], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 381, max= 752, per=24.93%, avg=547.94, stdev=95.94, samples=36 iops : min= 95, max= 188, avg=136.61, stdev=23.98, samples=36 lat (msec) : 10=0.04%, 20=0.04%, 100=27.97%, 250=71.95% cpu : usr=0.19%, sys=1.38%, ctx=1305, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148979: Fri Oct 24 09:50:15 2025 write: IOPS=137, BW=550KiB/s (563kB/s)(10.0MiB/18626msec); 0 zone resets slat (usec): min=25, max=9798, avg=143.79, stdev=374.99 clat (msec): min=8, max=209, avg=116.12, stdev=26.15 lat (msec): min=9, max=209, avg=116.27, stdev=26.15 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 77], 10.00th=[ 84], 20.00th=[ 95], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 113], 60.00th=[ 121], | 70.00th=[ 130], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 159], | 99.00th=[ 186], 99.50th=[ 201], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 209] bw ( KiB/s): min= 407, max= 768, per=24.92%, avg=547.72, stdev=90.64, samples=36 iops : min= 101, max= 192, avg=136.53, stdev=22.71, samples=36 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=27.03%, 250=72.73% cpu : usr=0.18%, sys=1.55%, ctx=1418, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2199KiB/s (2251kB/s), 550KiB/s-552KiB/s (563kB/s-566kB/s), io=40.0MiB (41.9MB), run=18539-18630msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148992: Fri Oct 24 09:50:37 2025 read: IOPS=81, BW=328KiB/s (336kB/s)(5048KiB/15400msec) slat (usec): min=28, max=36885, avg=6018.60, stdev=3165.94 clat (msec): min=8, max=209, avg=91.26, stdev=23.88 lat (msec): min=8, max=220, avg=97.29, stdev=24.88 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 123], 95.00th=[ 136], | 99.00th=[ 153], 99.50th=[ 161], 99.90th=[ 199], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 191, max= 453, per=25.22%, avg=326.37, stdev=67.30, samples=30 iops : min= 47, max= 113, avg=81.20, stdev=16.76, samples=30 write: IOPS=84, BW=337KiB/s (345kB/s)(5192KiB/15400msec); 0 zone resets slat (usec): min=25, max=4256, avg=126.47, stdev=217.39 clat (msec): min=30, max=206, avg=94.70, stdev=23.21 lat (msec): min=30, max=207, avg=94.82, stdev=23.20 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 75], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 93], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 113], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 161], 99.50th=[ 178], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 215, max= 461, per=25.06%, avg=337.53, stdev=69.18, samples=30 iops : min= 53, max= 115, avg=83.93, stdev=17.36, samples=30 lat (msec) : 10=0.08%, 20=0.04%, 50=0.90%, 100=63.71%, 250=35.27% cpu : usr=0.38%, sys=2.45%, ctx=2618, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148993: Fri Oct 24 09:50:37 2025 read: IOPS=80, BW=321KiB/s (329kB/s)(4976KiB/15496msec) slat (usec): min=1706, max=34178, avg=6056.70, stdev=3067.63 clat (msec): min=22, max=176, avg=91.09, stdev=22.82 lat (msec): min=28, max=190, avg=97.15, stdev=23.82 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 102], 80.00th=[ 110], 90.00th=[ 122], 95.00th=[ 131], | 99.00th=[ 155], 99.50th=[ 163], 99.90th=[ 174], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 137, max= 439, per=24.62%, avg=318.57, stdev=76.98, samples=30 iops : min= 34, max= 109, avg=79.23, stdev=19.25, samples=30 write: IOPS=84, BW=340KiB/s (348kB/s)(5264KiB/15496msec); 0 zone resets slat (usec): min=26, max=6043, avg=137.23, stdev=321.47 clat (msec): min=40, max=182, avg=95.54, stdev=23.17 lat (msec): min=40, max=182, avg=95.68, stdev=23.17 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 75], | 30.00th=[ 83], 40.00th=[ 89], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 107], 80.00th=[ 114], 90.00th=[ 127], 95.00th=[ 138], | 99.00th=[ 161], 99.50th=[ 165], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 198, max= 493, per=25.17%, avg=339.00, stdev=70.85, samples=30 iops : min= 49, max= 123, avg=84.33, stdev=17.69, samples=30 lat (msec) : 50=1.25%, 100=63.32%, 250=35.43% cpu : usr=0.31%, sys=2.55%, ctx=2666, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148994: Fri Oct 24 09:50:37 2025 read: IOPS=80, BW=322KiB/s (329kB/s)(4984KiB/15495msec) slat (usec): min=1795, max=23615, avg=5997.32, stdev=2824.47 clat (msec): min=18, max=205, avg=92.84, stdev=24.38 lat (msec): min=22, max=212, avg=98.84, stdev=25.21 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 79], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 99], | 70.00th=[ 106], 80.00th=[ 114], 90.00th=[ 124], 95.00th=[ 132], | 99.00th=[ 161], 99.50th=[ 186], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 148, max= 413, per=24.69%, avg=319.43, stdev=61.45, samples=30 iops : min= 37, max= 103, avg=79.43, stdev=15.29, samples=30 write: IOPS=84, BW=339KiB/s (347kB/s)(5256KiB/15495msec); 0 zone resets slat (usec): min=25, max=5594, avg=145.03, stdev=329.68 clat (msec): min=18, max=198, avg=93.95, stdev=23.29 lat (msec): min=18, max=198, avg=94.10, stdev=23.28 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 75], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 97], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 127], 95.00th=[ 138], | 99.00th=[ 153], 99.50th=[ 176], 99.90th=[ 188], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 171, max= 516, per=24.75%, avg=333.33, stdev=88.00, samples=30 iops : min= 42, max= 129, avg=82.90, stdev=22.06, samples=30 lat (msec) : 20=0.12%, 50=1.29%, 100=62.42%, 250=36.17% cpu : usr=0.39%, sys=2.72%, ctx=2633, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148995: Fri Oct 24 09:50:37 2025 read: IOPS=81, BW=326KiB/s (334kB/s)(5060KiB/15507msec) slat (usec): min=45, max=32012, avg=6048.82, stdev=3065.79 clat (msec): min=10, max=222, avg=91.75, stdev=24.71 lat (msec): min=15, max=227, avg=97.80, stdev=25.54 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 103], 80.00th=[ 112], 90.00th=[ 125], 95.00th=[ 136], | 99.00th=[ 165], 99.50th=[ 186], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 168, max= 477, per=25.02%, avg=323.80, stdev=77.46, samples=30 iops : min= 42, max= 119, avg=80.53, stdev=19.37, samples=30 write: IOPS=83, BW=334KiB/s (342kB/s)(5180KiB/15507msec); 0 zone resets slat (usec): min=26, max=8290, avg=147.88, stdev=354.55 clat (msec): min=43, max=198, avg=95.48, stdev=23.33 lat (msec): min=43, max=199, avg=95.63, stdev=23.34 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 100], | 70.00th=[ 106], 80.00th=[ 114], 90.00th=[ 127], 95.00th=[ 136], | 99.00th=[ 163], 99.50th=[ 186], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 198, max= 471, per=24.63%, avg=331.83, stdev=68.99, samples=30 iops : min= 49, max= 117, avg=82.53, stdev=17.27, samples=30 lat (msec) : 20=0.04%, 50=0.86%, 100=63.59%, 250=35.51% cpu : usr=0.39%, sys=2.54%, ctx=2656, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1294KiB/s (1325kB/s), 321KiB/s-328KiB/s (329kB/s-336kB/s), io=19.6MiB (20.5MB), run=15400-15507msec WRITE: bw=1347KiB/s (1380kB/s), 334KiB/s-340KiB/s (342kB/s-348kB/s), io=20.4MiB (21.4MB), run=15400-15507msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149001: Fri Oct 24 09:50:42 2025 read: IOPS=3, BW=144MiB/s (151MB/s)(40.0MiB/278msec) slat (nsec): min=23469k, max=23469k, avg=23468875.00, stdev= 0.00 clat (nsec): min=228710k, max=228710k, avg=228710112.00, stdev= 0.00 lat (nsec): min=252183k, max=252183k, avg=252182677.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.33%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=40.0MiB (41.9MB), run=278-278msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck