== sanity test 398c: run fio to test AIO ================= 22:36:11 (1788921371) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.303703 s, 138 MB/s osc.lustre-OST0000-osc-ffff9c4a05599800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183457: Tue Sep 8 22:36:33 2026 write: IOPS=152, BW=610KiB/s (625kB/s)(10.0MiB/16782msec); 0 zone resets slat (usec): min=24, max=5955, avg=197.38, stdev=395.29 clat (msec): min=19, max=200, avg=104.57, stdev=21.89 lat (msec): min=19, max=200, avg=104.77, stdev=21.89 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 75], 10.00th=[ 81], 20.00th=[ 88], | 30.00th=[ 93], 40.00th=[ 97], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 115], 80.00th=[ 124], 90.00th=[ 132], 95.00th=[ 140], | 99.00th=[ 161], 99.50th=[ 174], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 447, max= 891, per=25.17%, avg=608.55, stdev=91.23, samples=33 iops : min= 111, max= 222, avg=151.76, stdev=22.77, samples=33 lat (msec) : 20=0.12%, 50=0.94%, 100=42.66%, 250=56.29% cpu : usr=0.28%, sys=1.63%, ctx=2995, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183458: Tue Sep 8 22:36:33 2026 write: IOPS=151, BW=605KiB/s (620kB/s)(10.0MiB/16919msec); 0 zone resets slat (usec): min=24, max=6916, avg=174.97, stdev=354.30 clat (msec): min=18, max=188, avg=105.26, stdev=20.94 lat (msec): min=18, max=188, avg=105.43, stdev=20.95 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 75], 10.00th=[ 82], 20.00th=[ 89], | 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 105], 60.00th=[ 110], | 70.00th=[ 116], 80.00th=[ 124], 90.00th=[ 132], 95.00th=[ 142], | 99.00th=[ 157], 99.50th=[ 171], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 432, max= 780, per=24.99%, avg=604.21, stdev=80.69, samples=33 iops : min= 108, max= 195, avg=150.70, stdev=20.19, samples=33 lat (msec) : 20=0.04%, 50=0.31%, 100=42.62%, 250=57.03% cpu : usr=0.22%, sys=1.64%, ctx=2971, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183459: Tue Sep 8 22:36:33 2026 write: IOPS=151, BW=605KiB/s (619kB/s)(10.0MiB/16928msec); 0 zone resets slat (usec): min=21, max=5618, avg=182.82, stdev=375.61 clat (msec): min=9, max=172, avg=105.09, stdev=20.31 lat (msec): min=10, max=172, avg=105.27, stdev=20.31 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 75], 10.00th=[ 82], 20.00th=[ 88], | 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 105], 60.00th=[ 110], | 70.00th=[ 116], 80.00th=[ 124], 90.00th=[ 131], 95.00th=[ 140], | 99.00th=[ 155], 99.50th=[ 161], 99.90th=[ 171], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 448, max= 766, per=24.95%, avg=603.33, stdev=72.46, samples=33 iops : min= 112, max= 191, avg=150.48, stdev=18.11, samples=33 lat (msec) : 10=0.04%, 20=0.12%, 100=43.16%, 250=56.68% cpu : usr=0.25%, sys=1.58%, ctx=2646, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183460: Tue Sep 8 22:36:33 2026 write: IOPS=151, BW=605KiB/s (619kB/s)(10.0MiB/16936msec); 0 zone resets slat (usec): min=25, max=8157, avg=196.06, stdev=445.32 clat (msec): min=20, max=176, avg=105.23, stdev=20.32 lat (msec): min=20, max=177, avg=105.43, stdev=20.33 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 77], 10.00th=[ 83], 20.00th=[ 89], | 30.00th=[ 94], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 110], | 70.00th=[ 115], 80.00th=[ 124], 90.00th=[ 130], 95.00th=[ 140], | 99.00th=[ 155], 99.50th=[ 169], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 178] bw ( KiB/s): min= 447, max= 766, per=24.91%, avg=602.27, stdev=74.00, samples=33 iops : min= 111, max= 191, avg=150.21, stdev=18.54, samples=33 lat (msec) : 50=0.66%, 100=40.94%, 250=58.40% cpu : usr=0.32%, sys=1.53%, ctx=3061, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2419KiB/s (2477kB/s), 605KiB/s-610KiB/s (619kB/s-625kB/s), io=40.0MiB (41.9MB), run=16782-16936msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183472: Tue Sep 8 22:36:53 2026 read: IOPS=84, BW=338KiB/s (346kB/s)(5048KiB/14924msec) slat (usec): min=1932, max=23717, avg=6055.25, stdev=2541.46 clat (msec): min=16, max=155, avg=87.90, stdev=18.85 lat (msec): min=21, max=166, avg=93.96, stdev=19.63 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 77], 40.00th=[ 82], 50.00th=[ 87], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 104], 90.00th=[ 113], 95.00th=[ 122], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 153], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 215, max= 448, per=25.09%, avg=337.28, stdev=59.44, samples=29 iops : min= 53, max= 112, avg=84.17, stdev=14.92, samples=29 write: IOPS=86, BW=348KiB/s (356kB/s)(5192KiB/14924msec); 0 zone resets slat (usec): min=28, max=3980, avg=178.49, stdev=251.63 clat (msec): min=38, max=169, avg=91.34, stdev=18.03 lat (msec): min=38, max=169, avg=91.53, stdev=18.02 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 106], 90.00th=[ 115], 95.00th=[ 123], | 99.00th=[ 144], 99.50th=[ 153], 99.90th=[ 155], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 239, max= 488, per=24.90%, avg=348.34, stdev=61.89, samples=29 iops : min= 59, max= 122, avg=86.93, stdev=15.48, samples=29 lat (msec) : 20=0.04%, 50=0.39%, 100=72.46%, 250=27.11% cpu : usr=0.43%, sys=2.79%, ctx=4794, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183473: Tue Sep 8 22:36:53 2026 read: IOPS=83, BW=334KiB/s (342kB/s)(4976KiB/14889msec) slat (usec): min=1742, max=20217, avg=6069.61, stdev=2403.13 clat (msec): min=5, max=168, avg=88.24, stdev=19.22 lat (msec): min=11, max=175, avg=94.32, stdev=19.88 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 93], | 70.00th=[ 99], 80.00th=[ 104], 90.00th=[ 112], 95.00th=[ 122], | 99.00th=[ 138], 99.50th=[ 146], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 207, max= 456, per=24.88%, avg=334.38, stdev=59.81, samples=29 iops : min= 51, max= 114, avg=83.45, stdev=15.02, samples=29 write: IOPS=88, BW=354KiB/s (362kB/s)(5264KiB/14889msec); 0 zone resets slat (usec): min=34, max=8505, avg=203.76, stdev=402.18 clat (msec): min=27, max=161, avg=90.79, stdev=17.64 lat (msec): min=27, max=162, avg=90.99, stdev=17.67 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 77], | 30.00th=[ 81], 40.00th=[ 85], 50.00th=[ 89], 60.00th=[ 93], | 70.00th=[ 99], 80.00th=[ 105], 90.00th=[ 115], 95.00th=[ 123], | 99.00th=[ 140], 99.50th=[ 146], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 208, max= 512, per=25.28%, avg=353.72, stdev=67.78, samples=29 iops : min= 52, max= 128, avg=88.28, stdev=16.95, samples=29 lat (msec) : 10=0.04%, 20=0.08%, 50=0.39%, 100=73.09%, 250=26.41% cpu : usr=0.42%, sys=2.84%, ctx=4859, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183474: Tue Sep 8 22:36:53 2026 read: IOPS=83, BW=336KiB/s (344kB/s)(4984KiB/14852msec) slat (usec): min=31, max=20840, avg=5987.46, stdev=2491.26 clat (msec): min=12, max=181, avg=87.99, stdev=19.36 lat (msec): min=12, max=189, avg=93.98, stdev=20.17 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 91], | 70.00th=[ 96], 80.00th=[ 104], 90.00th=[ 114], 95.00th=[ 124], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 171], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 200, max= 472, per=25.11%, avg=337.48, stdev=70.66, samples=29 iops : min= 50, max= 118, avg=84.21, stdev=17.69, samples=29 write: IOPS=88, BW=354KiB/s (362kB/s)(5256KiB/14852msec); 0 zone resets slat (usec): min=33, max=6234, avg=184.30, stdev=357.22 clat (msec): min=28, max=168, avg=91.23, stdev=18.19 lat (msec): min=28, max=168, avg=91.42, stdev=18.22 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 85], 50.00th=[ 89], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 106], 90.00th=[ 116], 95.00th=[ 126], | 99.00th=[ 140], 99.50th=[ 153], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 264, max= 472, per=25.31%, avg=354.14, stdev=50.73, samples=29 iops : min= 66, max= 118, avg=88.38, stdev=12.72, samples=29 lat (msec) : 20=0.12%, 50=0.51%, 100=73.24%, 250=26.13% cpu : usr=0.36%, sys=2.83%, ctx=4759, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183475: Tue Sep 8 22:36:53 2026 read: IOPS=84, BW=339KiB/s (347kB/s)(5060KiB/14913msec) slat (usec): min=1830, max=18504, avg=6043.46, stdev=2456.17 clat (msec): min=45, max=165, avg=87.65, stdev=19.17 lat (msec): min=50, max=173, avg=93.70, stdev=19.90 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 60], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 82], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 96], 80.00th=[ 105], 90.00th=[ 114], 95.00th=[ 123], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 165], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 223, max= 480, per=25.32%, avg=340.28, stdev=64.89, samples=29 iops : min= 55, max= 120, avg=84.93, stdev=16.24, samples=29 write: IOPS=86, BW=347KiB/s (356kB/s)(5180KiB/14913msec); 0 zone resets slat (usec): min=28, max=7200, avg=184.37, stdev=338.87 clat (msec): min=47, max=165, avg=91.35, stdev=17.94 lat (msec): min=47, max=165, avg=91.54, stdev=17.96 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 89], 60.00th=[ 93], | 70.00th=[ 100], 80.00th=[ 106], 90.00th=[ 116], 95.00th=[ 125], | 99.00th=[ 142], 99.50th=[ 150], 99.90th=[ 165], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 192, max= 464, per=24.80%, avg=346.90, stdev=62.73, samples=29 iops : min= 48, max= 116, avg=86.55, stdev=15.76, samples=29 lat (msec) : 50=0.31%, 100=73.63%, 250=26.05% cpu : usr=0.38%, sys=2.88%, ctx=4895, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1345KiB/s (1377kB/s), 334KiB/s-339KiB/s (342kB/s-347kB/s), io=19.6MiB (20.5MB), run=14852-14924msec WRITE: bw=1400KiB/s (1433kB/s), 347KiB/s-354KiB/s (356kB/s-362kB/s), io=20.4MiB (21.4MB), run=14852-14924msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183481: Tue Sep 8 22:36:58 2026 read: IOPS=4, BW=188MiB/s (197MB/s)(40.0MiB/213msec) slat (nsec): min=13762k, max=13762k, avg=13762231.00, stdev= 0.00 clat (nsec): min=187964k, max=187964k, avg=187964131.00, stdev= 0.00 lat (nsec): min=201732k, max=201732k, avg=201731654.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.19%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=188MiB/s (197MB/s), 188MiB/s-188MiB/s (197MB/s-197MB/s), io=40.0MiB (41.9MB), run=213-213msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout