== sanity test 398c: run fio to test AIO ================= 19:20:29 (1786836029) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.259202 s, 162 MB/s osc.lustre-OST0000-osc-ffff9cbca049e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181448: Sat Aug 15 19:20:49 2026 write: IOPS=184, BW=736KiB/s (754kB/s)(10.0MiB/13906msec); 0 zone resets slat (usec): min=38, max=3814, avg=241.09, stdev=302.06 clat (msec): min=15, max=158, avg=86.34, stdev=14.51 lat (msec): min=15, max=158, avg=86.59, stdev=14.52 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 83], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 97], 90.00th=[ 105], 95.00th=[ 111], | 99.00th=[ 125], 99.50th=[ 129], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 606, max= 888, per=25.07%, avg=735.93, stdev=70.87, samples=27 iops : min= 151, max= 222, avg=183.78, stdev=17.82, samples=27 lat (msec) : 20=0.16%, 50=0.74%, 100=82.77%, 250=16.33% cpu : usr=0.26%, sys=2.60%, ctx=2721, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181449: Sat Aug 15 19:20:49 2026 write: IOPS=184, BW=738KiB/s (756kB/s)(10.0MiB/13871msec); 0 zone resets slat (usec): min=37, max=4445, avg=262.08, stdev=303.24 clat (msec): min=8, max=176, avg=86.29, stdev=14.90 lat (msec): min=8, max=176, avg=86.56, stdev=14.89 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 83], 50.00th=[ 86], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 97], 90.00th=[ 106], 95.00th=[ 111], | 99.00th=[ 125], 99.50th=[ 134], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 178] bw ( KiB/s): min= 582, max= 929, per=25.12%, avg=737.44, stdev=74.18, samples=27 iops : min= 145, max= 232, avg=184.15, stdev=18.64, samples=27 lat (msec) : 10=0.04%, 20=0.12%, 50=0.62%, 100=82.11%, 250=17.11% cpu : usr=0.43%, sys=2.83%, ctx=2793, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181450: Sat Aug 15 19:20:49 2026 write: IOPS=183, BW=735KiB/s (752kB/s)(10.0MiB/13939msec); 0 zone resets slat (usec): min=39, max=4422, avg=259.09, stdev=307.40 clat (msec): min=14, max=150, avg=86.47, stdev=14.30 lat (msec): min=15, max=150, avg=86.73, stdev=14.31 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 70], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 99], 90.00th=[ 107], 95.00th=[ 111], | 99.00th=[ 125], 99.50th=[ 132], 99.90th=[ 142], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 598, max= 816, per=24.99%, avg=733.74, stdev=60.61, samples=27 iops : min= 149, max= 204, avg=183.22, stdev=15.26, samples=27 lat (msec) : 20=0.04%, 50=0.20%, 100=82.89%, 250=16.88% cpu : usr=0.28%, sys=2.86%, ctx=2791, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181451: Sat Aug 15 19:20:49 2026 write: IOPS=183, BW=734KiB/s (752kB/s)(10.0MiB/13949msec); 0 zone resets slat (usec): min=38, max=4773, avg=241.36, stdev=284.37 clat (msec): min=15, max=139, avg=86.54, stdev=13.79 lat (msec): min=15, max=139, avg=86.78, stdev=13.79 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 77], | 30.00th=[ 80], 40.00th=[ 82], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 99], 90.00th=[ 106], 95.00th=[ 110], | 99.00th=[ 124], 99.50th=[ 130], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 614, max= 860, per=24.93%, avg=732.00, stdev=70.67, samples=27 iops : min= 153, max= 215, avg=182.78, stdev=17.79, samples=27 lat (msec) : 20=0.04%, 50=0.08%, 100=82.54%, 250=17.34% cpu : usr=0.29%, sys=2.52%, ctx=2764, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2936KiB/s (3007kB/s), 734KiB/s-738KiB/s (752kB/s-756kB/s), io=40.0MiB (41.9MB), run=13871-13949msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181462: Sat Aug 15 19:21:06 2026 read: IOPS=93, BW=373KiB/s (382kB/s)(5048KiB/13543msec) slat (usec): min=52, max=16550, avg=5695.78, stdev=2268.49 clat (msec): min=6, max=145, avg=80.30, stdev=16.91 lat (msec): min=12, max=153, avg=86.00, stdev=17.54 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 111], | 99.00th=[ 126], 99.50th=[ 132], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 270, max= 472, per=25.22%, avg=372.46, stdev=58.71, samples=26 iops : min= 67, max= 118, avg=92.77, stdev=14.76, samples=26 write: IOPS=95, BW=383KiB/s (393kB/s)(5192KiB/13543msec); 0 zone resets slat (usec): min=46, max=4006, avg=282.31, stdev=356.66 clat (msec): min=31, max=138, avg=82.79, stdev=15.65 lat (msec): min=31, max=138, avg=83.07, stdev=15.67 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 104], 95.00th=[ 112], | 99.00th=[ 126], 99.50th=[ 129], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 293, max= 542, per=24.81%, avg=381.65, stdev=59.03, samples=26 iops : min= 73, max= 135, avg=95.08, stdev=14.83, samples=26 lat (msec) : 10=0.04%, 20=0.04%, 50=1.45%, 100=86.17%, 250=12.30% cpu : usr=0.37%, sys=3.82%, ctx=4738, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181463: Sat Aug 15 19:21:06 2026 read: IOPS=91, BW=367KiB/s (376kB/s)(4976KiB/13566msec) slat (usec): min=1833, max=20224, avg=5717.82, stdev=2305.56 clat (msec): min=20, max=134, avg=79.62, stdev=15.59 lat (msec): min=27, max=139, avg=85.34, stdev=16.19 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 101], 95.00th=[ 108], | 99.00th=[ 120], 99.50th=[ 127], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 288, max= 488, per=24.68%, avg=364.58, stdev=56.83, samples=26 iops : min= 72, max= 122, avg=90.81, stdev=14.27, samples=26 write: IOPS=97, BW=388KiB/s (397kB/s)(5264KiB/13566msec); 0 zone resets slat (usec): min=56, max=3181, avg=275.43, stdev=290.81 clat (msec): min=44, max=146, avg=83.06, stdev=15.38 lat (msec): min=44, max=146, avg=83.34, stdev=15.37 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 91], 80.00th=[ 96], 90.00th=[ 104], 95.00th=[ 110], | 99.00th=[ 127], 99.50th=[ 133], 99.90th=[ 140], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 270, max= 520, per=25.26%, avg=388.42, stdev=63.09, samples=26 iops : min= 67, max= 130, avg=96.81, stdev=15.85, samples=26 lat (msec) : 50=0.82%, 100=87.73%, 250=11.45% cpu : usr=0.46%, sys=3.64%, ctx=4748, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181464: Sat Aug 15 19:21:06 2026 read: IOPS=91, BW=367KiB/s (376kB/s)(4984KiB/13583msec) slat (usec): min=1749, max=20235, avg=5694.92, stdev=2215.57 clat (msec): min=12, max=139, avg=80.10, stdev=16.07 lat (msec): min=17, max=147, avg=85.80, stdev=16.68 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 102], 95.00th=[ 108], | 99.00th=[ 124], 99.50th=[ 129], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 261, max= 479, per=24.92%, avg=368.08, stdev=66.20, samples=26 iops : min= 65, max= 119, avg=91.69, stdev=16.60, samples=26 write: IOPS=96, BW=387KiB/s (396kB/s)(5256KiB/13583msec); 0 zone resets slat (usec): min=61, max=5774, avg=286.28, stdev=355.94 clat (msec): min=45, max=141, avg=82.55, stdev=15.59 lat (msec): min=45, max=142, avg=82.84, stdev=15.61 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 96], 90.00th=[ 105], 95.00th=[ 111], | 99.00th=[ 123], 99.50th=[ 127], 99.90th=[ 136], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 287, max= 480, per=24.89%, avg=382.77, stdev=54.30, samples=26 iops : min= 71, max= 120, avg=95.35, stdev=13.65, samples=26 lat (msec) : 20=0.04%, 50=0.86%, 100=86.76%, 250=12.34% cpu : usr=0.32%, sys=3.81%, ctx=4799, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181465: Sat Aug 15 19:21:06 2026 read: IOPS=93, BW=373KiB/s (381kB/s)(5060KiB/13583msec) slat (usec): min=2126, max=15592, avg=5669.42, stdev=2174.70 clat (msec): min=11, max=141, avg=80.20, stdev=16.94 lat (msec): min=17, max=148, avg=85.87, stdev=17.51 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 94], 90.00th=[ 102], 95.00th=[ 111], | 99.00th=[ 129], 99.50th=[ 132], 99.90th=[ 140], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 287, max= 488, per=25.16%, avg=371.58, stdev=60.69, samples=26 iops : min= 71, max= 122, avg=92.54, stdev=15.26, samples=26 write: IOPS=95, BW=381KiB/s (391kB/s)(5180KiB/13583msec); 0 zone resets slat (usec): min=56, max=8146, avg=267.05, stdev=343.13 clat (msec): min=15, max=138, avg=82.78, stdev=14.86 lat (msec): min=15, max=138, avg=83.05, stdev=14.89 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 122], 99.50th=[ 128], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 302, max= 526, per=24.68%, avg=379.65, stdev=55.66, samples=26 iops : min= 75, max= 131, avg=94.58, stdev=14.02, samples=26 lat (msec) : 20=0.08%, 50=0.90%, 100=87.07%, 250=11.95% cpu : usr=0.39%, sys=3.84%, ctx=4750, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1477KiB/s (1513kB/s), 367KiB/s-373KiB/s (376kB/s-382kB/s), io=19.6MiB (20.5MB), run=13543-13583msec WRITE: bw=1538KiB/s (1575kB/s), 381KiB/s-388KiB/s (391kB/s-397kB/s), io=20.4MiB (21.4MB), run=13543-13583msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181471: Sat Aug 15 19:21:10 2026 read: IOPS=5, BW=223MiB/s (234MB/s)(40.0MiB/179msec) slat (nsec): min=16819k, max=16819k, avg=16819297.00, stdev= 0.00 clat (nsec): min=151606k, max=151606k, avg=151605533.00, stdev= 0.00 lat (nsec): min=168428k, max=168428k, avg=168428353.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.74%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=223MiB/s (234MB/s), 223MiB/s-223MiB/s (234MB/s-234MB/s), io=40.0MiB (41.9MB), run=179-179msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout