== sanity test 398c: run fio to test AIO ================= 11:30:41 (1788363041) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.268335 s, 156 MB/s osc.lustre-OST0000-osc-ffff9d4c12753800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=185797: Wed Sep 2 11:31:00 2026 write: IOPS=184, BW=739KiB/s (756kB/s)(10.0MiB/13862msec); 0 zone resets slat (usec): min=36, max=4902, avg=219.84, stdev=320.57 clat (msec): min=47, max=165, avg=85.86, stdev=16.06 lat (msec): min=47, max=165, avg=86.08, stdev=16.08 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 94], 80.00th=[ 101], 90.00th=[ 108], 95.00th=[ 114], | 99.00th=[ 127], 99.50th=[ 131], 99.90th=[ 150], 99.95th=[ 161], | 99.99th=[ 165] bw ( KiB/s): min= 534, max= 1016, per=25.01%, avg=738.93, stdev=92.70, samples=27 iops : min= 133, max= 254, avg=184.56, stdev=23.22, samples=27 lat (msec) : 50=0.31%, 100=79.45%, 250=20.23% cpu : usr=0.25%, sys=2.48%, ctx=2878, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185798: Wed Sep 2 11:31:00 2026 write: IOPS=184, BW=740KiB/s (758kB/s)(10.0MiB/13842msec); 0 zone resets slat (usec): min=42, max=42520, avg=240.31, stdev=883.64 clat (msec): min=10, max=164, avg=85.75, stdev=16.62 lat (msec): min=10, max=164, avg=85.99, stdev=16.60 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 63], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 100], 90.00th=[ 108], 95.00th=[ 115], | 99.00th=[ 131], 99.50th=[ 132], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 165] bw ( KiB/s): min= 558, max= 1016, per=25.09%, avg=741.19, stdev=90.14, samples=27 iops : min= 139, max= 254, avg=185.11, stdev=22.57, samples=27 lat (msec) : 20=0.16%, 50=0.23%, 100=80.70%, 250=18.91% cpu : usr=0.32%, sys=2.59%, ctx=2850, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185799: Wed Sep 2 11:31:00 2026 write: IOPS=184, BW=739KiB/s (756kB/s)(10.0MiB/13862msec); 0 zone resets slat (usec): min=37, max=5637, avg=207.98, stdev=293.31 clat (msec): min=15, max=140, avg=85.82, stdev=16.22 lat (msec): min=15, max=140, avg=86.03, stdev=16.24 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 93], 80.00th=[ 99], 90.00th=[ 108], 95.00th=[ 116], | 99.00th=[ 130], 99.50th=[ 132], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 140] bw ( KiB/s): min= 614, max= 912, per=24.97%, avg=737.67, stdev=79.95, samples=27 iops : min= 153, max= 228, avg=184.22, stdev=20.08, samples=27 lat (msec) : 20=0.08%, 50=0.23%, 100=81.45%, 250=18.24% cpu : usr=0.30%, sys=2.15%, ctx=2782, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185800: Wed Sep 2 11:31:00 2026 write: IOPS=185, BW=744KiB/s (762kB/s)(10.0MiB/13764msec); 0 zone resets slat (usec): min=38, max=5206, avg=232.58, stdev=319.33 clat (msec): min=30, max=176, avg=85.69, stdev=16.43 lat (msec): min=30, max=176, avg=85.92, stdev=16.43 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 108], 95.00th=[ 114], | 99.00th=[ 132], 99.50th=[ 133], 99.90th=[ 167], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 566, max= 976, per=25.16%, avg=743.37, stdev=86.62, samples=27 iops : min= 141, max= 244, avg=185.63, stdev=21.73, samples=27 lat (msec) : 50=0.62%, 100=81.09%, 250=18.28% cpu : usr=0.31%, sys=2.58%, ctx=2904, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2955KiB/s (3026kB/s), 739KiB/s-744KiB/s (756kB/s-762kB/s), io=40.0MiB (41.9MB), run=13764-13862msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=185811: Wed Sep 2 11:31:19 2026 read: IOPS=84, BW=339KiB/s (347kB/s)(5048KiB/14877msec) slat (usec): min=76, max=23648, avg=6122.12, stdev=2692.97 clat (msec): min=12, max=205, avg=88.51, stdev=22.25 lat (msec): min=15, max=212, avg=94.63, stdev=22.97 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 81], 50.00th=[ 87], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 106], 90.00th=[ 116], 95.00th=[ 128], | 99.00th=[ 155], 99.50th=[ 165], 99.90th=[ 201], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 247, max= 456, per=25.37%, avg=339.21, stdev=54.11, samples=29 iops : min= 61, max= 114, avg=84.45, stdev=13.58, samples=29 write: IOPS=87, BW=349KiB/s (357kB/s)(5192KiB/14877msec); 0 zone resets slat (usec): min=54, max=7866, avg=278.12, stdev=452.45 clat (msec): min=21, max=165, avg=90.81, stdev=19.71 lat (msec): min=21, max=166, avg=91.09, stdev=19.71 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 74], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 107], 90.00th=[ 117], 95.00th=[ 126], | 99.00th=[ 153], 99.50th=[ 159], 99.90th=[ 165], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 253, max= 512, per=25.12%, avg=349.72, stdev=69.80, samples=29 iops : min= 63, max= 128, avg=87.07, stdev=17.53, samples=29 lat (msec) : 20=0.16%, 50=0.39%, 100=71.45%, 250=28.01% cpu : usr=0.31%, sys=3.03%, ctx=4793, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185812: Wed Sep 2 11:31:19 2026 read: IOPS=82, BW=332KiB/s (340kB/s)(4976KiB/15001msec) slat (usec): min=1654, max=25640, avg=6244.74, stdev=2777.95 clat (msec): min=18, max=180, avg=87.96, stdev=19.66 lat (msec): min=28, max=186, avg=94.21, stdev=20.47 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 60], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 91], | 70.00th=[ 97], 80.00th=[ 104], 90.00th=[ 114], 95.00th=[ 125], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 150], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 239, max= 464, per=24.81%, avg=331.72, stdev=61.41, samples=29 iops : min= 59, max= 116, avg=82.59, stdev=15.44, samples=29 write: IOPS=87, BW=351KiB/s (359kB/s)(5264KiB/15001msec); 0 zone resets slat (usec): min=39, max=4952, avg=249.22, stdev=341.03 clat (msec): min=48, max=162, avg=92.23, stdev=20.12 lat (msec): min=48, max=162, avg=92.49, stdev=20.12 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 102], 80.00th=[ 109], 90.00th=[ 121], 95.00th=[ 131], | 99.00th=[ 148], 99.50th=[ 157], 99.90th=[ 161], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 264, max= 464, per=25.26%, avg=351.55, stdev=60.94, samples=29 iops : min= 66, max= 116, avg=87.52, stdev=15.29, samples=29 lat (msec) : 20=0.04%, 50=0.31%, 100=71.17%, 250=28.48% cpu : usr=0.33%, sys=2.95%, ctx=4759, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185813: Wed Sep 2 11:31:19 2026 read: IOPS=83, BW=333KiB/s (341kB/s)(4984KiB/14949msec) slat (usec): min=1859, max=30741, avg=6202.28, stdev=2790.31 clat (msec): min=16, max=165, avg=87.94, stdev=21.00 lat (msec): min=24, max=185, avg=94.14, stdev=21.88 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 81], 50.00th=[ 86], 60.00th=[ 91], | 70.00th=[ 97], 80.00th=[ 105], 90.00th=[ 117], 95.00th=[ 130], | 99.00th=[ 144], 99.50th=[ 153], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 191, max= 488, per=25.09%, avg=335.52, stdev=73.69, samples=29 iops : min= 47, max= 122, avg=83.52, stdev=18.49, samples=29 write: IOPS=87, BW=352KiB/s (360kB/s)(5256KiB/14949msec); 0 zone resets slat (usec): min=53, max=9192, avg=255.99, stdev=498.60 clat (msec): min=30, max=174, avg=91.63, stdev=19.81 lat (msec): min=30, max=174, avg=91.89, stdev=19.85 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 74], | 30.00th=[ 79], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 108], 90.00th=[ 118], 95.00th=[ 128], | 99.00th=[ 150], 99.50th=[ 163], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 231, max= 472, per=25.25%, avg=351.48, stdev=59.83, samples=29 iops : min= 57, max= 118, avg=87.52, stdev=15.07, samples=29 lat (msec) : 20=0.04%, 50=0.27%, 100=71.48%, 250=28.20% cpu : usr=0.38%, sys=3.05%, ctx=4809, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185814: Wed Sep 2 11:31:19 2026 read: IOPS=84, BW=337KiB/s (345kB/s)(5060KiB/15001msec) slat (usec): min=1690, max=25069, avg=6236.24, stdev=2943.75 clat (msec): min=18, max=183, avg=88.69, stdev=20.79 lat (msec): min=28, max=191, avg=94.93, stdev=21.52 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 60], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 106], 90.00th=[ 117], 95.00th=[ 126], | 99.00th=[ 146], 99.50th=[ 159], 99.90th=[ 182], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 239, max= 464, per=25.19%, avg=336.76, stdev=54.16, samples=29 iops : min= 59, max= 116, avg=83.83, stdev=13.56, samples=29 write: IOPS=86, BW=345KiB/s (354kB/s)(5180KiB/15001msec); 0 zone resets slat (usec): min=57, max=4651, avg=242.07, stdev=323.31 clat (msec): min=23, max=166, avg=91.37, stdev=19.39 lat (msec): min=23, max=166, avg=91.61, stdev=19.42 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 118], 95.00th=[ 126], | 99.00th=[ 142], 99.50th=[ 148], 99.90th=[ 163], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 223, max= 496, per=24.65%, avg=343.10, stdev=67.77, samples=29 iops : min= 55, max= 124, avg=85.41, stdev=17.07, samples=29 lat (msec) : 20=0.04%, 50=0.51%, 100=71.05%, 250=28.40% cpu : usr=0.25%, sys=3.12%, ctx=4666, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1338KiB/s (1370kB/s), 332KiB/s-339KiB/s (340kB/s-347kB/s), io=19.6MiB (20.5MB), run=14877-15001msec WRITE: bw=1393KiB/s (1426kB/s), 345KiB/s-352KiB/s (354kB/s-360kB/s), io=20.4MiB (21.4MB), run=14877-15001msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=185820: Wed Sep 2 11:31:24 2026 read: IOPS=5, BW=201MiB/s (211MB/s)(40.0MiB/199msec) slat (nsec): min=19240k, max=19240k, avg=19240046.00, stdev= 0.00 clat (nsec): min=169240k, max=169240k, avg=169240142.00, stdev= 0.00 lat (nsec): min=188484k, max=188484k, avg=188484080.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.55%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=201MiB/s (211MB/s), 201MiB/s-201MiB/s (211MB/s-211MB/s), io=40.0MiB (41.9MB), run=199-199msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout