== sanity test 398c: run fio to test AIO ================= 20:56:45 (1768010205) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.143412 s, 292 MB/s osc.lustre-OST0000-osc-ffff934fc8520000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=152010: Fri Jan 9 20:57:23 2026 write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34440msec); 0 zone resets slat (usec): min=26, max=7645, avg=141.19, stdev=166.22 clat (msec): min=59, max=330, avg=214.91, stdev=43.89 lat (msec): min=59, max=330, avg=215.06, stdev=43.90 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 249], 90.00th=[ 268], 95.00th=[ 279], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 200, max= 616, per=24.88%, avg=295.76, stdev=55.68, samples=68 iops : min= 50, max= 154, avg=73.94, stdev=13.92, samples=68 lat (msec) : 100=3.16%, 250=77.23%, 500=19.61% cpu : usr=0.12%, sys=0.96%, ctx=2815, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152011: Fri Jan 9 20:57:23 2026 write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34369msec); 0 zone resets slat (usec): min=24, max=1339, avg=133.38, stdev=73.70 clat (msec): min=31, max=375, avg=214.54, stdev=46.12 lat (msec): min=31, max=375, avg=214.68, stdev=46.13 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 138], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 279], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 176, max= 719, per=24.95%, avg=296.69, stdev=66.85, samples=68 iops : min= 44, max= 179, avg=74.16, stdev=16.64, samples=68 lat (msec) : 50=0.62%, 100=3.16%, 250=75.66%, 500=20.55% cpu : usr=0.15%, sys=0.88%, ctx=2686, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152012: Fri Jan 9 20:57:23 2026 write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34439msec); 0 zone resets slat (usec): min=24, max=692, avg=136.35, stdev=66.15 clat (msec): min=58, max=338, avg=214.90, stdev=44.20 lat (msec): min=58, max=338, avg=215.04, stdev=44.21 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 268], 95.00th=[ 279], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 334], | 99.99th=[ 338] bw ( KiB/s): min= 240, max= 616, per=24.88%, avg=295.88, stdev=54.30, samples=68 iops : min= 60, max= 154, avg=73.97, stdev=13.58, samples=68 lat (msec) : 100=2.85%, 250=76.99%, 500=20.16% cpu : usr=0.15%, sys=0.92%, ctx=2829, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152013: Fri Jan 9 20:57:23 2026 write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34417msec); 0 zone resets slat (usec): min=27, max=1533, avg=143.14, stdev=74.98 clat (msec): min=27, max=389, avg=214.78, stdev=45.41 lat (msec): min=27, max=389, avg=214.92, stdev=45.41 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 284], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 388] bw ( KiB/s): min= 192, max= 642, per=24.93%, avg=296.38, stdev=59.66, samples=68 iops : min= 48, max= 160, avg=74.09, stdev=14.87, samples=68 lat (msec) : 50=0.31%, 100=2.93%, 250=76.13%, 500=20.62% cpu : usr=0.15%, sys=0.96%, ctx=2899, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1189KiB/s (1218kB/s), 297KiB/s-298KiB/s (304kB/s-305kB/s), io=40.0MiB (41.9MB), run=34369-34440msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=152028: Fri Jan 9 20:57:45 2026 read: IOPS=65, BW=264KiB/s (270kB/s)(5048KiB/19135msec) slat (usec): min=943, max=8185, avg=2206.41, stdev=725.32 clat (msec): min=4, max=159, avg=76.23, stdev=14.26 lat (msec): min=6, max=162, avg=78.44, stdev=14.30 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 67], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 93], 95.00th=[ 99], | 99.00th=[ 111], 99.50th=[ 115], 99.90th=[ 146], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 136, max= 360, per=25.24%, avg=264.22, stdev=49.30, samples=37 iops : min= 34, max= 90, avg=66.05, stdev=12.32, samples=37 write: IOPS=67, BW=271KiB/s (278kB/s)(5192KiB/19135msec); 0 zone resets slat (usec): min=28, max=1489, avg=104.47, stdev=65.45 clat (msec): min=30, max=273, avg=159.08, stdev=33.98 lat (msec): min=30, max=273, avg=159.19, stdev=33.99 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 104], 10.00th=[ 113], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 203], 95.00th=[ 215], | 99.00th=[ 239], 99.50th=[ 245], 99.90th=[ 262], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 192, max= 424, per=24.86%, avg=271.03, stdev=37.42, samples=37 iops : min= 48, max= 106, avg=67.76, stdev= 9.36, samples=37 lat (msec) : 10=0.16%, 20=0.27%, 50=0.74%, 100=48.28%, 250=50.47% lat (msec) : 500=0.08% cpu : usr=0.23%, sys=1.65%, ctx=4683, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152029: Fri Jan 9 20:57:45 2026 read: IOPS=64, BW=260KiB/s (266kB/s)(4976KiB/19152msec) slat (usec): min=891, max=78324, avg=2261.03, stdev=2276.71 clat (msec): min=3, max=145, avg=76.15, stdev=13.51 lat (msec): min=5, max=146, avg=78.42, stdev=13.40 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 67], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 92], 95.00th=[ 99], | 99.00th=[ 108], 99.50th=[ 114], 99.90th=[ 144], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 160, max= 392, per=24.67%, avg=258.32, stdev=54.51, samples=38 iops : min= 40, max= 98, avg=64.58, stdev=13.63, samples=38 write: IOPS=68, BW=275KiB/s (281kB/s)(5264KiB/19152msec); 0 zone resets slat (usec): min=22, max=440, avg=101.85, stdev=50.63 clat (msec): min=36, max=273, avg=158.28, stdev=35.01 lat (msec): min=36, max=273, avg=158.38, stdev=35.01 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 203], 95.00th=[ 215], | 99.00th=[ 236], 99.50th=[ 243], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 208, max= 360, per=25.11%, avg=273.68, stdev=37.11, samples=38 iops : min= 52, max= 90, avg=68.42, stdev= 9.28, samples=38 lat (msec) : 4=0.08%, 10=0.08%, 20=0.12%, 50=1.09%, 100=48.36% lat (msec) : 250=50.12%, 500=0.16% cpu : usr=0.22%, sys=1.66%, ctx=4546, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152030: Fri Jan 9 20:57:45 2026 read: IOPS=65, BW=260KiB/s (266kB/s)(4984KiB/19152msec) slat (usec): min=34, max=6010, avg=2215.00, stdev=729.08 clat (msec): min=2, max=128, avg=76.36, stdev=13.24 lat (msec): min=5, max=131, avg=78.58, stdev=13.32 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 87], 90.00th=[ 92], 95.00th=[ 99], | 99.00th=[ 107], 99.50th=[ 112], 99.90th=[ 115], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 144, max= 376, per=24.84%, avg=260.03, stdev=56.26, samples=38 iops : min= 36, max= 94, avg=65.00, stdev=14.06, samples=38 write: IOPS=68, BW=274KiB/s (281kB/s)(5256KiB/19152msec); 0 zone resets slat (usec): min=24, max=432, avg=104.52, stdev=52.98 clat (msec): min=30, max=280, avg=158.42, stdev=35.25 lat (msec): min=30, max=280, avg=158.52, stdev=35.26 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 101], 10.00th=[ 111], 20.00th=[ 130], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 203], 95.00th=[ 215], | 99.00th=[ 239], 99.50th=[ 245], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 192, max= 368, per=24.95%, avg=272.00, stdev=34.30, samples=38 iops : min= 48, max= 92, avg=68.00, stdev= 8.57, samples=38 lat (msec) : 4=0.08%, 10=0.04%, 20=0.12%, 50=1.05%, 100=48.20% lat (msec) : 250=50.31%, 500=0.20% cpu : usr=0.21%, sys=1.66%, ctx=4561, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152031: Fri Jan 9 20:57:45 2026 read: IOPS=66, BW=265KiB/s (271kB/s)(5060KiB/19103msec) slat (usec): min=917, max=6343, avg=2197.69, stdev=716.67 clat (msec): min=3, max=181, avg=75.57, stdev=14.60 lat (msec): min=6, max=183, avg=77.77, stdev=14.62 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 91], 95.00th=[ 96], | 99.00th=[ 110], 99.50th=[ 114], 99.90th=[ 180], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 128, max= 400, per=25.53%, avg=267.32, stdev=58.77, samples=37 iops : min= 32, max= 100, avg=66.81, stdev=14.70, samples=37 write: IOPS=67, BW=271KiB/s (278kB/s)(5180KiB/19103msec); 0 zone resets slat (usec): min=33, max=428, avg=104.72, stdev=54.56 clat (msec): min=23, max=295, avg=159.42, stdev=34.92 lat (msec): min=23, max=296, avg=159.53, stdev=34.92 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 102], 10.00th=[ 113], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 203], 95.00th=[ 218], | 99.00th=[ 259], 99.50th=[ 275], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 216, max= 328, per=24.90%, avg=271.46, stdev=25.16, samples=37 iops : min= 54, max= 82, avg=67.86, stdev= 6.29, samples=37 lat (msec) : 4=0.04%, 10=0.12%, 20=0.39%, 50=0.70%, 100=48.87% lat (msec) : 250=49.30%, 500=0.59% cpu : usr=0.18%, sys=1.71%, ctx=4614, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1048KiB/s (1073kB/s), 260KiB/s-265KiB/s (266kB/s-271kB/s), io=19.6MiB (20.5MB), run=19103-19152msec WRITE: bw=1091KiB/s (1117kB/s), 271KiB/s-275KiB/s (278kB/s-281kB/s), io=20.4MiB (21.4MB), run=19103-19152msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=152037: Fri Jan 9 20:57:49 2026 read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec) slat (nsec): min=14182k, max=14182k, avg=14182044.00, stdev= 0.00 clat (nsec): min=81206k, max=81206k, avg=81206474.00, stdev= 0.00 lat (nsec): min=95394k, max=95394k, avg=95394010.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[81265], 5.00th=[81265], 10.00th=[81265], 20.00th=[81265], | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265], | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=12.12%, ctx=40, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout