== sanity test 398c: run fio to test AIO ================= 06:27:48 (1783074468) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.134471 s, 312 MB/s osc.lustre-OST0000-osc-ffff98fad91dc800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147986: Fri Jul 3 06:28:28 2026 write: IOPS=69, BW=280KiB/s (286kB/s)(10.0MiB/36605msec); 0 zone resets slat (usec): min=38, max=3727, avg=228.30, stdev=144.06 clat (msec): min=68, max=400, avg=228.29, stdev=49.18 lat (msec): min=68, max=401, avg=228.52, stdev=49.18 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 159], 10.00th=[ 178], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 215], 50.00th=[ 228], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 167, max= 616, per=24.93%, avg=278.75, stdev=56.91, samples=72 iops : min= 41, max= 154, avg=69.60, stdev=14.29, samples=72 lat (msec) : 100=2.27%, 250=67.46%, 500=30.27% cpu : usr=0.12%, sys=1.34%, ctx=2677, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147987: Fri Jul 3 06:28:28 2026 write: IOPS=70, BW=280KiB/s (287kB/s)(10.0MiB/36554msec); 0 zone resets slat (usec): min=41, max=4116, avg=231.73, stdev=158.90 clat (msec): min=12, max=446, avg=228.00, stdev=50.77 lat (msec): min=12, max=446, avg=228.24, stdev=50.77 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 159], 10.00th=[ 178], 20.00th=[ 194], | 30.00th=[ 205], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 251], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 309], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 401], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 183, max= 629, per=24.98%, avg=279.26, stdev=58.99, samples=72 iops : min= 45, max= 157, avg=69.72, stdev=14.79, samples=72 lat (msec) : 20=0.04%, 50=0.35%, 100=1.91%, 250=68.01%, 500=29.69% cpu : usr=0.13%, sys=1.34%, ctx=2708, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147988: Fri Jul 3 06:28:28 2026 write: IOPS=69, BW=280KiB/s (287kB/s)(10.0MiB/36587msec); 0 zone resets slat (usec): min=35, max=2450, avg=213.42, stdev=147.76 clat (msec): min=35, max=464, avg=228.31, stdev=51.97 lat (msec): min=36, max=464, avg=228.53, stdev=51.98 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 146], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 380], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 464] bw ( KiB/s): min= 192, max= 616, per=24.97%, avg=279.19, stdev=56.73, samples=72 iops : min= 48, max= 154, avg=69.71, stdev=14.22, samples=72 lat (msec) : 50=0.62%, 100=1.25%, 250=66.60%, 500=31.52% cpu : usr=0.14%, sys=1.24%, ctx=2470, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147989: Fri Jul 3 06:28:28 2026 write: IOPS=69, BW=280KiB/s (286kB/s)(10.0MiB/36605msec); 0 zone resets slat (usec): min=55, max=3482, avg=232.22, stdev=154.28 clat (msec): min=62, max=416, avg=228.26, stdev=50.04 lat (msec): min=63, max=416, avg=228.50, stdev=50.05 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 159], 10.00th=[ 178], 20.00th=[ 194], | 30.00th=[ 205], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 249], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 393], 99.95th=[ 401], | 99.99th=[ 418] bw ( KiB/s): min= 159, max= 616, per=24.92%, avg=278.64, stdev=58.48, samples=72 iops : min= 39, max= 154, avg=69.57, stdev=14.69, samples=72 lat (msec) : 100=2.58%, 250=68.44%, 500=28.98% cpu : usr=0.14%, sys=1.34%, ctx=2723, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1119KiB/s (1146kB/s), 280KiB/s-280KiB/s (286kB/s-287kB/s), io=40.0MiB (41.9MB), run=36554-36605msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148005: Fri Jul 3 06:28:55 2026 read: IOPS=53, BW=213KiB/s (218kB/s)(5048KiB/23731msec) slat (usec): min=1215, max=11986, avg=3040.09, stdev=1255.08 clat (msec): min=4, max=179, avg=92.36, stdev=22.26 lat (msec): min=8, max=187, avg=95.40, stdev=22.68 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 133], | 99.00th=[ 159], 99.50th=[ 169], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 103, max= 352, per=25.18%, avg=212.53, stdev=57.92, samples=47 iops : min= 25, max= 88, avg=53.04, stdev=14.57, samples=47 write: IOPS=54, BW=219KiB/s (224kB/s)(5192KiB/23731msec); 0 zone resets slat (usec): min=60, max=2813, avg=216.61, stdev=149.52 clat (msec): min=69, max=372, avg=198.76, stdev=54.21 lat (msec): min=69, max=372, avg=198.97, stdev=54.22 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 116], 10.00th=[ 134], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 152, max= 304, per=24.86%, avg=218.51, stdev=42.39, samples=47 iops : min= 38, max= 76, avg=54.53, stdev=10.66, samples=47 lat (msec) : 10=0.16%, 20=0.20%, 50=0.31%, 100=34.69%, 250=55.23% lat (msec) : 500=9.41% cpu : usr=0.28%, sys=2.28%, ctx=4852, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148006: Fri Jul 3 06:28:55 2026 read: IOPS=52, BW=210KiB/s (215kB/s)(4976KiB/23695msec) slat (usec): min=57, max=13705, avg=3119.89, stdev=1418.77 clat (msec): min=3, max=187, avg=92.78, stdev=23.35 lat (msec): min=4, max=192, avg=95.90, stdev=23.83 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 108], 90.00th=[ 124], 95.00th=[ 138], | 99.00th=[ 167], 99.50th=[ 176], 99.90th=[ 182], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 88, max= 352, per=24.80%, avg=209.28, stdev=64.66, samples=47 iops : min= 22, max= 88, avg=52.21, stdev=16.21, samples=47 write: IOPS=55, BW=222KiB/s (227kB/s)(5264KiB/23695msec); 0 zone resets slat (usec): min=44, max=3601, avg=219.16, stdev=182.47 clat (msec): min=35, max=388, avg=196.97, stdev=55.47 lat (msec): min=35, max=388, avg=197.19, stdev=55.48 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 114], 10.00th=[ 132], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 192], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 372], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 136, max= 312, per=25.18%, avg=221.34, stdev=41.39, samples=47 iops : min= 34, max= 78, avg=55.23, stdev=10.42, samples=47 lat (msec) : 4=0.04%, 10=0.04%, 20=0.20%, 50=0.66%, 100=34.34% lat (msec) : 250=55.62%, 500=9.10% cpu : usr=0.29%, sys=2.26%, ctx=4761, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148007: Fri Jul 3 06:28:55 2026 read: IOPS=52, BW=210KiB/s (215kB/s)(4984KiB/23751msec) slat (usec): min=1182, max=11119, avg=3072.83, stdev=1227.73 clat (usec): min=1942, max=184191, avg=92599.54, stdev=23282.96 lat (msec): min=4, max=189, avg=95.68, stdev=23.68 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 110], 90.00th=[ 123], 95.00th=[ 133], | 99.00th=[ 159], 99.50th=[ 167], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 95, max= 368, per=24.86%, avg=209.79, stdev=54.25, samples=47 iops : min= 23, max= 92, avg=52.34, stdev=13.64, samples=47 write: IOPS=55, BW=221KiB/s (227kB/s)(5256KiB/23751msec); 0 zone resets slat (usec): min=59, max=2158, avg=225.87, stdev=143.76 clat (msec): min=51, max=388, avg=197.48, stdev=54.76 lat (msec): min=51, max=388, avg=197.71, stdev=54.77 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 115], 10.00th=[ 133], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 120, max= 304, per=24.97%, avg=219.51, stdev=47.54, samples=47 iops : min= 30, max= 76, avg=54.79, stdev=11.95, samples=47 lat (msec) : 2=0.04%, 4=0.04%, 10=0.08%, 20=0.23%, 50=0.39% lat (msec) : 100=32.97%, 250=57.62%, 500=8.63% cpu : usr=0.22%, sys=2.35%, ctx=4756, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148008: Fri Jul 3 06:28:55 2026 read: IOPS=53, BW=214KiB/s (219kB/s)(5060KiB/23640msec) slat (usec): min=1209, max=10451, avg=3045.31, stdev=1231.88 clat (msec): min=3, max=211, avg=93.15, stdev=24.24 lat (msec): min=6, max=214, avg=96.20, stdev=24.63 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 102], 80.00th=[ 110], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 167], 99.50th=[ 178], 99.90th=[ 186], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 127, max= 320, per=25.42%, avg=214.57, stdev=50.33, samples=46 iops : min= 31, max= 80, avg=53.54, stdev=12.63, samples=46 write: IOPS=54, BW=219KiB/s (224kB/s)(5180KiB/23640msec); 0 zone resets slat (usec): min=58, max=4039, avg=221.78, stdev=166.75 clat (msec): min=39, max=371, avg=197.11, stdev=54.16 lat (msec): min=39, max=371, avg=197.33, stdev=54.17 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 118], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 165], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 338], 99.50th=[ 355], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 127, max= 344, per=24.87%, avg=218.61, stdev=46.64, samples=46 iops : min= 31, max= 86, avg=54.54, stdev=11.75, samples=46 lat (msec) : 4=0.04%, 10=0.08%, 20=0.16%, 50=0.47%, 100=33.91% lat (msec) : 250=56.37%, 500=8.98% cpu : usr=0.27%, sys=2.35%, ctx=4849, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=845KiB/s (865kB/s), 210KiB/s-214KiB/s (215kB/s-219kB/s), io=19.6MiB (20.5MB), run=23640-23751msec WRITE: bw=880KiB/s (901kB/s), 219KiB/s-222KiB/s (224kB/s-227kB/s), io=20.4MiB (21.4MB), run=23640-23751msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148014: Fri Jul 3 06:28:59 2026 read: IOPS=6, BW=256MiB/s (269MB/s)(40.0MiB/156msec) slat (nsec): min=15295k, max=15295k, avg=15295237.00, stdev= 0.00 clat (nsec): min=133417k, max=133417k, avg=133416504.00, stdev= 0.00 lat (nsec): min=148717k, max=148717k, avg=148717074.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.74%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=40.0MiB (41.9MB), run=156-156msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout