== sanity test 398c: run fio to test AIO ================= 23:06:55 (1786936015) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.149436 s, 281 MB/s osc.lustre-OST0000-osc-ffff97e2852ce000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=161330: Sun Aug 16 23:07:27 2026 write: IOPS=90, BW=361KiB/s (369kB/s)(10.0MiB/28400msec); 0 zone resets slat (usec): min=40, max=849, avg=205.75, stdev=94.10 clat (msec): min=42, max=328, avg=177.11, stdev=32.68 lat (msec): min=42, max=329, avg=177.31, stdev=32.69 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 123], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 201], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 264], 99.50th=[ 268], 99.90th=[ 292], 99.95th=[ 321], | 99.99th=[ 330] bw ( KiB/s): min= 256, max= 612, per=24.96%, avg=359.93, stdev=48.75, samples=56 iops : min= 64, max= 153, avg=89.98, stdev=12.19, samples=56 lat (msec) : 50=0.62%, 100=0.98%, 250=96.33%, 500=2.07% cpu : usr=0.13%, sys=1.50%, ctx=2994, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=161331: Sun Aug 16 23:07:27 2026 write: IOPS=90, BW=361KiB/s (370kB/s)(10.0MiB/28370msec); 0 zone resets slat (usec): min=40, max=965, avg=206.03, stdev=97.73 clat (msec): min=33, max=343, avg=176.99, stdev=33.64 lat (msec): min=34, max=344, avg=177.20, stdev=33.65 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 116], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 180], 60.00th=[ 186], | 70.00th=[ 192], 80.00th=[ 201], 90.00th=[ 213], 95.00th=[ 226], | 99.00th=[ 262], 99.50th=[ 275], 99.90th=[ 309], 99.95th=[ 321], | 99.99th=[ 342] bw ( KiB/s): min= 248, max= 650, per=24.96%, avg=359.89, stdev=53.54, samples=56 iops : min= 62, max= 162, avg=89.96, stdev=13.33, samples=56 lat (msec) : 50=0.62%, 100=2.27%, 250=95.39%, 500=1.72% cpu : usr=0.13%, sys=1.48%, ctx=2894, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=161332: Sun Aug 16 23:07:27 2026 write: IOPS=90, BW=361KiB/s (369kB/s)(10.0MiB/28400msec); 0 zone resets slat (usec): min=40, max=813, avg=206.06, stdev=95.28 clat (msec): min=62, max=287, avg=177.07, stdev=31.62 lat (msec): min=63, max=287, avg=177.27, stdev=31.62 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 123], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 180], 60.00th=[ 186], | 70.00th=[ 192], 80.00th=[ 201], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 259], 99.50th=[ 266], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 288] bw ( KiB/s): min= 240, max= 612, per=24.90%, avg=359.07, stdev=51.97, samples=56 iops : min= 60, max= 153, avg=89.77, stdev=12.99, samples=56 lat (msec) : 100=2.70%, 250=95.78%, 500=1.52% cpu : usr=0.14%, sys=1.47%, ctx=2961, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=161333: Sun Aug 16 23:07:27 2026 write: IOPS=90, BW=361KiB/s (369kB/s)(10.0MiB/28399msec); 0 zone resets slat (usec): min=47, max=774, avg=205.32, stdev=90.46 clat (msec): min=41, max=308, avg=177.07, stdev=32.24 lat (msec): min=41, max=308, avg=177.28, stdev=32.24 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 120], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 192], 80.00th=[ 201], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 262], 99.50th=[ 275], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 309] bw ( KiB/s): min= 248, max= 625, per=24.91%, avg=359.16, stdev=52.00, samples=56 iops : min= 62, max= 156, avg=89.79, stdev=12.98, samples=56 lat (msec) : 50=0.27%, 100=2.19%, 250=96.05%, 500=1.48% cpu : usr=0.14%, sys=1.48%, ctx=3000, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1442KiB/s (1477kB/s), 361KiB/s-361KiB/s (369kB/s-370kB/s), io=40.0MiB (41.9MB), run=28370-28400msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=161346: Sun Aug 16 23:07:50 2026 read: IOPS=66, BW=264KiB/s (271kB/s)(5048KiB/19090msec) slat (usec): min=44, max=4875, avg=2320.86, stdev=728.87 clat (msec): min=3, max=195, avg=73.50, stdev=14.48 lat (msec): min=5, max=197, avg=75.82, stdev=14.56 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 88], 95.00th=[ 92], | 99.00th=[ 107], 99.50th=[ 124], 99.90th=[ 180], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 160, max= 344, per=25.31%, avg=265.22, stdev=45.44, samples=37 iops : min= 40, max= 86, avg=66.30, stdev=11.36, samples=37 write: IOPS=67, BW=272KiB/s (279kB/s)(5192KiB/19090msec); 0 zone resets slat (usec): min=50, max=634, avg=175.57, stdev=80.55 clat (msec): min=37, max=298, avg=161.17, stdev=34.16 lat (msec): min=37, max=298, avg=161.35, stdev=34.16 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 239], 99.50th=[ 251], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 216, max= 320, per=24.97%, avg=272.41, stdev=27.11, samples=37 iops : min= 54, max= 80, avg=68.08, stdev= 6.77, samples=37 lat (msec) : 4=0.04%, 10=0.08%, 20=0.23%, 50=1.37%, 100=48.55% lat (msec) : 250=49.45%, 500=0.27% cpu : usr=0.15%, sys=2.25%, ctx=4874, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=161347: Sun Aug 16 23:07:50 2026 read: IOPS=65, BW=260KiB/s (266kB/s)(4976KiB/19133msec) slat (usec): min=1022, max=6283, avg=2354.47, stdev=731.43 clat (msec): min=2, max=117, avg=72.77, stdev=13.29 lat (msec): min=4, max=118, avg=75.13, stdev=13.31 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 83], 90.00th=[ 87], 95.00th=[ 92], | 99.00th=[ 102], 99.50th=[ 106], 99.90th=[ 113], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 160, max= 392, per=24.84%, avg=260.32, stdev=53.32, samples=37 iops : min= 40, max= 98, avg=65.08, stdev=13.33, samples=37 write: IOPS=68, BW=275KiB/s (282kB/s)(5264KiB/19133msec); 0 zone resets slat (usec): min=53, max=621, avg=177.50, stdev=79.58 clat (msec): min=49, max=263, avg=160.88, stdev=34.53 lat (msec): min=49, max=263, avg=161.06, stdev=34.53 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 100], 10.00th=[ 114], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 192], 90.00th=[ 205], 95.00th=[ 218], | 99.00th=[ 241], 99.50th=[ 249], 99.90th=[ 255], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 216, max= 376, per=25.15%, avg=274.43, stdev=33.35, samples=37 iops : min= 54, max= 94, avg=68.59, stdev= 8.34, samples=37 lat (msec) : 4=0.16%, 10=0.08%, 20=0.23%, 50=1.29%, 100=48.87% lat (msec) : 250=49.18%, 500=0.20% cpu : usr=0.15%, sys=2.28%, ctx=4808, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=161348: Sun Aug 16 23:07:50 2026 read: IOPS=65, BW=261KiB/s (267kB/s)(4984KiB/19124msec) slat (usec): min=1029, max=6554, avg=2405.49, stdev=736.67 clat (msec): min=4, max=125, avg=73.30, stdev=12.48 lat (msec): min=6, max=127, avg=75.71, stdev=12.53 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 54], 10.00th=[ 60], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 83], 90.00th=[ 88], 95.00th=[ 93], | 99.00th=[ 105], 99.50th=[ 107], 99.90th=[ 123], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 152, max= 376, per=25.03%, avg=262.27, stdev=54.33, samples=37 iops : min= 38, max= 94, avg=65.57, stdev=13.58, samples=37 write: IOPS=68, BW=275KiB/s (281kB/s)(5256KiB/19124msec); 0 zone resets slat (usec): min=51, max=1808, avg=180.03, stdev=95.68 clat (msec): min=22, max=259, avg=160.35, stdev=34.65 lat (msec): min=22, max=259, avg=160.53, stdev=34.65 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 236], 99.50th=[ 245], 99.90th=[ 255], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 200, max= 344, per=25.05%, avg=273.27, stdev=34.60, samples=37 iops : min= 50, max= 86, avg=68.30, stdev= 8.66, samples=37 lat (msec) : 10=0.08%, 20=0.12%, 50=1.25%, 100=48.59%, 250=49.84% lat (msec) : 500=0.12% cpu : usr=0.20%, sys=2.25%, ctx=4847, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=161349: Sun Aug 16 23:07:50 2026 read: IOPS=66, BW=265KiB/s (271kB/s)(5060KiB/19113msec) slat (usec): min=1010, max=5286, avg=2370.53, stdev=727.58 clat (msec): min=2, max=172, avg=73.84, stdev=13.16 lat (msec): min=5, max=174, avg=76.21, stdev=13.24 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 83], 90.00th=[ 88], 95.00th=[ 93], | 99.00th=[ 109], 99.50th=[ 117], 99.90th=[ 157], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 152, max= 352, per=25.43%, avg=266.51, stdev=49.45, samples=37 iops : min= 38, max= 88, avg=66.62, stdev=12.37, samples=37 write: IOPS=67, BW=271KiB/s (278kB/s)(5180KiB/19113msec); 0 zone resets slat (usec): min=52, max=692, avg=181.80, stdev=82.27 clat (msec): min=71, max=288, avg=160.79, stdev=34.33 lat (msec): min=71, max=288, avg=160.97, stdev=34.32 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 205], 95.00th=[ 218], | 99.00th=[ 245], 99.50th=[ 268], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 200, max= 336, per=24.83%, avg=270.92, stdev=33.05, samples=37 iops : min= 50, max= 84, avg=67.73, stdev= 8.26, samples=37 lat (msec) : 4=0.12%, 10=0.04%, 20=0.16%, 50=0.74%, 100=49.34% lat (msec) : 250=49.22%, 500=0.39% cpu : usr=0.23%, sys=2.24%, ctx=4892, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1049KiB/s (1074kB/s), 260KiB/s-265KiB/s (266kB/s-271kB/s), io=19.6MiB (20.5MB), run=19090-19133msec WRITE: bw=1092KiB/s (1118kB/s), 271KiB/s-275KiB/s (278kB/s-282kB/s), io=20.4MiB (21.4MB), run=19090-19133msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=161356: Sun Aug 16 23:07:53 2026 read: IOPS=8, BW=325MiB/s (341MB/s)(40.0MiB/123msec) slat (nsec): min=9169.9k, max=9169.9k, avg=9169910.00, stdev= 0.00 clat (nsec): min=107889k, max=107889k, avg=107889288.00, stdev= 0.00 lat (nsec): min=117073k, max=117073k, avg=117072830.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.56%, ctx=40, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=40.0MiB (41.9MB), run=123-123msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout