== sanity test 398c: run fio to test AIO ================= 10:36:18 (1784126178) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.129908 s, 323 MB/s osc.lustre-OST0000-osc-ffff8aa78a089000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=153996: Wed Jul 15 10:36:53 2026 write: IOPS=81, BW=326KiB/s (333kB/s)(10.0MiB/31449msec); 0 zone resets slat (usec): min=35, max=822, avg=210.58, stdev=104.27 clat (msec): min=41, max=342, avg=196.18, stdev=42.69 lat (msec): min=41, max=342, avg=196.40, stdev=42.71 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 131], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 207], | 70.00th=[ 218], 80.00th=[ 230], 90.00th=[ 249], 95.00th=[ 262], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 224, max= 609, per=24.93%, avg=324.53, stdev=60.25, samples=62 iops : min= 56, max= 152, avg=81.13, stdev=15.04, samples=62 lat (msec) : 50=0.62%, 100=2.58%, 250=87.77%, 500=9.02% cpu : usr=0.17%, sys=1.37%, ctx=2837, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153997: Wed Jul 15 10:36:53 2026 write: IOPS=81, BW=326KiB/s (334kB/s)(10.0MiB/31383msec); 0 zone resets slat (usec): min=34, max=1315, avg=200.89, stdev=110.03 clat (msec): min=25, max=336, avg=195.83, stdev=43.68 lat (msec): min=25, max=336, avg=196.03, stdev=43.70 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 126], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 218], 80.00th=[ 230], 90.00th=[ 249], 95.00th=[ 268], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 338] bw ( KiB/s): min= 200, max= 653, per=24.97%, avg=325.11, stdev=66.01, samples=62 iops : min= 50, max= 163, avg=81.27, stdev=16.48, samples=62 lat (msec) : 50=0.62%, 100=2.66%, 250=86.99%, 500=9.73% cpu : usr=0.18%, sys=1.29%, ctx=2783, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153998: Wed Jul 15 10:36:53 2026 write: IOPS=81, BW=326KiB/s (333kB/s)(10.0MiB/31449msec); 0 zone resets slat (usec): min=38, max=765, avg=221.92, stdev=102.17 clat (msec): min=67, max=337, avg=196.14, stdev=41.41 lat (msec): min=67, max=337, avg=196.37, stdev=41.43 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 131], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 230], 90.00th=[ 249], 95.00th=[ 264], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 338] bw ( KiB/s): min= 208, max= 609, per=24.86%, avg=323.63, stdev=58.07, samples=62 iops : min= 52, max= 152, avg=80.90, stdev=14.50, samples=62 lat (msec) : 100=2.73%, 250=88.52%, 500=8.75% cpu : usr=0.15%, sys=1.48%, ctx=2863, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153999: Wed Jul 15 10:36:53 2026 write: IOPS=81, BW=326KiB/s (333kB/s)(10.0MiB/31457msec); 0 zone resets slat (usec): min=47, max=1215, avg=213.75, stdev=102.81 clat (msec): min=19, max=334, avg=196.25, stdev=42.48 lat (msec): min=19, max=334, avg=196.47, stdev=42.49 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 130], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 230], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 232, max= 628, per=24.83%, avg=323.29, stdev=61.33, samples=62 iops : min= 58, max= 157, avg=80.82, stdev=15.33, samples=62 lat (msec) : 20=0.08%, 50=0.04%, 100=2.89%, 250=88.44%, 500=8.55% cpu : usr=0.12%, sys=1.43%, ctx=2849, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1302KiB/s (1333kB/s), 326KiB/s-326KiB/s (333kB/s-334kB/s), io=40.0MiB (41.9MB), run=31383-31457msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154013: Wed Jul 15 10:37:17 2026 read: IOPS=59, BW=239KiB/s (245kB/s)(5048KiB/21079msec) slat (usec): min=1063, max=7035, avg=2581.99, stdev=899.52 clat (msec): min=2, max=160, avg=82.53, stdev=16.82 lat (msec): min=4, max=162, avg=85.11, stdev=16.92 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 96], 90.00th=[ 104], 95.00th=[ 109], | 99.00th=[ 123], 99.50th=[ 125], 99.90th=[ 157], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 136, max= 368, per=25.16%, avg=239.49, stdev=50.04, samples=41 iops : min= 34, max= 92, avg=59.85, stdev=12.51, samples=41 write: IOPS=61, BW=246KiB/s (252kB/s)(5192KiB/21079msec); 0 zone resets slat (usec): min=55, max=3104, avg=200.94, stdev=137.21 clat (msec): min=54, max=291, avg=176.27, stdev=41.31 lat (msec): min=54, max=292, avg=176.47, stdev=41.32 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 109], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 176, max= 320, per=24.77%, avg=245.44, stdev=32.01, samples=41 iops : min= 44, max= 80, avg=61.34, stdev= 7.99, samples=41 lat (msec) : 4=0.04%, 10=0.12%, 20=0.20%, 50=0.74%, 100=42.97% lat (msec) : 250=53.28%, 500=2.66% cpu : usr=0.23%, sys=2.29%, ctx=4996, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154014: Wed Jul 15 10:37:17 2026 read: IOPS=59, BW=236KiB/s (242kB/s)(4976KiB/21046msec) slat (usec): min=1078, max=5946, avg=2567.29, stdev=856.07 clat (msec): min=2, max=160, avg=82.38, stdev=16.33 lat (msec): min=4, max=162, avg=84.95, stdev=16.43 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 122], 99.50th=[ 125], 99.90th=[ 159], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 128, max= 376, per=24.83%, avg=236.34, stdev=61.59, samples=41 iops : min= 32, max= 94, avg=59.07, stdev=15.39, samples=41 write: IOPS=62, BW=250KiB/s (256kB/s)(5264KiB/21046msec); 0 zone resets slat (usec): min=48, max=754, avg=187.92, stdev=89.62 clat (msec): min=68, max=293, avg=174.92, stdev=41.55 lat (msec): min=68, max=293, avg=175.11, stdev=41.56 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 107], 10.00th=[ 126], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 279], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 168, max= 384, per=25.17%, avg=249.41, stdev=40.67, samples=41 iops : min= 42, max= 96, avg=62.34, stdev=10.16, samples=41 lat (msec) : 4=0.12%, 20=0.20%, 50=0.35%, 100=43.40%, 250=53.24% lat (msec) : 500=2.70% cpu : usr=0.17%, sys=2.30%, ctx=4866, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154015: Wed Jul 15 10:37:17 2026 read: IOPS=59, BW=236KiB/s (242kB/s)(4984KiB/21079msec) slat (usec): min=981, max=6376, avg=2556.80, stdev=866.08 clat (msec): min=3, max=125, avg=82.23, stdev=15.92 lat (msec): min=6, max=129, avg=84.79, stdev=16.03 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 120], 99.50th=[ 123], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 144, max= 360, per=24.88%, avg=236.88, stdev=55.57, samples=41 iops : min= 36, max= 90, avg=59.22, stdev=13.89, samples=41 write: IOPS=62, BW=249KiB/s (255kB/s)(5256KiB/21079msec); 0 zone resets slat (usec): min=47, max=667, avg=191.30, stdev=92.15 clat (msec): min=46, max=293, avg=175.46, stdev=42.70 lat (msec): min=47, max=293, avg=175.65, stdev=42.71 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 106], 10.00th=[ 122], 20.00th=[ 140], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 176, max= 328, per=24.91%, avg=246.83, stdev=36.99, samples=41 iops : min= 44, max= 82, avg=61.71, stdev= 9.25, samples=41 lat (msec) : 4=0.04%, 10=0.04%, 20=0.20%, 50=0.62%, 100=43.48% lat (msec) : 250=52.85%, 500=2.77% cpu : usr=0.19%, sys=2.27%, ctx=4911, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154016: Wed Jul 15 10:37:17 2026 read: IOPS=60, BW=242KiB/s (248kB/s)(5060KiB/20930msec) slat (usec): min=86, max=6906, avg=2562.91, stdev=912.87 clat (msec): min=3, max=194, avg=82.16, stdev=17.48 lat (msec): min=5, max=197, avg=84.72, stdev=17.62 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 57], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 111], | 99.00th=[ 121], 99.50th=[ 123], 99.90th=[ 148], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 160, max= 344, per=25.56%, avg=243.37, stdev=46.48, samples=41 iops : min= 40, max= 86, avg=60.83, stdev=11.60, samples=41 write: IOPS=61, BW=247KiB/s (253kB/s)(5180KiB/20930msec); 0 zone resets slat (usec): min=57, max=916, avg=194.79, stdev=97.61 clat (msec): min=26, max=308, avg=175.44, stdev=43.15 lat (msec): min=26, max=308, avg=175.64, stdev=43.15 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 108], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 168, max= 344, per=24.87%, avg=246.41, stdev=40.08, samples=41 iops : min= 42, max= 86, avg=61.59, stdev= 9.98, samples=41 lat (msec) : 4=0.08%, 10=0.27%, 20=0.27%, 50=0.90%, 100=43.40% lat (msec) : 250=51.72%, 500=3.36% cpu : usr=0.22%, sys=2.26%, ctx=4986, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=952KiB/s (975kB/s), 236KiB/s-242KiB/s (242kB/s-248kB/s), io=19.6MiB (20.5MB), run=20930-21079msec WRITE: bw=991KiB/s (1015kB/s), 246KiB/s-250KiB/s (252kB/s-256kB/s), io=20.4MiB (21.4MB), run=20930-21079msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154022: Wed Jul 15 10:37:20 2026 read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec) slat (nsec): min=10425k, max=10425k, avg=10424672.00, stdev= 0.00 clat (nsec): min=95494k, max=95494k, avg=95494178.00, stdev= 0.00 lat (nsec): min=105923k, max=105923k, avg=105922692.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[95945], 5.00th=[95945], 10.00th=[95945], 20.00th=[95945], | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.26%, ctx=40, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout