== sanity test 398c: run fio to test AIO ================= 23:05:08 (1784343908) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.164145 s, 256 MB/s osc.lustre-OST0000-osc-ffff92a550d7b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=195921: Fri Jul 17 23:05:24 2026 write: IOPS=218, BW=874KiB/s (895kB/s)(10.0MiB/11719msec); 0 zone resets slat (usec): min=36, max=31539, avg=265.65, stdev=676.02 clat (msec): min=9, max=120, avg=72.57, stdev=15.08 lat (msec): min=16, max=121, avg=72.84, stdev=15.05 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 91], 95.00th=[ 99], | 99.00th=[ 109], 99.50th=[ 113], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 122] bw ( KiB/s): min= 648, max= 1160, per=24.93%, avg=871.35, stdev=126.04, samples=23 iops : min= 162, max= 290, avg=217.74, stdev=31.48, samples=23 lat (msec) : 10=0.04%, 20=0.55%, 50=4.10%, 100=90.82%, 250=4.49% cpu : usr=0.42%, sys=3.77%, ctx=2757, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195922: Fri Jul 17 23:05:24 2026 write: IOPS=219, BW=878KiB/s (899kB/s)(10.0MiB/11662msec); 0 zone resets slat (usec): min=54, max=8810, avg=249.25, stdev=291.01 clat (msec): min=10, max=121, avg=72.55, stdev=15.12 lat (msec): min=10, max=121, avg=72.80, stdev=15.14 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 91], 95.00th=[ 100], | 99.00th=[ 110], 99.50th=[ 114], 99.90th=[ 117], 99.95th=[ 120], | 99.99th=[ 122] bw ( KiB/s): min= 688, max= 1168, per=24.97%, avg=872.74, stdev=133.06, samples=23 iops : min= 172, max= 292, avg=218.09, stdev=33.20, samples=23 lat (msec) : 20=0.66%, 50=3.44%, 100=91.33%, 250=4.57% cpu : usr=0.38%, sys=3.62%, ctx=2744, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195923: Fri Jul 17 23:05:24 2026 write: IOPS=218, BW=874KiB/s (895kB/s)(10.0MiB/11710msec); 0 zone resets slat (usec): min=38, max=26625, avg=258.38, stdev=583.25 clat (msec): min=9, max=122, avg=72.53, stdev=14.98 lat (msec): min=10, max=122, avg=72.79, stdev=14.95 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 91], 95.00th=[ 99], | 99.00th=[ 110], 99.50th=[ 113], 99.90th=[ 121], 99.95th=[ 122], | 99.99th=[ 123] bw ( KiB/s): min= 672, max= 1160, per=24.94%, avg=871.70, stdev=128.28, samples=23 iops : min= 168, max= 290, avg=217.83, stdev=32.04, samples=23 lat (msec) : 10=0.04%, 20=0.43%, 50=3.83%, 100=91.45%, 250=4.26% cpu : usr=0.35%, sys=3.83%, ctx=2854, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195924: Fri Jul 17 23:05:24 2026 write: IOPS=218, BW=874KiB/s (895kB/s)(10.0MiB/11719msec); 0 zone resets slat (usec): min=39, max=4143, avg=220.34, stdev=232.46 clat (msec): min=5, max=122, avg=72.59, stdev=14.98 lat (msec): min=6, max=122, avg=72.81, stdev=14.98 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 90], 95.00th=[ 100], | 99.00th=[ 113], 99.50th=[ 114], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 123] bw ( KiB/s): min= 648, max= 1160, per=24.90%, avg=870.22, stdev=122.39, samples=23 iops : min= 162, max= 290, avg=217.48, stdev=30.59, samples=23 lat (msec) : 10=0.16%, 50=4.14%, 100=90.82%, 250=4.88% cpu : usr=0.34%, sys=3.37%, ctx=2912, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3495KiB/s (3579kB/s), 874KiB/s-878KiB/s (895kB/s-899kB/s), io=40.0MiB (41.9MB), run=11662-11719msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=195935: Fri Jul 17 23:05:39 2026 read: IOPS=114, BW=457KiB/s (468kB/s)(5048KiB/11034msec) slat (usec): min=64, max=17454, avg=4507.51, stdev=1837.21 clat (msec): min=13, max=117, avg=65.52, stdev=12.41 lat (msec): min=19, max=120, avg=70.03, stdev=12.84 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 75], 90.00th=[ 83], 95.00th=[ 88], | 99.00th=[ 97], 99.50th=[ 103], 99.90th=[ 115], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 383, max= 584, per=25.05%, avg=454.38, stdev=53.79, samples=21 iops : min= 95, max= 146, avg=113.33, stdev=13.61, samples=21 write: IOPS=117, BW=471KiB/s (482kB/s)(5192KiB/11034msec); 0 zone resets slat (usec): min=54, max=2547, avg=218.17, stdev=212.88 clat (msec): min=29, max=114, avg=67.44, stdev=11.76 lat (msec): min=29, max=114, avg=67.66, stdev=11.77 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 51], 10.00th=[ 54], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 101], 99.50th=[ 105], 99.90th=[ 112], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 352, max= 608, per=24.75%, avg=467.57, stdev=58.51, samples=21 iops : min= 88, max= 152, avg=116.57, stdev=14.60, samples=21 lat (msec) : 20=0.08%, 50=6.95%, 100=92.19%, 250=0.78% cpu : usr=0.42%, sys=4.00%, ctx=4622, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195936: Fri Jul 17 23:05:39 2026 read: IOPS=112, BW=451KiB/s (461kB/s)(4976KiB/11045msec) slat (usec): min=1692, max=16438, avg=4563.69, stdev=1796.81 clat (msec): min=8, max=112, avg=65.61, stdev=12.43 lat (msec): min=17, max=123, avg=70.18, stdev=12.87 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 112], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 383, max= 544, per=24.60%, avg=446.24, stdev=41.08, samples=21 iops : min= 95, max= 136, avg=111.29, stdev=10.29, samples=21 write: IOPS=119, BW=477KiB/s (488kB/s)(5264KiB/11045msec); 0 zone resets slat (usec): min=57, max=3397, avg=221.16, stdev=223.71 clat (msec): min=29, max=104, avg=67.07, stdev=10.76 lat (msec): min=30, max=104, avg=67.30, stdev=10.76 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 54], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 95], 99.50th=[ 99], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 351, max= 582, per=25.08%, avg=473.71, stdev=68.06, samples=21 iops : min= 87, max= 145, avg=118.14, stdev=17.07, samples=21 lat (msec) : 10=0.04%, 50=6.29%, 100=93.12%, 250=0.55% cpu : usr=0.45%, sys=4.09%, ctx=4653, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195937: Fri Jul 17 23:05:39 2026 read: IOPS=112, BW=451KiB/s (462kB/s)(4984KiB/11057msec) slat (usec): min=1572, max=17282, avg=4536.04, stdev=1804.32 clat (msec): min=12, max=107, avg=65.32, stdev=12.44 lat (msec): min=19, max=113, avg=69.86, stdev=12.77 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 83], 95.00th=[ 88], | 99.00th=[ 100], 99.50th=[ 103], 99.90th=[ 107], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 334, max= 558, per=24.67%, avg=447.52, stdev=61.94, samples=21 iops : min= 83, max= 139, avg=111.62, stdev=15.54, samples=21 write: IOPS=118, BW=475KiB/s (487kB/s)(5256KiB/11057msec); 0 zone resets slat (usec): min=53, max=3266, avg=217.28, stdev=207.69 clat (msec): min=14, max=116, avg=67.18, stdev=11.26 lat (msec): min=14, max=116, avg=67.40, stdev=11.26 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 97], 99.50th=[ 105], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 344, max= 608, per=24.74%, avg=467.33, stdev=64.93, samples=21 iops : min= 86, max= 152, avg=116.57, stdev=16.31, samples=21 lat (msec) : 20=0.20%, 50=6.02%, 100=92.93%, 250=0.86% cpu : usr=0.45%, sys=4.07%, ctx=4669, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195938: Fri Jul 17 23:05:39 2026 read: IOPS=114, BW=458KiB/s (469kB/s)(5060KiB/11056msec) slat (usec): min=1487, max=14557, avg=4447.64, stdev=1731.85 clat (msec): min=12, max=109, avg=65.24, stdev=11.72 lat (msec): min=16, max=115, avg=69.69, stdev=12.15 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 96], 99.50th=[ 102], 99.90th=[ 107], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 352, max= 563, per=24.88%, avg=451.29, stdev=57.33, samples=21 iops : min= 88, max= 140, avg=112.52, stdev=14.36, samples=21 write: IOPS=117, BW=469KiB/s (480kB/s)(5180KiB/11056msec); 0 zone resets slat (usec): min=57, max=4230, avg=225.91, stdev=259.97 clat (msec): min=42, max=116, avg=67.42, stdev=11.39 lat (msec): min=42, max=116, avg=67.65, stdev=11.39 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 89], | 99.00th=[ 96], 99.50th=[ 100], 99.90th=[ 112], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 367, max= 576, per=24.66%, avg=465.86, stdev=47.72, samples=21 iops : min= 91, max= 144, avg=116.19, stdev=12.00, samples=21 lat (msec) : 20=0.08%, 50=5.78%, 100=93.63%, 250=0.51% cpu : usr=0.47%, sys=3.93%, ctx=4767, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1815KiB/s (1859kB/s), 451KiB/s-458KiB/s (461kB/s-469kB/s), io=19.6MiB (20.5MB), run=11034-11057msec WRITE: bw=1889KiB/s (1935kB/s), 469KiB/s-477KiB/s (480kB/s-488kB/s), io=20.4MiB (21.4MB), run=11034-11057msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=195944: Fri Jul 17 23:05:43 2026 read: IOPS=5, BW=220MiB/s (230MB/s)(40.0MiB/182msec) slat (nsec): min=13546k, max=13546k, avg=13545667.00, stdev= 0.00 clat (nsec): min=161502k, max=161502k, avg=161501931.00, stdev= 0.00 lat (nsec): min=175051k, max=175051k, avg=175050919.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.52%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=40.0MiB (41.9MB), run=182-182msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout