== sanity test 398c: run fio to test AIO ================= 21:50:57 (1783648257) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.175411 s, 239 MB/s osc.lustre-OST0000-osc-ffff92f5066bb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=192294: Thu Jul 9 21:51:10 2026 write: IOPS=280, BW=1122KiB/s (1149kB/s)(10.0MiB/9123msec); 0 zone resets slat (usec): min=38, max=2865, avg=215.02, stdev=160.85 clat (msec): min=9, max=103, avg=56.73, stdev=11.56 lat (msec): min=9, max=103, avg=56.94, stdev=11.55 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 80], | 99.00th=[ 96], 99.50th=[ 102], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 728, max= 1256, per=24.80%, avg=1109.18, stdev=130.78, samples=17 iops : min= 182, max= 314, avg=277.24, stdev=32.66, samples=17 lat (msec) : 10=0.08%, 20=0.08%, 50=28.95%, 100=70.31%, 250=0.59% cpu : usr=0.38%, sys=4.39%, ctx=2627, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192295: Thu Jul 9 21:51:10 2026 write: IOPS=279, BW=1118KiB/s (1145kB/s)(10.0MiB/9156msec); 0 zone resets slat (usec): min=52, max=26095, avg=236.66, stdev=534.78 clat (msec): min=9, max=104, avg=56.65, stdev=11.50 lat (msec): min=10, max=104, avg=56.89, stdev=11.47 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 80], | 99.00th=[ 95], 99.50th=[ 100], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 105] bw ( KiB/s): min= 744, max= 1296, per=24.94%, avg=1115.50, stdev=132.41, samples=18 iops : min= 186, max= 324, avg=278.83, stdev=33.09, samples=18 lat (msec) : 10=0.04%, 20=0.04%, 50=29.49%, 100=70.27%, 250=0.16% cpu : usr=0.49%, sys=4.80%, ctx=2675, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192296: Thu Jul 9 21:51:10 2026 write: IOPS=279, BW=1119KiB/s (1146kB/s)(10.0MiB/9152msec); 0 zone resets slat (usec): min=36, max=28493, avg=226.47, stdev=579.67 clat (msec): min=5, max=104, avg=56.71, stdev=11.75 lat (msec): min=5, max=104, avg=56.94, stdev=11.72 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 81], | 99.00th=[ 94], 99.50th=[ 100], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 736, max= 1280, per=24.93%, avg=1115.33, stdev=131.38, samples=18 iops : min= 184, max= 320, avg=278.78, stdev=32.82, samples=18 lat (msec) : 10=0.04%, 20=0.16%, 50=29.69%, 100=69.73%, 250=0.39% cpu : usr=0.54%, sys=4.48%, ctx=2777, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192297: Thu Jul 9 21:51:10 2026 write: IOPS=279, BW=1118KiB/s (1145kB/s)(10.0MiB/9156msec); 0 zone resets slat (usec): min=40, max=2855, avg=208.00, stdev=174.69 clat (msec): min=4, max=104, avg=56.66, stdev=11.68 lat (msec): min=5, max=104, avg=56.87, stdev=11.68 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 81], | 99.00th=[ 96], 99.50th=[ 102], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 760, max= 1320, per=24.90%, avg=1113.56, stdev=130.21, samples=18 iops : min= 190, max= 330, avg=278.33, stdev=32.53, samples=18 lat (msec) : 10=0.04%, 20=0.16%, 50=28.40%, 100=70.55%, 250=0.86% cpu : usr=0.46%, sys=4.33%, ctx=2663, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4474KiB/s (4581kB/s), 1118KiB/s-1122KiB/s (1145kB/s-1149kB/s), io=40.0MiB (41.9MB), run=9123-9156msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=192308: Thu Jul 9 21:51:22 2026 read: IOPS=142, BW=570KiB/s (583kB/s)(5048KiB/8862msec) slat (usec): min=1292, max=10341, avg=3682.66, stdev=1339.64 clat (msec): min=24, max=104, avg=52.61, stdev=10.84 lat (msec): min=26, max=109, avg=56.30, stdev=11.26 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 52], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 62], 90.00th=[ 69], 95.00th=[ 73], | 99.00th=[ 82], 99.50th=[ 86], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 361, max= 726, per=25.17%, avg=569.88, stdev=103.61, samples=17 iops : min= 90, max= 181, avg=142.35, stdev=25.92, samples=17 write: IOPS=146, BW=586KiB/s (600kB/s)(5192KiB/8862msec); 0 zone resets slat (usec): min=57, max=2336, avg=196.13, stdev=170.64 clat (usec): min=31877, max=98748, avg=53379.61, stdev=10213.19 lat (usec): min=32009, max=99281, avg=53577.01, stdev=10213.51 clat percentiles (usec): | 1.00th=[34866], 5.00th=[39060], 10.00th=[41681], 20.00th=[44827], | 30.00th=[47449], 40.00th=[49021], 50.00th=[51643], 60.00th=[53740], | 70.00th=[57410], 80.00th=[61080], 90.00th=[67634], 95.00th=[72877], | 99.00th=[81265], 99.50th=[84411], 99.90th=[95945], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 400, max= 710, per=24.77%, avg=583.94, stdev=86.32, samples=17 iops : min= 100, max= 177, avg=145.88, stdev=21.54, samples=17 lat (msec) : 50=44.73%, 100=55.20%, 250=0.08% cpu : usr=0.54%, sys=4.64%, ctx=4636, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192309: Thu Jul 9 21:51:22 2026 read: IOPS=140, BW=562KiB/s (575kB/s)(4976KiB/8859msec) slat (usec): min=1348, max=12115, avg=3659.43, stdev=1358.23 clat (usec): min=26534, max=96346, avg=52839.66, stdev=11165.04 lat (msec): min=28, max=104, avg=56.50, stdev=11.61 clat percentiles (usec): | 1.00th=[31589], 5.00th=[37487], 10.00th=[40109], 20.00th=[43254], | 30.00th=[46924], 40.00th=[48497], 50.00th=[50594], 60.00th=[53740], | 70.00th=[57410], 80.00th=[62129], 90.00th=[68682], 95.00th=[73925], | 99.00th=[84411], 99.50th=[89654], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 342, max= 696, per=24.77%, avg=560.76, stdev=97.15, samples=17 iops : min= 85, max= 174, avg=140.06, stdev=24.33, samples=17 write: IOPS=148, BW=594KiB/s (608kB/s)(5264KiB/8859msec); 0 zone resets slat (usec): min=55, max=1848, avg=187.40, stdev=130.32 clat (usec): min=29227, max=97943, avg=53269.79, stdev=10358.30 lat (usec): min=29384, max=98046, avg=53458.35, stdev=10364.20 clat percentiles (usec): | 1.00th=[35914], 5.00th=[40109], 10.00th=[42206], 20.00th=[44827], | 30.00th=[46924], 40.00th=[49021], 50.00th=[51119], 60.00th=[53740], | 70.00th=[56886], 80.00th=[61604], 90.00th=[67634], 95.00th=[73925], | 99.00th=[83362], 99.50th=[87557], 99.90th=[92799], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 394, max= 718, per=25.28%, avg=595.88, stdev=91.25, samples=17 iops : min= 98, max= 179, avg=148.82, stdev=22.89, samples=17 lat (msec) : 50=45.62%, 100=54.38% cpu : usr=0.55%, sys=4.56%, ctx=4540, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192310: Thu Jul 9 21:51:22 2026 read: IOPS=140, BW=563KiB/s (577kB/s)(4984KiB/8849msec) slat (usec): min=1436, max=12301, avg=3680.95, stdev=1300.06 clat (usec): min=20204, max=94911, avg=52592.92, stdev=10942.42 lat (usec): min=24571, max=99467, avg=56277.25, stdev=11257.09 clat percentiles (usec): | 1.00th=[33162], 5.00th=[37487], 10.00th=[39584], 20.00th=[42730], | 30.00th=[46400], 40.00th=[49021], 50.00th=[51119], 60.00th=[53740], | 70.00th=[56886], 80.00th=[61604], 90.00th=[67634], 95.00th=[72877], | 99.00th=[84411], 99.50th=[89654], 99.90th=[93848], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 355, max= 678, per=25.01%, avg=566.24, stdev=85.60, samples=17 iops : min= 88, max= 169, avg=141.41, stdev=21.50, samples=17 write: IOPS=148, BW=594KiB/s (608kB/s)(5256KiB/8849msec); 0 zone resets slat (usec): min=41, max=3894, avg=203.01, stdev=191.22 clat (usec): min=31725, max=97048, avg=53455.78, stdev=10370.07 lat (usec): min=31907, max=97090, avg=53661.60, stdev=10372.99 clat percentiles (usec): | 1.00th=[36439], 5.00th=[39584], 10.00th=[41681], 20.00th=[44827], | 30.00th=[46924], 40.00th=[49546], 50.00th=[51643], 60.00th=[54789], | 70.00th=[57410], 80.00th=[61080], 90.00th=[68682], 95.00th=[73925], | 99.00th=[82314], 99.50th=[83362], 99.90th=[90702], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 424, max= 733, per=25.15%, avg=592.88, stdev=100.74, samples=17 iops : min= 106, max= 183, avg=148.12, stdev=25.15, samples=17 lat (msec) : 50=43.83%, 100=56.17% cpu : usr=0.49%, sys=4.74%, ctx=4615, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192311: Thu Jul 9 21:51:22 2026 read: IOPS=143, BW=574KiB/s (588kB/s)(5060KiB/8815msec) slat (usec): min=92, max=18178, avg=3653.10, stdev=1464.01 clat (msec): min=8, max=103, avg=52.39, stdev=11.13 lat (msec): min=8, max=108, avg=56.05, stdev=11.57 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 38], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 85], 99.50th=[ 89], 99.90th=[ 101], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 426, max= 720, per=25.44%, avg=576.06, stdev=86.41, samples=17 iops : min= 106, max= 180, avg=143.88, stdev=21.60, samples=17 write: IOPS=146, BW=588KiB/s (602kB/s)(5180KiB/8815msec); 0 zone resets slat (usec): min=59, max=2095, avg=198.29, stdev=154.70 clat (msec): min=32, max=100, avg=53.76, stdev=10.20 lat (msec): min=32, max=100, avg=53.96, stdev=10.21 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 63], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 85], 99.50th=[ 87], 99.90th=[ 91], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 424, max= 720, per=24.89%, avg=586.65, stdev=82.43, samples=17 iops : min= 106, max= 180, avg=146.53, stdev=20.68, samples=17 lat (msec) : 10=0.08%, 20=0.04%, 50=42.93%, 100=56.84%, 250=0.12% cpu : usr=0.48%, sys=4.70%, ctx=4662, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2265KiB/s (2319kB/s), 562KiB/s-574KiB/s (575kB/s-588kB/s), io=19.6MiB (20.5MB), run=8815-8862msec WRITE: bw=2357KiB/s (2414kB/s), 586KiB/s-594KiB/s (600kB/s-608kB/s), io=20.4MiB (21.4MB), run=8815-8862msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=192316: Thu Jul 9 21:51:26 2026 read: IOPS=7, BW=288MiB/s (302MB/s)(40.0MiB/139msec) slat (nsec): min=12424k, max=12424k, avg=12424297.00, stdev= 0.00 clat (nsec): min=120623k, max=120623k, avg=120623081.00, stdev= 0.00 lat (nsec): min=133051k, max=133051k, avg=133050870.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.91%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=40.0MiB (41.9MB), run=139-139msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout