== sanity test 398c: run fio to test AIO ================= 10:31:57 (1746801117)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.14669 s, 286 MB/s
osc.lustre-OST0000-osc-ffff8800b5c5e000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=27652: Fri May  9 10:32:06 2025
  write: IOPS=357, BW=1431KiB/s (1465kB/s)(10.0MiB/7156msec)
    slat (usec): min=24, max=346, avg=83.15, stdev=47.35
    clat (usec): min=6441, max=82466, avg=44455.98, stdev=6135.40
     lat (usec): min=6555, max=82532, avg=44540.93, stdev=6139.64
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[34341], 10.00th=[36963], 20.00th=[40633],
     | 30.00th=[43254], 40.00th=[45351], 50.00th=[46400], 60.00th=[46924],
     | 70.00th=[47973], 80.00th=[48497], 90.00th=[49546], 95.00th=[50070],
     | 99.00th=[50594], 99.50th=[54264], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1312, max= 1704, per=25.01%, avg=1430.86, stdev=131.52, samples=14
   iops        : min=  328, max=  426, avg=357.71, stdev=32.88, samples=14
  lat (msec)   : 10=0.04%, 20=0.66%, 50=95.59%, 100=3.71%
  cpu          : usr=0.48%, sys=3.34%, ctx=1597, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=27653: Fri May  9 10:32:06 2025
  write: IOPS=359, BW=1437KiB/s (1472kB/s)(10.0MiB/7124msec)
    slat (usec): min=23, max=725, avg=95.56, stdev=54.21
    clat (usec): min=4933, max=78318, avg=44297.24, stdev=6239.74
     lat (usec): min=5026, max=78462, avg=44394.74, stdev=6243.13
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[32375], 10.00th=[36439], 20.00th=[40633],
     | 30.00th=[43779], 40.00th=[45351], 50.00th=[46400], 60.00th=[46924],
     | 70.00th=[47449], 80.00th=[48497], 90.00th=[49546], 95.00th=[50070],
     | 99.00th=[52167], 99.50th=[52691], 99.90th=[73925], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1288, max= 1772, per=25.10%, avg=1435.71, stdev=149.49, samples=14
   iops        : min=  322, max=  443, avg=358.93, stdev=37.37, samples=14
  lat (msec)   : 10=0.16%, 20=0.86%, 50=94.80%, 100=4.18%
  cpu          : usr=0.53%, sys=3.75%, ctx=2219, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=27654: Fri May  9 10:32:06 2025
  write: IOPS=357, BW=1430KiB/s (1464kB/s)(10.0MiB/7160msec)
    slat (usec): min=23, max=440, avg=86.15, stdev=48.44
    clat (usec): min=9956, max=58171, avg=44428.86, stdev=5481.51
     lat (usec): min=10069, max=58295, avg=44516.81, stdev=5489.75
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[34341], 10.00th=[36439], 20.00th=[40633],
     | 30.00th=[43779], 40.00th=[45351], 50.00th=[46400], 60.00th=[46924],
     | 70.00th=[47449], 80.00th=[48497], 90.00th=[49546], 95.00th=[50070],
     | 99.00th=[50070], 99.50th=[51643], 99.90th=[53740], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1280, max= 1704, per=24.91%, avg=1424.86, stdev=137.03, samples=14
   iops        : min=  320, max=  426, avg=356.21, stdev=34.26, samples=14
  lat (msec)   : 10=0.04%, 20=0.62%, 50=96.52%, 100=2.81%
  cpu          : usr=0.54%, sys=3.28%, ctx=1839, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=27655: Fri May  9 10:32:06 2025
  write: IOPS=357, BW=1432KiB/s (1466kB/s)(10.0MiB/7151msec)
    slat (usec): min=23, max=488, avg=89.38, stdev=48.87
    clat (usec): min=9605, max=82246, avg=44459.31, stdev=5857.55
     lat (usec): min=9718, max=82322, avg=44550.62, stdev=5861.67
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[34341], 10.00th=[36439], 20.00th=[40633],
     | 30.00th=[43779], 40.00th=[45351], 50.00th=[46400], 60.00th=[46924],
     | 70.00th=[47449], 80.00th=[48497], 90.00th=[49546], 95.00th=[50070],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[78119], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1280, max= 1688, per=25.04%, avg=1432.57, stdev=131.17, samples=14
   iops        : min=  320, max=  422, avg=358.14, stdev=32.79, samples=14
  lat (msec)   : 10=0.04%, 20=0.66%, 50=96.09%, 100=3.20%
  cpu          : usr=0.50%, sys=3.64%, ctx=1954, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5721KiB/s (5858kB/s), 1430KiB/s-1437KiB/s (1464kB/s-1472kB/s), io=40.0MiB (41.9MB), run=7124-7160msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=27670: Fri May  9 10:32:10 2025
   read: IOPS=422, BW=1689KiB/s (1730kB/s)(5048KiB/2988msec)
    slat (usec): min=703, max=3389, avg=1329.55, stdev=350.40
    clat (usec): min=9686, max=42097, avg=17441.88, stdev=3939.75
     lat (usec): min=11037, max=44755, avg=18774.27, stdev=4102.55
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12780], 10.00th=[13566], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[16188], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21627], 95.00th=[24511],
     | 99.00th=[34341], 99.50th=[36963], 99.90th=[40109], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1192, max= 1792, per=24.25%, avg=1627.20, stdev=250.67, samples=5
   iops        : min=  298, max=  448, avg=406.80, stdev=62.67, samples=5
  write: IOPS=434, BW=1738KiB/s (1779kB/s)(5192KiB/2988msec)
    slat (usec): min=13, max=1069, avg=66.79, stdev=47.09
    clat (usec): min=11238, max=42025, avg=18061.02, stdev=3816.13
     lat (usec): min=11292, max=42100, avg=18129.20, stdev=3817.79
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14222], 10.00th=[14746], 20.00th=[15533],
     | 30.00th=[16188], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[21365], 95.00th=[25297],
     | 99.00th=[34341], 99.50th=[37487], 99.90th=[40109], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1144, max= 1952, per=24.57%, avg=1716.80, stdev=326.56, samples=5
   iops        : min=  286, max=  488, avg=429.20, stdev=81.64, samples=5
  lat (msec)   : 10=0.04%, 20=84.10%, 50=15.86%
  cpu          : usr=1.54%, sys=10.08%, ctx=2313, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=27671: Fri May  9 10:32:10 2025
   read: IOPS=418, BW=1673KiB/s (1713kB/s)(4976KiB/2975msec)
    slat (usec): min=34, max=3455, avg=1326.68, stdev=389.48
    clat (usec): min=3382, max=39605, avg=17594.44, stdev=4159.86
     lat (usec): min=3558, max=42326, avg=18924.22, stdev=4341.27
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12780], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[16188], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[21627], 95.00th=[25560],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[38536], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1120, max= 1848, per=24.29%, avg=1630.40, stdev=290.18, samples=5
   iops        : min=  280, max=  462, avg=407.60, stdev=72.55, samples=5
  write: IOPS=442, BW=1769KiB/s (1812kB/s)(5264KiB/2975msec)
    slat (usec): min=12, max=650, avg=66.54, stdev=44.23
    clat (usec): min=3568, max=40400, avg=18101.40, stdev=3807.16
     lat (usec): min=3638, max=40441, avg=18169.43, stdev=3810.93
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14091], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[21103], 95.00th=[26084],
     | 99.00th=[33424], 99.50th=[35914], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1392, max= 1880, per=25.08%, avg=1752.00, stdev=203.17, samples=5
   iops        : min=  348, max=  470, avg=438.00, stdev=50.79, samples=5
  lat (msec)   : 4=0.08%, 10=0.20%, 20=82.81%, 50=16.91%
  cpu          : usr=1.38%, sys=10.22%, ctx=2351, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=27672: Fri May  9 10:32:10 2025
   read: IOPS=417, BW=1672KiB/s (1712kB/s)(4984KiB/2981msec)
    slat (usec): min=706, max=3354, avg=1348.56, stdev=381.47
    clat (usec): min=1522, max=40267, avg=17255.39, stdev=3987.22
     lat (usec): min=3540, max=42547, avg=18606.68, stdev=4168.44
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12649], 10.00th=[13435], 20.00th=[14484],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[21627], 95.00th=[24249],
     | 99.00th=[33817], 99.50th=[35914], 99.90th=[39584], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1208, max= 1832, per=24.32%, avg=1632.00, stdev=250.57, samples=5
   iops        : min=  302, max=  458, avg=408.00, stdev=62.64, samples=5
  write: IOPS=440, BW=1763KiB/s (1805kB/s)(5256KiB/2981msec)
    slat (usec): min=12, max=966, avg=68.87, stdev=51.91
    clat (usec): min=3354, max=37788, avg=18239.64, stdev=3950.00
     lat (usec): min=3480, max=37867, avg=18309.95, stdev=3956.43
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14222], 10.00th=[14746], 20.00th=[15533],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19792], 90.00th=[21890], 95.00th=[26084],
     | 99.00th=[34341], 99.50th=[36963], 99.90th=[37487], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1144, max= 1944, per=24.59%, avg=1718.40, stdev=331.08, samples=5
   iops        : min=  286, max=  486, avg=429.60, stdev=82.77, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.20%, 20=82.93%, 50=16.80%
  cpu          : usr=1.34%, sys=10.37%, ctx=2381, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=27673: Fri May  9 10:32:10 2025
   read: IOPS=423, BW=1692KiB/s (1733kB/s)(5060KiB/2990msec)
    slat (usec): min=677, max=3876, avg=1326.52, stdev=361.48
    clat (usec): min=9370, max=44264, avg=17349.94, stdev=3913.13
     lat (usec): min=10981, max=46949, avg=18679.46, stdev=4081.88
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12649], 10.00th=[13566], 20.00th=[14484],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[20841], 95.00th=[24773],
     | 99.00th=[33817], 99.50th=[36439], 99.90th=[40109], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min= 1144, max= 1808, per=24.41%, avg=1638.40, stdev=277.96, samples=5
   iops        : min=  286, max=  452, avg=409.60, stdev=69.49, samples=5
  write: IOPS=433, BW=1732KiB/s (1774kB/s)(5180KiB/2990msec)
    slat (usec): min=12, max=266, avg=64.36, stdev=34.38
    clat (usec): min=11075, max=42445, avg=18127.55, stdev=3746.64
     lat (usec): min=11109, max=42511, avg=18193.51, stdev=3750.81
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[13960], 10.00th=[14746], 20.00th=[15664],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19792], 90.00th=[21627], 95.00th=[25297],
     | 99.00th=[33162], 99.50th=[36963], 99.90th=[41681], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1160, max= 1864, per=24.25%, avg=1694.40, stdev=300.90, samples=5
   iops        : min=  290, max=  466, avg=423.60, stdev=75.22, samples=5
  lat (msec)   : 10=0.08%, 20=83.71%, 50=16.21%
  cpu          : usr=1.07%, sys=10.54%, ctx=2397, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6712KiB/s (6873kB/s), 1672KiB/s-1692KiB/s (1712kB/s-1733kB/s), io=19.6MiB (20.5MB), run=2975-2990msec
  WRITE: bw=6987KiB/s (7155kB/s), 1732KiB/s-1769KiB/s (1774kB/s-1812kB/s), io=20.4MiB (21.4MB), run=2975-2990msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=27679: Fri May  9 10:32:12 2025
   read: IOPS=6, BW=272MiB/s (285MB/s)(40.0MiB/147msec)
    slat (nsec): min=5880.1k, max=5880.1k, avg=5880146.00, stdev= 0.00
    clat (nsec): min=135148k, max=135148k, avg=135147857.00, stdev= 0.00
     lat (nsec): min=141041k, max=141041k, avg=141040615.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  136], 10.00th=[  136], 20.00th=[  136],
     | 30.00th=[  136], 40.00th=[  136], 50.00th=[  136], 60.00th=[  136],
     | 70.00th=[  136], 80.00th=[  136], 90.00th=[  136], 95.00th=[  136],
     | 99.00th=[  136], 99.50th=[  136], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.11%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=40.0MiB (41.9MB), run=147-147msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout