== sanity test 398c: run fio to test AIO ================= 13:57:52 (1736794672)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.134154 s, 313 MB/s
osc.lustre-OST0000-osc-ffff88008c2a5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23831: Mon Jan 13 13:58:01 2025
  write: IOPS=375, BW=1503KiB/s (1539kB/s)(10.0MiB/6814msec)
    slat (usec): min=24, max=546, avg=68.71, stdev=42.98
    clat (usec): min=26921, max=68973, avg=42289.27, stdev=4333.51
     lat (usec): min=27061, max=69085, avg=42360.02, stdev=4333.78
    clat percentiles (usec):
     |  1.00th=[31327],  5.00th=[34866], 10.00th=[36439], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41157], 50.00th=[42206], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[45876], 90.00th=[47449], 95.00th=[47973],
     | 99.00th=[50070], 99.50th=[59507], 99.90th=[64750], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1392, max= 1656, per=24.94%, avg=1499.15, stdev=89.97, samples=13
   iops        : min=  348, max=  414, avg=374.77, stdev=22.51, samples=13
  lat (msec)   : 50=99.10%, 100=0.90%
  cpu          : usr=0.35%, sys=3.21%, ctx=853, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23832: Mon Jan 13 13:58:01 2025
  write: IOPS=377, BW=1508KiB/s (1545kB/s)(10.0MiB/6789msec)
    slat (usec): min=19, max=706, avg=83.76, stdev=47.44
    clat (usec): min=8593, max=76206, avg=42200.84, stdev=4662.20
     lat (usec): min=8713, max=76295, avg=42286.58, stdev=4664.04
    clat percentiles (usec):
     |  1.00th=[31065],  5.00th=[35390], 10.00th=[37487], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41157], 50.00th=[42206], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[45876], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[53216], 99.50th=[56361], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1320, max= 1640, per=25.10%, avg=1509.00, stdev=92.85, samples=13
   iops        : min=  330, max=  410, avg=377.23, stdev=23.23, samples=13
  lat (msec)   : 10=0.16%, 20=0.08%, 50=98.55%, 100=1.21%
  cpu          : usr=0.71%, sys=3.65%, ctx=1446, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23833: Mon Jan 13 13:58:01 2025
  write: IOPS=376, BW=1504KiB/s (1540kB/s)(10.0MiB/6807msec)
    slat (usec): min=21, max=724, avg=87.00, stdev=46.18
    clat (usec): min=9239, max=75698, avg=42292.95, stdev=4422.64
     lat (usec): min=9345, max=75849, avg=42382.14, stdev=4423.06
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[35390], 10.00th=[37487], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41157], 50.00th=[42206], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[45876], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[52167], 99.50th=[57410], 99.90th=[74974], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1376, max= 1656, per=24.97%, avg=1501.00, stdev=82.25, samples=13
   iops        : min=  344, max=  414, avg=375.23, stdev=20.59, samples=13
  lat (msec)   : 10=0.08%, 50=98.67%, 100=1.25%
  cpu          : usr=0.53%, sys=4.04%, ctx=1559, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23834: Mon Jan 13 13:58:01 2025
  write: IOPS=375, BW=1504KiB/s (1540kB/s)(10.0MiB/6809msec)
    slat (usec): min=14, max=905, avg=81.14, stdev=49.32
    clat (usec): min=28210, max=60420, avg=42284.21, stdev=4175.93
     lat (usec): min=28309, max=60634, avg=42367.33, stdev=4177.42
    clat percentiles (usec):
     |  1.00th=[31065],  5.00th=[35390], 10.00th=[37487], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41157], 50.00th=[42206], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[45876], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[51643], 99.50th=[56886], 99.90th=[59507], 99.95th=[60031],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1376, max= 1664, per=24.94%, avg=1499.15, stdev=81.93, samples=13
   iops        : min=  344, max=  416, avg=374.77, stdev=20.51, samples=13
  lat (msec)   : 50=98.83%, 100=1.17%
  cpu          : usr=0.68%, sys=3.41%, ctx=1472, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6011KiB/s (6155kB/s), 1503KiB/s-1508KiB/s (1539kB/s-1545kB/s), io=40.0MiB (41.9MB), run=6789-6814msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23849: Mon Jan 13 13:58:05 2025
   read: IOPS=411, BW=1645KiB/s (1685kB/s)(5048KiB/3068msec)
    slat (usec): min=660, max=4406, avg=1226.50, stdev=353.07
    clat (usec): min=9189, max=38215, avg=17968.24, stdev=3496.95
     lat (usec): min=10337, max=40424, avg=19196.52, stdev=3654.07
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16712], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[22152], 95.00th=[24511],
     | 99.00th=[30016], 99.50th=[32637], 99.90th=[35390], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1232, max= 1824, per=25.05%, avg=1637.33, stdev=209.45, samples=6
   iops        : min=  308, max=  456, avg=409.33, stdev=52.36, samples=6
  write: IOPS=423, BW=1692KiB/s (1733kB/s)(5192KiB/3068msec)
    slat (usec): min=13, max=284, avg=46.45, stdev=26.71
    clat (usec): min=4988, max=35954, avg=18691.56, stdev=3320.25
     lat (usec): min=5054, max=35995, avg=18739.14, stdev=3327.20
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14746], 10.00th=[15401], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17695], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[22152], 95.00th=[24511],
     | 99.00th=[32375], 99.50th=[33817], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1200, max= 1904, per=24.86%, avg=1692.00, stdev=264.45, samples=6
   iops        : min=  300, max=  476, avg=423.00, stdev=66.11, samples=6
  lat (msec)   : 10=0.12%, 20=77.03%, 50=22.85%
  cpu          : usr=0.55%, sys=7.14%, ctx=2816, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23850: Mon Jan 13 13:58:05 2025
   read: IOPS=406, BW=1627KiB/s (1666kB/s)(4976KiB/3059msec)
    slat (usec): min=584, max=4594, avg=1207.18, stdev=341.20
    clat (usec): min=10540, max=36885, avg=17918.86, stdev=3585.41
     lat (usec): min=11591, max=38738, avg=19127.92, stdev=3736.95
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13304], 10.00th=[14091], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[22152], 95.00th=[24773],
     | 99.00th=[29754], 99.50th=[32637], 99.90th=[35914], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1112, max= 1888, per=24.87%, avg=1625.33, stdev=265.21, samples=6
   iops        : min=  278, max=  472, avg=406.33, stdev=66.30, samples=6
  write: IOPS=430, BW=1721KiB/s (1762kB/s)(5264KiB/3059msec)
    slat (usec): min=12, max=355, avg=45.65, stdev=27.83
    clat (usec): min=2970, max=35451, avg=18640.86, stdev=3290.15
     lat (usec): min=3000, max=35515, avg=18687.57, stdev=3295.63
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[14615], 10.00th=[15401], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19268], 80.00th=[20579], 90.00th=[22152], 95.00th=[24773],
     | 99.00th=[31327], 99.50th=[33817], 99.90th=[34341], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1344, max= 1880, per=25.33%, avg=1724.00, stdev=203.44, samples=6
   iops        : min=  336, max=  470, avg=431.00, stdev=50.86, samples=6
  lat (msec)   : 4=0.04%, 20=77.46%, 50=22.50%
  cpu          : usr=0.88%, sys=6.74%, ctx=2628, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23851: Mon Jan 13 13:58:05 2025
   read: IOPS=409, BW=1637KiB/s (1676kB/s)(4984KiB/3045msec)
    slat (usec): min=24, max=4186, avg=1228.51, stdev=372.07
    clat (usec): min=4060, max=36033, avg=17927.19, stdev=3467.44
     lat (usec): min=4106, max=37112, avg=19157.60, stdev=3624.00
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13304], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[21890], 95.00th=[24511],
     | 99.00th=[29492], 99.50th=[31589], 99.90th=[33817], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1312, max= 1816, per=25.09%, avg=1640.00, stdev=200.29, samples=6
   iops        : min=  328, max=  454, avg=410.00, stdev=50.07, samples=6
  write: IOPS=431, BW=1726KiB/s (1768kB/s)(5256KiB/3045msec)
    slat (usec): min=12, max=852, avg=47.02, stdev=34.74
    clat (usec): min=10443, max=35452, avg=18761.50, stdev=3462.36
     lat (usec): min=10473, max=35571, avg=18809.56, stdev=3468.82
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14746], 10.00th=[15401], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20317], 90.00th=[22152], 95.00th=[25822],
     | 99.00th=[32113], 99.50th=[33817], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1312, max= 1952, per=25.33%, avg=1724.00, stdev=233.99, samples=6
   iops        : min=  328, max=  488, avg=431.00, stdev=58.50, samples=6
  lat (msec)   : 10=0.12%, 20=77.70%, 50=22.19%
  cpu          : usr=0.69%, sys=6.87%, ctx=2723, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23852: Mon Jan 13 13:58:05 2025
   read: IOPS=412, BW=1648KiB/s (1688kB/s)(5060KiB/3070msec)
    slat (usec): min=643, max=3052, avg=1223.80, stdev=320.95
    clat (usec): min=4410, max=39564, avg=17920.09, stdev=3561.63
     lat (usec): min=5579, max=41584, avg=19145.72, stdev=3709.43
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13304], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[21627], 95.00th=[24511],
     | 99.00th=[29754], 99.50th=[32113], 99.90th=[34341], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1176, max= 1816, per=25.13%, avg=1642.67, stdev=236.92, samples=6
   iops        : min=  294, max=  454, avg=410.67, stdev=59.23, samples=6
  write: IOPS=421, BW=1687KiB/s (1728kB/s)(5180KiB/3070msec)
    slat (usec): min=13, max=1058, avg=45.48, stdev=38.34
    clat (usec): min=5585, max=35988, avg=18655.35, stdev=3399.66
     lat (usec): min=5621, max=36046, avg=18701.94, stdev=3406.60
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14746], 10.00th=[15533], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[20055], 90.00th=[21890], 95.00th=[25035],
     | 99.00th=[32637], 99.50th=[33817], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1184, max= 1928, per=24.63%, avg=1676.00, stdev=262.31, samples=6
   iops        : min=  296, max=  482, avg=419.00, stdev=65.58, samples=6
  lat (msec)   : 10=0.39%, 20=78.79%, 50=20.82%
  cpu          : usr=0.62%, sys=6.84%, ctx=2669, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6537KiB/s (6694kB/s), 1627KiB/s-1648KiB/s (1666kB/s-1688kB/s), io=19.6MiB (20.5MB), run=3045-3070msec
  WRITE: bw=6805KiB/s (6969kB/s), 1687KiB/s-1726KiB/s (1728kB/s-1768kB/s), io=20.4MiB (21.4MB), run=3045-3070msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23859: Mon Jan 13 13:58:06 2025
   read: IOPS=6, BW=272MiB/s (285MB/s)(40.0MiB/147msec)
    slat (nsec): min=6002.9k, max=6002.9k, avg=6002926.00, stdev= 0.00
    clat (nsec): min=134259k, max=134259k, avg=134259031.00, stdev= 0.00
     lat (nsec): min=140271k, max=140271k, avg=140271476.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  136], 10.00th=[  136], 20.00th=[  136],
     | 30.00th=[  136], 40.00th=[  136], 50.00th=[  136], 60.00th=[  136],
     | 70.00th=[  136], 80.00th=[  136], 90.00th=[  136], 95.00th=[  136],
     | 99.00th=[  136], 99.50th=[  136], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.42%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=40.0MiB (41.9MB), run=147-147msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout