== sanity test 398c: run fio to test AIO ================= 03:52:02 (1746085922)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.104234 s, 402 MB/s
osc.lustre-OST0000-osc-ffff880098ad5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=22354: Thu May  1 03:52:07 2025
  write: IOPS=775, BW=3103KiB/s (3178kB/s)(10.0MiB/3300msec)
    slat (usec): min=11, max=492, avg=43.17, stdev=27.90
    clat (usec): min=1744, max=35970, avg=20518.30, stdev=2958.89
     lat (usec): min=1804, max=36010, avg=20562.53, stdev=2961.58
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[15533], 10.00th=[17433], 20.00th=[18482],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21627], 80.00th=[22676], 90.00th=[23987], 95.00th=[25297],
     | 99.00th=[28443], 99.50th=[28705], 99.90th=[30016], 99.95th=[33817],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2976, max= 3321, per=25.07%, avg=3105.50, stdev=145.40, samples=6
   iops        : min=  744, max=  830, avg=776.33, stdev=36.27, samples=6
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.04%, 20=47.15%, 50=52.73%
  cpu          : usr=0.55%, sys=3.79%, ctx=1158, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22355: Thu May  1 03:52:07 2025
  write: IOPS=778, BW=3113KiB/s (3188kB/s)(10.0MiB/3289msec)
    slat (usec): min=12, max=335, avg=48.40, stdev=25.92
    clat (usec): min=2460, max=36031, avg=20448.14, stdev=3135.33
     lat (usec): min=2510, max=36059, avg=20497.71, stdev=3136.20
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[15401], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[19006], 40.00th=[19792], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23987], 95.00th=[25560],
     | 99.00th=[28443], 99.50th=[29754], 99.90th=[34341], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2952, max= 3417, per=25.24%, avg=3125.50, stdev=171.07, samples=6
   iops        : min=  738, max=  854, avg=781.33, stdev=42.68, samples=6
  lat (msec)   : 4=0.12%, 10=0.55%, 20=44.88%, 50=54.45%
  cpu          : usr=0.55%, sys=4.53%, ctx=1682, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22356: Thu May  1 03:52:07 2025
  write: IOPS=774, BW=3098KiB/s (3173kB/s)(10.0MiB/3305msec)
    slat (usec): min=11, max=610, avg=42.97, stdev=30.28
    clat (usec): min=11687, max=37115, avg=20531.75, stdev=2825.04
     lat (usec): min=11755, max=37146, avg=20575.80, stdev=2828.24
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15926], 10.00th=[17433], 20.00th=[18744],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22676], 90.00th=[23987], 95.00th=[25297],
     | 99.00th=[27395], 99.50th=[29492], 99.90th=[31327], 99.95th=[33162],
     | 99.99th=[36963]
   bw (  KiB/s): min= 2944, max= 3328, per=25.04%, avg=3101.33, stdev=159.89, samples=6
   iops        : min=  736, max=  832, avg=775.33, stdev=39.97, samples=6
  lat (msec)   : 20=46.48%, 50=53.52%
  cpu          : usr=0.82%, sys=3.54%, ctx=1099, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22357: Thu May  1 03:52:07 2025
  write: IOPS=774, BW=3096KiB/s (3171kB/s)(10.0MiB/3307msec)
    slat (usec): min=11, max=352, avg=45.68, stdev=26.03
    clat (usec): min=4017, max=30542, avg=20517.56, stdev=2859.62
     lat (usec): min=4060, max=30577, avg=20564.29, stdev=2863.57
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15795], 10.00th=[17433], 20.00th=[18482],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23987], 95.00th=[25560],
     | 99.00th=[28181], 99.50th=[28443], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 2944, max= 3217, per=24.95%, avg=3089.50, stdev=121.47, samples=6
   iops        : min=  736, max=  804, avg=772.33, stdev=30.32, samples=6
  lat (msec)   : 10=0.12%, 20=45.23%, 50=54.65%
  cpu          : usr=0.88%, sys=3.75%, ctx=1417, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.1MiB/s (12.7MB/s), 3096KiB/s-3113KiB/s (3171kB/s-3188kB/s), io=40.0MiB (41.9MB), run=3289-3307msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=22370: Thu May  1 03:52:11 2025
   read: IOPS=443, BW=1775KiB/s (1818kB/s)(5048KiB/2844msec)
    slat (usec): min=606, max=2960, avg=1149.45, stdev=279.45
    clat (usec): min=2148, max=30838, avg=16889.61, stdev=3287.42
     lat (usec): min=3299, max=32683, avg=18040.79, stdev=3385.63
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12780], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19530], 90.00th=[21627], 95.00th=[23200],
     | 99.00th=[26608], 99.50th=[27395], 99.90th=[29754], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1480, max= 1944, per=25.36%, avg=1788.80, stdev=196.90, samples=5
   iops        : min=  370, max=  486, avg=447.20, stdev=49.23, samples=5
  write: IOPS=456, BW=1826KiB/s (1869kB/s)(5192KiB/2844msec)
    slat (usec): min=13, max=1001, avg=44.69, stdev=38.41
    clat (usec): min=11068, max=29955, avg=17156.49, stdev=3029.43
     lat (usec): min=11124, max=29979, avg=17202.18, stdev=3031.12
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13435], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21103], 95.00th=[23462],
     | 99.00th=[27132], 99.50th=[28181], 99.90th=[29492], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1456, max= 2072, per=25.54%, avg=1875.20, stdev=253.34, samples=5
   iops        : min=  364, max=  518, avg=468.80, stdev=63.33, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=83.40%, 50=16.41%
  cpu          : usr=0.91%, sys=6.86%, ctx=2439, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22371: Thu May  1 03:52:11 2025
   read: IOPS=439, BW=1759KiB/s (1801kB/s)(4976KiB/2829msec)
    slat (usec): min=675, max=4194, avg=1136.82, stdev=299.77
    clat (usec): min=3022, max=32623, avg=16931.40, stdev=3354.76
     lat (usec): min=4186, max=34245, avg=18069.92, stdev=3453.68
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12518], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21365], 95.00th=[23462],
     | 99.00th=[27395], 99.50th=[28705], 99.90th=[31327], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1472, max= 1936, per=25.25%, avg=1780.80, stdev=195.19, samples=5
   iops        : min=  368, max=  484, avg=445.20, stdev=48.80, samples=5
  write: IOPS=465, BW=1861KiB/s (1905kB/s)(5264KiB/2829msec)
    slat (usec): min=11, max=345, avg=44.80, stdev=26.24
    clat (usec): min=8179, max=37413, avg=17062.21, stdev=3113.83
     lat (usec): min=8204, max=37473, avg=17108.08, stdev=3116.08
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13435], 10.00th=[13829], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19006], 90.00th=[21365], 95.00th=[23462],
     | 99.00th=[26608], 99.50th=[27132], 99.90th=[34866], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1480, max= 2104, per=26.04%, avg=1912.00, stdev=254.43, samples=5
   iops        : min=  370, max=  526, avg=478.00, stdev=63.61, samples=5
  lat (msec)   : 4=0.12%, 10=0.27%, 20=84.26%, 50=15.35%
  cpu          : usr=1.06%, sys=6.82%, ctx=2537, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22372: Thu May  1 03:52:11 2025
   read: IOPS=437, BW=1752KiB/s (1794kB/s)(4984KiB/2845msec)
    slat (usec): min=654, max=2823, avg=1136.93, stdev=274.74
    clat (usec): min=4011, max=37886, avg=16881.04, stdev=3379.40
     lat (usec): min=5185, max=40165, avg=18019.56, stdev=3483.32
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12387], 10.00th=[13042], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21365], 95.00th=[23200],
     | 99.00th=[26870], 99.50th=[27395], 99.90th=[36439], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1384, max= 2040, per=25.38%, avg=1790.40, stdev=249.25, samples=5
   iops        : min=  346, max=  510, avg=447.60, stdev=62.31, samples=5
  write: IOPS=461, BW=1847KiB/s (1892kB/s)(5256KiB/2845msec)
    slat (usec): min=11, max=1568, avg=44.94, stdev=47.93
    clat (usec): min=10743, max=34879, avg=17184.81, stdev=3061.86
     lat (usec): min=10778, max=34940, avg=17230.79, stdev=3066.36
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13566], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[21365], 95.00th=[23462],
     | 99.00th=[26084], 99.50th=[26608], 99.90th=[33817], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1576, max= 2056, per=25.56%, avg=1876.80, stdev=201.80, samples=5
   iops        : min=  394, max=  514, avg=469.20, stdev=50.45, samples=5
  lat (msec)   : 10=0.12%, 20=83.48%, 50=16.41%
  cpu          : usr=0.70%, sys=6.93%, ctx=2508, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22373: Thu May  1 03:52:11 2025
   read: IOPS=449, BW=1796KiB/s (1839kB/s)(5060KiB/2817msec)
    slat (usec): min=10, max=3559, avg=1140.09, stdev=304.40
    clat (usec): min=2052, max=30196, avg=16805.95, stdev=3351.41
     lat (usec): min=2081, max=32110, avg=17947.55, stdev=3456.00
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[12387], 10.00th=[13173], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21103], 95.00th=[23200],
     | 99.00th=[27132], 99.50th=[28181], 99.90th=[29492], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1512, max= 1984, per=26.00%, avg=1833.60, stdev=190.19, samples=5
   iops        : min=  378, max=  496, avg=458.40, stdev=47.55, samples=5
  write: IOPS=459, BW=1839KiB/s (1883kB/s)(5180KiB/2817msec)
    slat (usec): min=12, max=398, avg=44.52, stdev=25.28
    clat (usec): min=2086, max=34006, avg=17137.53, stdev=3093.18
     lat (usec): min=2172, max=34056, avg=17183.24, stdev=3094.79
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[13435], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21103], 95.00th=[23200],
     | 99.00th=[25822], 99.50th=[26346], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1424, max= 2072, per=25.41%, avg=1865.60, stdev=256.59, samples=5
   iops        : min=  356, max=  518, avg=466.40, stdev=64.15, samples=5
  lat (msec)   : 4=0.23%, 10=0.39%, 20=83.24%, 50=16.13%
  cpu          : usr=1.17%, sys=6.68%, ctx=2479, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7054KiB/s (7223kB/s), 1752KiB/s-1796KiB/s (1794kB/s-1839kB/s), io=19.6MiB (20.5MB), run=2817-2845msec
  WRITE: bw=7343KiB/s (7520kB/s), 1826KiB/s-1861KiB/s (1869kB/s-1905kB/s), io=20.4MiB (21.4MB), run=2817-2845msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=22379: Thu May  1 03:52:12 2025
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=6951.9k, max=6951.9k, avg=6951917.00, stdev= 0.00
    clat (nsec): min=63921k, max=63921k, avg=63920708.00, stdev= 0.00
     lat (nsec): min=70898k, max=70898k, avg=70897859.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[63701],  5.00th=[63701], 10.00th=[63701], 20.00th=[63701],
     | 30.00th=[63701], 40.00th=[63701], 50.00th=[63701], 60.00th=[63701],
     | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701],
     | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.33%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout