== sanity test 398c: run fio to test AIO ================= 21:26:02 (1743211562)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.135194 s, 310 MB/s
osc.lustre-OST0000-osc-ffff8800a91c6000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=13118: Fri Mar 28 21:27:03 2025
  write: IOPS=42, BW=170KiB/s (175kB/s)(10.0MiB/60070msec)
    slat (usec): min=25, max=333, avg=91.84, stdev=39.13
    clat (msec): min=120, max=569, avg=375.02, stdev=66.11
     lat (msec): min=120, max=569, avg=375.11, stdev=66.11
    clat percentiles (msec):
     |  1.00th=[  171],  5.00th=[  266], 10.00th=[  305], 20.00th=[  330],
     | 30.00th=[  347], 40.00th=[  363], 50.00th=[  380], 60.00th=[  393],
     | 70.00th=[  405], 80.00th=[  426], 90.00th=[  456], 95.00th=[  477],
     | 99.00th=[  527], 99.50th=[  542], 99.90th=[  567], 99.95th=[  567],
     | 99.99th=[  567]
   bw (  KiB/s): min=  127, max=  383, per=24.89%, avg=169.47, stdev=33.47, samples=120
   iops        : min=   31, max=   95, avg=42.23, stdev= 8.34, samples=120
  lat (msec)   : 250=3.91%, 500=93.63%, 750=2.46%
  cpu          : usr=0.10%, sys=0.49%, ctx=1702, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13119: Fri Mar 28 21:27:03 2025
  write: IOPS=42, BW=170KiB/s (175kB/s)(10.0MiB/60070msec)
    slat (usec): min=14, max=635, avg=89.68, stdev=40.26
    clat (msec): min=97, max=578, avg=375.13, stdev=66.62
     lat (msec): min=97, max=578, avg=375.22, stdev=66.62
    clat percentiles (msec):
     |  1.00th=[  138],  5.00th=[  262], 10.00th=[  300], 20.00th=[  330],
     | 30.00th=[  351], 40.00th=[  363], 50.00th=[  376], 60.00th=[  393],
     | 70.00th=[  405], 80.00th=[  426], 90.00th=[  456], 95.00th=[  481],
     | 99.00th=[  527], 99.50th=[  531], 99.90th=[  567], 99.95th=[  567],
     | 99.99th=[  575]
   bw (  KiB/s): min=  112, max=  368, per=24.89%, avg=169.53, stdev=35.19, samples=120
   iops        : min=   28, max=   92, avg=42.28, stdev= 8.81, samples=120
  lat (msec)   : 100=0.04%, 250=3.98%, 500=93.36%, 750=2.62%
  cpu          : usr=0.07%, sys=0.52%, ctx=1614, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13120: Fri Mar 28 21:27:03 2025
  write: IOPS=42, BW=171KiB/s (175kB/s)(10.0MiB/59983msec)
    slat (usec): min=27, max=517, avg=90.97, stdev=40.07
    clat (msec): min=41, max=570, avg=374.58, stdev=68.98
     lat (msec): min=41, max=570, avg=374.67, stdev=68.98
    clat percentiles (msec):
     |  1.00th=[  138],  5.00th=[  259], 10.00th=[  300], 20.00th=[  330],
     | 30.00th=[  347], 40.00th=[  363], 50.00th=[  380], 60.00th=[  393],
     | 70.00th=[  405], 80.00th=[  426], 90.00th=[  456], 95.00th=[  477],
     | 99.00th=[  523], 99.50th=[  535], 99.90th=[  558], 99.95th=[  567],
     | 99.99th=[  567]
   bw (  KiB/s): min=  119, max=  376, per=24.96%, avg=170.00, stdev=34.79, samples=119
   iops        : min=   29, max=   94, avg=42.39, stdev= 8.69, samples=119
  lat (msec)   : 50=0.16%, 100=0.55%, 250=3.59%, 500=93.44%, 750=2.27%
  cpu          : usr=0.13%, sys=0.47%, ctx=1699, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13121: Fri Mar 28 21:27:03 2025
  write: IOPS=42, BW=170KiB/s (175kB/s)(10.0MiB/60088msec)
    slat (usec): min=21, max=506, avg=89.98, stdev=38.52
    clat (msec): min=99, max=578, avg=375.03, stdev=66.38
     lat (msec): min=99, max=578, avg=375.12, stdev=66.38
    clat percentiles (msec):
     |  1.00th=[  171],  5.00th=[  266], 10.00th=[  305], 20.00th=[  330],
     | 30.00th=[  347], 40.00th=[  363], 50.00th=[  376], 60.00th=[  393],
     | 70.00th=[  405], 80.00th=[  426], 90.00th=[  456], 95.00th=[  477],
     | 99.00th=[  531], 99.50th=[  542], 99.90th=[  567], 99.95th=[  567],
     | 99.99th=[  584]
   bw (  KiB/s): min=  104, max=  384, per=24.89%, avg=169.49, stdev=33.82, samples=120
   iops        : min=   26, max=   96, avg=42.24, stdev= 8.45, samples=120
  lat (msec)   : 100=0.62%, 250=3.20%, 500=93.52%, 750=2.66%
  cpu          : usr=0.10%, sys=0.48%, ctx=1655, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=682KiB/s (698kB/s), 170KiB/s-171KiB/s (175kB/s-175kB/s), io=40.0MiB (41.9MB), run=59983-60088msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=13178: Fri Mar 28 21:27:40 2025
   read: IOPS=35, BW=144KiB/s (147kB/s)(5048KiB/35124msec)
    slat (usec): min=1232, max=4460, avg=2346.59, stdev=417.90
    clat (msec): min=17, max=346, avg=143.78, stdev=27.39
     lat (msec): min=20, max=348, avg=146.13, stdev=27.43
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  106], 10.00th=[  114], 20.00th=[  125],
     | 30.00th=[  131], 40.00th=[  138], 50.00th=[  144], 60.00th=[  150],
     | 70.00th=[  157], 80.00th=[  165], 90.00th=[  176], 95.00th=[  186],
     | 99.00th=[  209], 99.50th=[  218], 99.90th=[  313], 99.95th=[  347],
     | 99.99th=[  347]
   bw (  KiB/s): min=   56, max=  248, per=25.16%, avg=143.19, stdev=38.02, samples=70
   iops        : min=   14, max=   62, avg=35.70, stdev= 9.50, samples=70
  write: IOPS=36, BW=148KiB/s (151kB/s)(5192KiB/35124msec)
    slat (usec): min=23, max=2429, avg=96.44, stdev=75.73
    clat (msec): min=96, max=489, avg=290.19, stdev=67.67
     lat (msec): min=96, max=489, avg=290.29, stdev=67.68
    clat percentiles (msec):
     |  1.00th=[  153],  5.00th=[  184], 10.00th=[  205], 20.00th=[  232],
     | 30.00th=[  251], 40.00th=[  268], 50.00th=[  288], 60.00th=[  305],
     | 70.00th=[  326], 80.00th=[  347], 90.00th=[  380], 95.00th=[  409],
     | 99.00th=[  451], 99.50th=[  460], 99.90th=[  481], 99.95th=[  489],
     | 99.99th=[  489]
   bw (  KiB/s): min=   79, max=  216, per=24.86%, avg=147.41, stdev=28.98, samples=70
   iops        : min=   19, max=   54, avg=36.76, stdev= 7.25, samples=70
  lat (msec)   : 20=0.08%, 50=0.31%, 100=1.29%, 250=62.23%, 500=36.09%
  cpu          : usr=0.19%, sys=1.33%, ctx=2909, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13179: Fri Mar 28 21:27:40 2025
   read: IOPS=35, BW=141KiB/s (145kB/s)(4976KiB/35221msec)
    slat (usec): min=1348, max=5499, avg=2351.75, stdev=432.37
    clat (msec): min=2, max=219, avg=144.30, stdev=27.27
     lat (msec): min=4, max=221, avg=146.66, stdev=27.32
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[  105], 10.00th=[  115], 20.00th=[  126],
     | 30.00th=[  133], 40.00th=[  138], 50.00th=[  144], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  165], 90.00th=[  178], 95.00th=[  188],
     | 99.00th=[  205], 99.50th=[  209], 99.90th=[  218], 99.95th=[  220],
     | 99.99th=[  220]
   bw (  KiB/s): min=   56, max=  240, per=24.66%, avg=140.33, stdev=40.56, samples=70
   iops        : min=   14, max=   60, avg=34.99, stdev=10.15, samples=70
  write: IOPS=37, BW=149KiB/s (153kB/s)(5264KiB/35221msec)
    slat (usec): min=23, max=2315, avg=95.97, stdev=72.97
    clat (msec): min=109, max=486, avg=289.10, stdev=68.66
     lat (msec): min=109, max=486, avg=289.20, stdev=68.66
    clat percentiles (msec):
     |  1.00th=[  134],  5.00th=[  184], 10.00th=[  205], 20.00th=[  230],
     | 30.00th=[  249], 40.00th=[  268], 50.00th=[  284], 60.00th=[  305],
     | 70.00th=[  326], 80.00th=[  351], 90.00th=[  384], 95.00th=[  409],
     | 99.00th=[  451], 99.50th=[  460], 99.90th=[  485], 99.95th=[  489],
     | 99.99th=[  489]
   bw (  KiB/s): min=   88, max=  223, per=25.09%, avg=148.79, stdev=29.13, samples=70
   iops        : min=   22, max=   55, avg=37.10, stdev= 7.22, samples=70
  lat (msec)   : 4=0.12%, 10=0.04%, 50=0.35%, 100=1.21%, 250=62.73%
  lat (msec)   : 500=35.55%
  cpu          : usr=0.22%, sys=1.28%, ctx=2982, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13180: Fri Mar 28 21:27:40 2025
   read: IOPS=35, BW=142KiB/s (145kB/s)(4984KiB/35200msec)
    slat (usec): min=33, max=5297, avg=2348.80, stdev=452.32
    clat (msec): min=3, max=327, avg=143.80, stdev=27.61
     lat (msec): min=3, max=329, avg=146.15, stdev=27.69
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[  105], 10.00th=[  114], 20.00th=[  125],
     | 30.00th=[  132], 40.00th=[  138], 50.00th=[  146], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  165], 90.00th=[  176], 95.00th=[  184],
     | 99.00th=[  205], 99.50th=[  209], 99.90th=[  326], 99.95th=[  330],
     | 99.99th=[  330]
   bw (  KiB/s): min=   64, max=  240, per=24.94%, avg=141.93, stdev=41.94, samples=70
   iops        : min=   16, max=   60, avg=35.39, stdev=10.45, samples=70
  write: IOPS=37, BW=149KiB/s (153kB/s)(5256KiB/35200msec)
    slat (usec): min=28, max=3317, avg=100.16, stdev=98.53
    clat (msec): min=27, max=499, avg=289.64, stdev=70.72
     lat (msec): min=27, max=499, avg=289.74, stdev=70.72
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  184], 10.00th=[  205], 20.00th=[  230],
     | 30.00th=[  251], 40.00th=[  268], 50.00th=[  288], 60.00th=[  309],
     | 70.00th=[  326], 80.00th=[  347], 90.00th=[  384], 95.00th=[  414],
     | 99.00th=[  456], 99.50th=[  464], 99.90th=[  489], 99.95th=[  502],
     | 99.99th=[  502]
   bw (  KiB/s): min=   80, max=  232, per=25.07%, avg=148.67, stdev=34.06, samples=70
   iops        : min=   20, max=   58, avg=37.07, stdev= 8.46, samples=70
  lat (msec)   : 4=0.12%, 10=0.04%, 20=0.04%, 50=0.27%, 100=1.72%
  lat (msec)   : 250=61.64%, 500=36.17%
  cpu          : usr=0.14%, sys=1.37%, ctx=3119, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13181: Fri Mar 28 21:27:40 2025
   read: IOPS=35, BW=144KiB/s (147kB/s)(5060KiB/35220msec)
    slat (usec): min=1124, max=5446, avg=2336.67, stdev=436.43
    clat (msec): min=4, max=348, avg=144.83, stdev=29.28
     lat (msec): min=5, max=350, avg=147.17, stdev=29.31
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[  104], 10.00th=[  113], 20.00th=[  126],
     | 30.00th=[  132], 40.00th=[  138], 50.00th=[  146], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  165], 90.00th=[  178], 95.00th=[  188],
     | 99.00th=[  209], 99.50th=[  236], 99.90th=[  347], 99.95th=[  351],
     | 99.99th=[  351]
   bw (  KiB/s): min=   55, max=  248, per=25.24%, avg=143.61, stdev=37.30, samples=70
   iops        : min=   13, max=   62, avg=35.79, stdev= 9.33, samples=70
  write: IOPS=36, BW=147KiB/s (151kB/s)(5180KiB/35220msec)
    slat (usec): min=22, max=410, avg=95.55, stdev=42.08
    clat (msec): min=100, max=497, avg=290.66, stdev=68.41
     lat (msec): min=100, max=497, avg=290.75, stdev=68.41
    clat percentiles (msec):
     |  1.00th=[  150],  5.00th=[  184], 10.00th=[  205], 20.00th=[  234],
     | 30.00th=[  253], 40.00th=[  271], 50.00th=[  288], 60.00th=[  305],
     | 70.00th=[  321], 80.00th=[  347], 90.00th=[  380], 95.00th=[  414],
     | 99.00th=[  460], 99.50th=[  468], 99.90th=[  489], 99.95th=[  498],
     | 99.99th=[  498]
   bw (  KiB/s): min=   72, max=  240, per=24.78%, avg=146.93, stdev=24.86, samples=70
   iops        : min=   18, max=   60, avg=36.61, stdev= 6.22, samples=70
  lat (msec)   : 10=0.12%, 20=0.04%, 50=0.27%, 100=1.72%, 250=61.60%
  lat (msec)   : 500=36.25%
  cpu          : usr=0.21%, sys=1.29%, ctx=3048, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=570KiB/s (583kB/s), 141KiB/s-144KiB/s (145kB/s-147kB/s), io=19.6MiB (20.5MB), run=35124-35221msec
  WRITE: bw=593KiB/s (607kB/s), 147KiB/s-149KiB/s (151kB/s-153kB/s), io=20.4MiB (21.4MB), run=35124-35221msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=13213: Fri Mar 28 21:27:41 2025
   read: IOPS=6, BW=280MiB/s (293MB/s)(40.0MiB/143msec)
    slat (nsec): min=8628.6k, max=8628.6k, avg=8628574.00, stdev= 0.00
    clat (nsec): min=127756k, max=127756k, avg=127756135.00, stdev= 0.00
     lat (nsec): min=136400k, max=136400k, avg=136399857.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  128],  5.00th=[  128], 10.00th=[  128], 20.00th=[  128],
     | 30.00th=[  128], 40.00th=[  128], 50.00th=[  128], 60.00th=[  128],
     | 70.00th=[  128], 80.00th=[  128], 90.00th=[  128], 95.00th=[  128],
     | 99.00th=[  128], 99.50th=[  128], 99.90th=[  128], 99.95th=[  128],
     | 99.99th=[  128]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.63%, ctx=23, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=280MiB/s (293MB/s), 280MiB/s-280MiB/s (293MB/s-293MB/s), io=40.0MiB (41.9MB), run=143-143msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout