== sanity test 398c: run fio to test AIO ================= 13:59:25 (1721411965)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.158699 s, 264 MB/s
osc.lustre-OST0000-osc-ffff8800a8bbd800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=28781: Fri Jul 19 13:59:55 2024
  write: IOPS=90, BW=363KiB/s (372kB/s)(10.0MiB/28191msec)
    slat (usec): min=12, max=355, avg=64.56, stdev=40.35
    clat (msec): min=21, max=387, avg=176.02, stdev=36.14
     lat (msec): min=21, max=387, avg=176.08, stdev=36.14
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[  117], 10.00th=[  134], 20.00th=[  150],
     | 30.00th=[  161], 40.00th=[  171], 50.00th=[  178], 60.00th=[  186],
     | 70.00th=[  192], 80.00th=[  201], 90.00th=[  218], 95.00th=[  232],
     | 99.00th=[  266], 99.50th=[  284], 99.90th=[  363], 99.95th=[  376],
     | 99.99th=[  388]
   bw (  KiB/s): min=  256, max=  648, per=25.05%, avg=363.45, stdev=60.69, samples=56
   iops        : min=   64, max=  162, avg=90.77, stdev=15.17, samples=56
  lat (msec)   : 50=0.12%, 100=3.16%, 250=94.41%, 500=2.30%
  cpu          : usr=0.17%, sys=0.62%, ctx=2121, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28782: Fri Jul 19 13:59:55 2024
  write: IOPS=90, BW=363KiB/s (372kB/s)(10.0MiB/28225msec)
    slat (usec): min=12, max=330, avg=63.79, stdev=39.46
    clat (msec): min=32, max=311, avg=176.24, stdev=33.26
     lat (msec): min=32, max=311, avg=176.31, stdev=33.27
    clat percentiles (msec):
     |  1.00th=[   91],  5.00th=[  125], 10.00th=[  138], 20.00th=[  150],
     | 30.00th=[  161], 40.00th=[  171], 50.00th=[  178], 60.00th=[  184],
     | 70.00th=[  192], 80.00th=[  201], 90.00th=[  218], 95.00th=[  232],
     | 99.00th=[  264], 99.50th=[  275], 99.90th=[  296], 99.95th=[  300],
     | 99.99th=[  313]
   bw (  KiB/s): min=  256, max=  520, per=24.96%, avg=362.16, stdev=50.58, samples=56
   iops        : min=   64, max=  130, avg=90.45, stdev=12.64, samples=56
  lat (msec)   : 50=0.04%, 100=2.07%, 250=95.78%, 500=2.11%
  cpu          : usr=0.10%, sys=0.71%, ctx=2012, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28783: Fri Jul 19 13:59:55 2024
  write: IOPS=90, BW=363KiB/s (372kB/s)(10.0MiB/28225msec)
    slat (usec): min=12, max=535, avg=67.71, stdev=43.26
    clat (msec): min=56, max=298, avg=176.23, stdev=33.92
     lat (msec): min=56, max=298, avg=176.30, stdev=33.92
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  124], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  161], 40.00th=[  171], 50.00th=[  178], 60.00th=[  186],
     | 70.00th=[  192], 80.00th=[  203], 90.00th=[  215], 95.00th=[  232],
     | 99.00th=[  259], 99.50th=[  279], 99.90th=[  300], 99.95th=[  300],
     | 99.99th=[  300]
   bw (  KiB/s): min=  256, max=  560, per=24.96%, avg=362.16, stdev=51.01, samples=56
   iops        : min=   64, max=  140, avg=90.45, stdev=12.76, samples=56
  lat (msec)   : 100=2.38%, 250=95.70%, 500=1.91%
  cpu          : usr=0.13%, sys=0.67%, ctx=2305, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28784: Fri Jul 19 13:59:55 2024
  write: IOPS=90, BW=363KiB/s (372kB/s)(10.0MiB/28225msec)
    slat (usec): min=11, max=537, avg=64.38, stdev=40.81
    clat (msec): min=56, max=323, avg=176.22, stdev=33.98
     lat (msec): min=56, max=324, avg=176.28, stdev=33.98
    clat percentiles (msec):
     |  1.00th=[   90],  5.00th=[  125], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  161], 40.00th=[  171], 50.00th=[  178], 60.00th=[  184],
     | 70.00th=[  192], 80.00th=[  201], 90.00th=[  218], 95.00th=[  232],
     | 99.00th=[  266], 99.50th=[  279], 99.90th=[  326], 99.95th=[  326],
     | 99.99th=[  326]
   bw (  KiB/s): min=  256, max=  592, per=25.02%, avg=363.05, stdev=53.51, samples=56
   iops        : min=   64, max=  148, avg=90.70, stdev=13.37, samples=56
  lat (msec)   : 100=2.19%, 250=95.43%, 500=2.38%
  cpu          : usr=0.13%, sys=0.66%, ctx=2097, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1451KiB/s (1486kB/s), 363KiB/s-363KiB/s (372kB/s-372kB/s), io=40.0MiB (41.9MB), run=28191-28225msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=28817: Fri Jul 19 14:00:13 2024
   read: IOPS=75, BW=303KiB/s (310kB/s)(5048KiB/16677msec)
    slat (usec): min=475, max=2352, avg=788.99, stdev=204.12
    clat (msec): min=10, max=205, avg=68.97, stdev=15.31
     lat (msec): min=11, max=206, avg=69.76, stdev=15.32
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   51], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   68], 60.00th=[   72],
     | 70.00th=[   75], 80.00th=[   80], 90.00th=[   86], 95.00th=[   91],
     | 99.00th=[  106], 99.50th=[  108], 99.90th=[  194], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=  184, max=  416, per=25.33%, avg=304.18, stdev=55.20, samples=33
   iops        : min=   46, max=  104, avg=76.00, stdev=13.79, samples=33
  write: IOPS=77, BW=311KiB/s (319kB/s)(5192KiB/16677msec)
    slat (usec): min=12, max=345, avg=49.13, stdev=25.69
    clat (msec): min=58, max=266, avg=137.43, stdev=32.59
     lat (msec): min=58, max=266, avg=137.48, stdev=32.59
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[   84], 10.00th=[   93], 20.00th=[  109],
     | 30.00th=[  121], 40.00th=[  131], 50.00th=[  140], 60.00th=[  146],
     | 70.00th=[  155], 80.00th=[  165], 90.00th=[  180], 95.00th=[  192],
     | 99.00th=[  215], 99.50th=[  220], 99.90th=[  266], 99.95th=[  266],
     | 99.99th=[  266]
   bw (  KiB/s): min=  239, max=  400, per=24.92%, avg=311.45, stdev=37.48, samples=33
   iops        : min=   59, max=  100, avg=77.82, stdev= 9.42, samples=33
  lat (msec)   : 20=0.23%, 50=2.07%, 100=53.32%, 250=44.26%, 500=0.12%
  cpu          : usr=0.21%, sys=1.33%, ctx=3061, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28818: Fri Jul 19 14:00:13 2024
   read: IOPS=74, BW=298KiB/s (305kB/s)(4976KiB/16701msec)
    slat (usec): min=463, max=2186, avg=786.08, stdev=190.81
    clat (msec): min=8, max=151, avg=68.75, stdev=13.48
     lat (msec): min=9, max=152, avg=69.53, stdev=13.48
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   50], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   66], 50.00th=[   69], 60.00th=[   72],
     | 70.00th=[   75], 80.00th=[   80], 90.00th=[   86], 95.00th=[   90],
     | 99.00th=[  102], 99.50th=[  106], 99.90th=[  150], 99.95th=[  153],
     | 99.99th=[  153]
   bw (  KiB/s): min=  176, max=  416, per=24.72%, avg=296.91, stdev=58.63, samples=33
   iops        : min=   44, max=  104, avg=74.18, stdev=14.70, samples=33
  write: IOPS=78, BW=315KiB/s (323kB/s)(5264KiB/16701msec)
    slat (usec): min=12, max=296, avg=49.57, stdev=25.41
    clat (msec): min=57, max=226, avg=137.11, stdev=32.36
     lat (msec): min=57, max=226, avg=137.16, stdev=32.36
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   84], 10.00th=[   92], 20.00th=[  108],
     | 30.00th=[  118], 40.00th=[  131], 50.00th=[  140], 60.00th=[  146],
     | 70.00th=[  155], 80.00th=[  165], 90.00th=[  180], 95.00th=[  190],
     | 99.00th=[  211], 99.50th=[  215], 99.90th=[  228], 99.95th=[  228],
     | 99.99th=[  228]
   bw (  KiB/s): min=  256, max=  400, per=25.23%, avg=315.33, stdev=43.18, samples=33
   iops        : min=   64, max=  100, avg=78.79, stdev=10.80, samples=33
  lat (msec)   : 10=0.08%, 20=0.12%, 50=2.58%, 100=52.73%, 250=44.49%
  cpu          : usr=0.19%, sys=1.33%, ctx=3109, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28819: Fri Jul 19 14:00:13 2024
   read: IOPS=74, BW=299KiB/s (306kB/s)(4984KiB/16677msec)
    slat (usec): min=12, max=2289, avg=779.33, stdev=201.85
    clat (usec): min=853, max=168168, avg=68889.31, stdev=14089.63
     lat (usec): min=867, max=169029, avg=69670.41, stdev=14115.85
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   51], 10.00th=[   55], 20.00th=[   59],
     | 30.00th=[   63], 40.00th=[   66], 50.00th=[   69], 60.00th=[   72],
     | 70.00th=[   75], 80.00th=[   80], 90.00th=[   86], 95.00th=[   92],
     | 99.00th=[  102], 99.50th=[  107], 99.90th=[  144], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  200, max=  456, per=25.04%, avg=300.79, stdev=57.78, samples=33
   iops        : min=   50, max=  114, avg=75.15, stdev=14.44, samples=33
  write: IOPS=78, BW=315KiB/s (323kB/s)(5256KiB/16677msec)
    slat (usec): min=12, max=202, avg=47.62, stdev=22.34
    clat (msec): min=15, max=244, avg=136.81, stdev=33.27
     lat (msec): min=15, max=244, avg=136.86, stdev=33.27
    clat percentiles (msec):
     |  1.00th=[   52],  5.00th=[   83], 10.00th=[   94], 20.00th=[  109],
     | 30.00th=[  120], 40.00th=[  130], 50.00th=[  140], 60.00th=[  148],
     | 70.00th=[  155], 80.00th=[  163], 90.00th=[  180], 95.00th=[  192],
     | 99.00th=[  211], 99.50th=[  215], 99.90th=[  230], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  224, max=  392, per=25.11%, avg=313.88, stdev=35.13, samples=33
   iops        : min=   56, max=   98, avg=78.42, stdev= 8.85, samples=33
  lat (usec)   : 1000=0.12%
  lat (msec)   : 2=0.12%, 20=0.43%, 50=2.38%, 100=52.30%, 250=44.65%
  cpu          : usr=0.26%, sys=1.28%, ctx=2820, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28820: Fri Jul 19 14:00:13 2024
   read: IOPS=75, BW=304KiB/s (311kB/s)(5060KiB/16666msec)
    slat (usec): min=461, max=2524, avg=776.49, stdev=182.59
    clat (msec): min=8, max=156, avg=68.47, stdev=13.57
     lat (msec): min=9, max=157, avg=69.25, stdev=13.58
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   50], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   68], 60.00th=[   71],
     | 70.00th=[   74], 80.00th=[   80], 90.00th=[   86], 95.00th=[   91],
     | 99.00th=[  104], 99.50th=[  108], 99.90th=[  157], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  216, max=  480, per=25.37%, avg=304.64, stdev=54.55, samples=33
   iops        : min=   54, max=  120, avg=76.09, stdev=13.66, samples=33
  write: IOPS=77, BW=311KiB/s (318kB/s)(5180KiB/16666msec)
    slat (usec): min=12, max=284, avg=48.84, stdev=24.42
    clat (msec): min=58, max=301, avg=137.96, stdev=33.51
     lat (msec): min=58, max=301, avg=138.01, stdev=33.51
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[   84], 10.00th=[   93], 20.00th=[  109],
     | 30.00th=[  120], 40.00th=[  131], 50.00th=[  140], 60.00th=[  148],
     | 70.00th=[  155], 80.00th=[  165], 90.00th=[  180], 95.00th=[  192],
     | 99.00th=[  220], 99.50th=[  255], 99.90th=[  288], 99.95th=[  300],
     | 99.99th=[  300]
   bw (  KiB/s): min=  240, max=  376, per=24.82%, avg=310.21, stdev=36.77, samples=33
   iops        : min=   60, max=   94, avg=77.48, stdev= 9.21, samples=33
  lat (msec)   : 10=0.04%, 20=0.20%, 50=2.62%, 100=52.97%, 250=43.91%
  lat (msec)   : 500=0.27%
  cpu          : usr=0.21%, sys=1.32%, ctx=2989, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1202KiB/s (1230kB/s), 298KiB/s-304KiB/s (305kB/s-311kB/s), io=19.6MiB (20.5MB), run=16666-16701msec
  WRITE: bw=1251KiB/s (1281kB/s), 311KiB/s-315KiB/s (318kB/s-323kB/s), io=20.4MiB (21.4MB), run=16666-16701msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=28837: Fri Jul 19 14:00:14 2024
   read: IOPS=7, BW=313MiB/s (328MB/s)(40.0MiB/128msec)
    slat (nsec): min=7168.9k, max=7168.9k, avg=7168901.00, stdev= 0.00
    clat (nsec): min=115611k, max=115611k, avg=115610697.00, stdev= 0.00
     lat (nsec): min=122789k, max=122789k, avg=122789448.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  116],  5.00th=[  116], 10.00th=[  116], 20.00th=[  116],
     | 30.00th=[  116], 40.00th=[  116], 50.00th=[  116], 60.00th=[  116],
     | 70.00th=[  116], 80.00th=[  116], 90.00th=[  116], 95.00th=[  116],
     | 99.00th=[  116], 99.50th=[  116], 99.90th=[  116], 99.95th=[  116],
     | 99.99th=[  116]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.51%, ctx=14, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=40.0MiB (41.9MB), run=128-128msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck