== sanity test 398c: run fio to test AIO ================= 09:15:35 (1726838135)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.206818 s, 203 MB/s
osc.lustre-OST0000-osc-ffff8800b6cf7000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=7006: Fri Sep 20 09:15:52 2024
  write: IOPS=177, BW=708KiB/s (725kB/s)(10.0MiB/14459msec)
    slat (usec): min=12, max=4298, avg=112.01, stdev=268.83
    clat (msec): min=12, max=139, avg=89.96, stdev=17.10
     lat (msec): min=12, max=139, avg=90.07, stdev=17.10
    clat percentiles (msec):
     |  1.00th=[   49],  5.00th=[   64], 10.00th=[   70], 20.00th=[   77],
     | 30.00th=[   82], 40.00th=[   86], 50.00th=[   89], 60.00th=[   94],
     | 70.00th=[   99], 80.00th=[  106], 90.00th=[  113], 95.00th=[  120],
     | 99.00th=[  130], 99.50th=[  136], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  576, max=  888, per=24.84%, avg=703.36, stdev=73.66, samples=28
   iops        : min=  144, max=  222, avg=175.71, stdev=18.45, samples=28
  lat (msec)   : 20=0.16%, 50=1.33%, 100=72.23%, 250=26.29%
  cpu          : usr=0.74%, sys=2.42%, ctx=1087, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7007: Fri Sep 20 09:15:52 2024
  write: IOPS=177, BW=709KiB/s (726kB/s)(10.0MiB/14453msec)
    slat (usec): min=14, max=3282, avg=120.80, stdev=280.89
    clat (msec): min=9, max=139, avg=90.02, stdev=16.93
     lat (msec): min=9, max=139, avg=90.14, stdev=16.93
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   65], 10.00th=[   70], 20.00th=[   77],
     | 30.00th=[   82], 40.00th=[   86], 50.00th=[   89], 60.00th=[   94],
     | 70.00th=[   99], 80.00th=[  105], 90.00th=[  112], 95.00th=[  120],
     | 99.00th=[  129], 99.50th=[  133], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  566, max=  896, per=24.85%, avg=703.79, stdev=73.03, samples=28
   iops        : min=  141, max=  224, avg=175.82, stdev=18.32, samples=28
  lat (msec)   : 10=0.04%, 20=0.04%, 50=1.48%, 100=71.72%, 250=26.72%
  cpu          : usr=0.56%, sys=2.50%, ctx=1055, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7008: Fri Sep 20 09:15:52 2024
  write: IOPS=177, BW=709KiB/s (726kB/s)(10.0MiB/14443msec)
    slat (usec): min=11, max=8068, avg=106.86, stdev=285.06
    clat (msec): min=23, max=138, avg=89.91, stdev=17.27
     lat (msec): min=23, max=138, avg=90.02, stdev=17.28
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   64], 10.00th=[   70], 20.00th=[   78],
     | 30.00th=[   82], 40.00th=[   85], 50.00th=[   89], 60.00th=[   93],
     | 70.00th=[  100], 80.00th=[  106], 90.00th=[  113], 95.00th=[  118],
     | 99.00th=[  128], 99.50th=[  130], 99.90th=[  138], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  566, max=  896, per=24.96%, avg=706.93, stdev=71.12, samples=28
   iops        : min=  141, max=  224, avg=176.61, stdev=17.84, samples=28
  lat (msec)   : 50=2.15%, 100=69.14%, 250=28.71%
  cpu          : usr=0.50%, sys=2.51%, ctx=1063, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7009: Fri Sep 20 09:15:52 2024
  write: IOPS=177, BW=710KiB/s (727kB/s)(10.0MiB/14421msec)
    slat (usec): min=12, max=3732, avg=113.79, stdev=272.21
    clat (msec): min=12, max=149, avg=89.84, stdev=17.52
     lat (msec): min=13, max=150, avg=89.95, stdev=17.52
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   64], 10.00th=[   70], 20.00th=[   77],
     | 30.00th=[   82], 40.00th=[   86], 50.00th=[   89], 60.00th=[   95],
     | 70.00th=[  100], 80.00th=[  106], 90.00th=[  112], 95.00th=[  120],
     | 99.00th=[  128], 99.50th=[  129], 99.90th=[  140], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  552, max=  950, per=25.09%, avg=710.54, stdev=79.19, samples=28
   iops        : min=  138, max=  237, avg=177.46, stdev=19.78, samples=28
  lat (msec)   : 20=0.20%, 50=1.72%, 100=69.96%, 250=28.12%
  cpu          : usr=1.07%, sys=2.16%, ctx=1180, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2833KiB/s (2901kB/s), 708KiB/s-710KiB/s (725kB/s-727kB/s), io=40.0MiB (41.9MB), run=14421-14459msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=7031: Fri Sep 20 09:16:07 2024
   read: IOPS=93, BW=375KiB/s (384kB/s)(5048KiB/13465msec)
    slat (usec): min=1723, max=16464, avg=5556.11, stdev=2281.33
    clat (msec): min=12, max=142, avg=79.62, stdev=17.96
     lat (msec): min=15, max=155, avg=85.18, stdev=18.71
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   54], 10.00th=[   58], 20.00th=[   65],
     | 30.00th=[   70], 40.00th=[   74], 50.00th=[   79], 60.00th=[   83],
     | 70.00th=[   88], 80.00th=[   94], 90.00th=[  105], 95.00th=[  111],
     | 99.00th=[  130], 99.50th=[  133], 99.90th=[  142], 99.95th=[  144],
     | 99.99th=[  144]
   bw (  KiB/s): min=  312, max=  456, per=25.26%, avg=376.19, stdev=49.51, samples=26
   iops        : min=   78, max=  114, avg=93.77, stdev=12.33, samples=26
  write: IOPS=96, BW=386KiB/s (395kB/s)(5192KiB/13465msec)
    slat (usec): min=14, max=2189, avg=122.22, stdev=212.35
    clat (msec): min=35, max=139, avg=82.09, stdev=17.02
     lat (msec): min=35, max=139, avg=82.21, stdev=17.03
    clat percentiles (msec):
     |  1.00th=[   48],  5.00th=[   57], 10.00th=[   63], 20.00th=[   68],
     | 30.00th=[   72], 40.00th=[   78], 50.00th=[   82], 60.00th=[   86],
     | 70.00th=[   90], 80.00th=[   96], 90.00th=[  106], 95.00th=[  112],
     | 99.00th=[  130], 99.50th=[  136], 99.90th=[  138], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  295, max=  496, per=24.93%, avg=386.65, stdev=57.14, samples=26
   iops        : min=   73, max=  124, avg=96.38, stdev=14.36, samples=26
  lat (msec)   : 20=0.08%, 50=2.27%, 100=83.75%, 250=13.91%
  cpu          : usr=0.88%, sys=5.64%, ctx=2386, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7032: Fri Sep 20 09:16:07 2024
   read: IOPS=92, BW=369KiB/s (378kB/s)(4976KiB/13467msec)
    slat (usec): min=1372, max=17678, avg=5464.60, stdev=2208.49
    clat (msec): min=13, max=140, avg=79.61, stdev=17.45
     lat (msec): min=19, max=145, avg=85.08, stdev=18.03
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   53], 10.00th=[   58], 20.00th=[   65],
     | 30.00th=[   70], 40.00th=[   75], 50.00th=[   79], 60.00th=[   84],
     | 70.00th=[   89], 80.00th=[   94], 90.00th=[  104], 95.00th=[  110],
     | 99.00th=[  125], 99.50th=[  128], 99.90th=[  136], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  296, max=  472, per=24.82%, avg=369.54, stdev=48.50, samples=26
   iops        : min=   74, max=  118, avg=92.23, stdev=12.09, samples=26
  write: IOPS=97, BW=391KiB/s (400kB/s)(5264KiB/13467msec)
    slat (usec): min=15, max=4754, avg=137.65, stdev=305.38
    clat (msec): min=20, max=140, avg=82.15, stdev=16.61
     lat (msec): min=20, max=140, avg=82.29, stdev=16.62
    clat percentiles (msec):
     |  1.00th=[   48],  5.00th=[   57], 10.00th=[   62], 20.00th=[   69],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   82], 60.00th=[   86],
     | 70.00th=[   91], 80.00th=[   96], 90.00th=[  105], 95.00th=[  111],
     | 99.00th=[  123], 99.50th=[  128], 99.90th=[  134], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  280, max=  552, per=25.33%, avg=392.92, stdev=62.91, samples=26
   iops        : min=   70, max=  138, avg=98.08, stdev=15.68, samples=26
  lat (msec)   : 20=0.04%, 50=2.23%, 100=84.02%, 250=13.71%
  cpu          : usr=0.38%, sys=6.15%, ctx=2378, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7033: Fri Sep 20 09:16:07 2024
   read: IOPS=92, BW=370KiB/s (379kB/s)(4984KiB/13469msec)
    slat (usec): min=1660, max=17540, avg=5459.14, stdev=2229.38
    clat (msec): min=4, max=142, avg=79.03, stdev=17.84
     lat (msec): min=11, max=146, avg=84.49, stdev=18.59
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   52], 10.00th=[   57], 20.00th=[   65],
     | 30.00th=[   70], 40.00th=[   73], 50.00th=[   78], 60.00th=[   83],
     | 70.00th=[   88], 80.00th=[   94], 90.00th=[  103], 95.00th=[  110],
     | 99.00th=[  125], 99.50th=[  131], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  272, max=  504, per=25.02%, avg=372.50, stdev=59.75, samples=26
   iops        : min=   68, max=  126, avg=92.96, stdev=14.97, samples=26
  write: IOPS=97, BW=390KiB/s (400kB/s)(5256KiB/13469msec)
    slat (usec): min=13, max=5243, avg=136.69, stdev=326.42
    clat (msec): min=37, max=143, avg=82.76, stdev=17.27
     lat (msec): min=37, max=143, avg=82.90, stdev=17.26
    clat percentiles (msec):
     |  1.00th=[   50],  5.00th=[   56], 10.00th=[   63], 20.00th=[   68],
     | 30.00th=[   74], 40.00th=[   78], 50.00th=[   82], 60.00th=[   86],
     | 70.00th=[   91], 80.00th=[   97], 90.00th=[  107], 95.00th=[  114],
     | 99.00th=[  128], 99.50th=[  136], 99.90th=[  144], 99.95th=[  144],
     | 99.99th=[  144]
   bw (  KiB/s): min=  255, max=  510, per=25.15%, avg=390.00, stdev=57.20, samples=26
   iops        : min=   63, max=  127, avg=97.35, stdev=14.34, samples=26
  lat (msec)   : 10=0.04%, 50=2.58%, 100=83.05%, 250=14.34%
  cpu          : usr=1.83%, sys=4.74%, ctx=2336, majf=0, minf=34
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7034: Fri Sep 20 09:16:07 2024
   read: IOPS=94, BW=379KiB/s (388kB/s)(5060KiB/13362msec)
    slat (usec): min=19, max=18991, avg=5384.81, stdev=2288.40
    clat (msec): min=12, max=168, avg=78.84, stdev=18.57
     lat (msec): min=16, max=170, avg=84.23, stdev=19.26
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   52], 10.00th=[   57], 20.00th=[   65],
     | 30.00th=[   70], 40.00th=[   74], 50.00th=[   79], 60.00th=[   83],
     | 70.00th=[   87], 80.00th=[   93], 90.00th=[  103], 95.00th=[  111],
     | 99.00th=[  127], 99.50th=[  136], 99.90th=[  163], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  296, max=  448, per=25.58%, avg=380.85, stdev=46.66, samples=26
   iops        : min=   74, max=  112, avg=94.92, stdev=11.60, samples=26
  write: IOPS=96, BW=388KiB/s (397kB/s)(5180KiB/13362msec)
    slat (usec): min=15, max=3375, avg=124.56, stdev=237.25
    clat (msec): min=14, max=185, avg=82.41, stdev=17.90
     lat (msec): min=15, max=185, avg=82.53, stdev=17.89
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   56], 10.00th=[   61], 20.00th=[   68],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   83], 60.00th=[   86],
     | 70.00th=[   91], 80.00th=[   96], 90.00th=[  105], 95.00th=[  112],
     | 99.00th=[  133], 99.50th=[  142], 99.90th=[  186], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  288, max=  510, per=25.03%, avg=388.19, stdev=57.85, samples=26
   iops        : min=   72, max=  127, avg=96.77, stdev=14.53, samples=26
  lat (msec)   : 20=0.23%, 50=2.73%, 100=83.55%, 250=13.48%
  cpu          : usr=1.33%, sys=5.20%, ctx=2466, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1490KiB/s (1526kB/s), 369KiB/s-379KiB/s (378kB/s-388kB/s), io=19.6MiB (20.5MB), run=13362-13469msec
  WRITE: bw=1551KiB/s (1588kB/s), 386KiB/s-391KiB/s (395kB/s-400kB/s), io=20.4MiB (21.4MB), run=13362-13469msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=7049: Fri Sep 20 09:16:09 2024
   read: IOPS=2, BW=102MiB/s (106MB/s)(40.0MiB/394msec)
    slat (nsec): min=28416k, max=28416k, avg=28416145.00, stdev= 0.00
    clat (nsec): min=353006k, max=353006k, avg=353005824.00, stdev= 0.00
     lat (nsec): min=381450k, max=381450k, avg=381449956.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  355],  5.00th=[  355], 10.00th=[  355], 20.00th=[  355],
     | 30.00th=[  355], 40.00th=[  355], 50.00th=[  355], 60.00th=[  355],
     | 70.00th=[  355], 80.00th=[  355], 90.00th=[  355], 95.00th=[  355],
     | 99.00th=[  355], 99.50th=[  355], 99.90th=[  355], 99.95th=[  355],
     | 99.99th=[  355]
  lat (msec)   : 500=100.00%
  cpu          : usr=0.00%, sys=2.80%, ctx=15, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=102MiB/s (106MB/s), 102MiB/s-102MiB/s (106MB/s-106MB/s), io=40.0MiB (41.9MB), run=394-394msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout