== sanity test 398c: run fio to test AIO ================= 10:25:42 (1724768742)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0832683 s, 504 MB/s
osc.lustre-OST0000-osc-ffff8800b3c40800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23885: Tue Aug 27 10:26:09 2024
  write: IOPS=99, BW=400KiB/s (409kB/s)(10.0MiB/25620msec)
    slat (usec): min=13, max=716, avg=70.65, stdev=45.38
    clat (msec): min=14, max=307, avg=159.95, stdev=34.17
     lat (msec): min=14, max=307, avg=160.02, stdev=34.17
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[  102], 10.00th=[  118], 20.00th=[  136],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  203], 95.00th=[  213],
     | 99.00th=[  243], 99.50th=[  251], 99.90th=[  288], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  KiB/s): min=  304, max=  736, per=25.03%, avg=399.16, stdev=71.33, samples=51
   iops        : min=   76, max=  184, avg=99.75, stdev=17.83, samples=51
  lat (msec)   : 20=0.12%, 50=0.23%, 100=4.18%, 250=95.00%, 500=0.47%
  cpu          : usr=0.16%, sys=0.77%, ctx=2376, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23886: Tue Aug 27 10:26:09 2024
  write: IOPS=99, BW=399KiB/s (408kB/s)(10.0MiB/25678msec)
    slat (usec): min=13, max=639, avg=72.52, stdev=48.64
    clat (msec): min=65, max=303, avg=160.25, stdev=32.90
     lat (msec): min=65, max=303, avg=160.32, stdev=32.91
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[  102], 10.00th=[  120], 20.00th=[  136],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  178], 80.00th=[  186], 90.00th=[  203], 95.00th=[  215],
     | 99.00th=[  241], 99.50th=[  245], 99.90th=[  271], 99.95th=[  271],
     | 99.99th=[  305]
   bw (  KiB/s): min=  295, max=  640, per=25.00%, avg=398.73, stdev=64.72, samples=51
   iops        : min=   73, max=  160, avg=99.67, stdev=16.20, samples=51
  lat (msec)   : 100=4.80%, 250=94.88%, 500=0.31%
  cpu          : usr=0.13%, sys=0.76%, ctx=2566, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23887: Tue Aug 27 10:26:09 2024
  write: IOPS=99, BW=399KiB/s (409kB/s)(10.0MiB/25661msec)
    slat (usec): min=13, max=438, avg=68.32, stdev=43.29
    clat (msec): min=59, max=286, avg=160.17, stdev=32.79
     lat (msec): min=59, max=286, avg=160.24, stdev=32.79
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  102], 10.00th=[  120], 20.00th=[  134],
     | 30.00th=[  144], 40.00th=[  155], 50.00th=[  161], 60.00th=[  169],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  203], 95.00th=[  213],
     | 99.00th=[  241], 99.50th=[  245], 99.90th=[  266], 99.95th=[  275],
     | 99.99th=[  288]
   bw (  KiB/s): min=  280, max=  640, per=25.02%, avg=399.04, stdev=67.89, samples=51
   iops        : min=   70, max=  160, avg=99.75, stdev=17.00, samples=51
  lat (msec)   : 100=4.80%, 250=94.84%, 500=0.35%
  cpu          : usr=0.17%, sys=0.74%, ctx=2228, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23888: Tue Aug 27 10:26:09 2024
  write: IOPS=99, BW=399KiB/s (408kB/s)(10.0MiB/25678msec)
    slat (usec): min=13, max=431, avg=71.33, stdev=46.91
    clat (msec): min=62, max=261, avg=160.31, stdev=32.38
     lat (msec): min=62, max=261, avg=160.39, stdev=32.38
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  104], 10.00th=[  120], 20.00th=[  136],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  178], 80.00th=[  186], 90.00th=[  203], 95.00th=[  215],
     | 99.00th=[  241], 99.50th=[  247], 99.90th=[  257], 99.95th=[  257],
     | 99.99th=[  262]
   bw (  KiB/s): min=  288, max=  640, per=24.94%, avg=397.84, stdev=63.87, samples=51
   iops        : min=   72, max=  160, avg=99.39, stdev=15.97, samples=51
  lat (msec)   : 100=4.26%, 250=95.43%, 500=0.31%
  cpu          : usr=0.18%, sys=0.72%, ctx=2404, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1595KiB/s (1633kB/s), 399KiB/s-400KiB/s (408kB/s-409kB/s), io=40.0MiB (41.9MB), run=25620-25678msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23918: Tue Aug 27 10:26:26 2024
   read: IOPS=77, BW=309KiB/s (317kB/s)(5048KiB/16314msec)
    slat (usec): min=709, max=4550, avg=1196.96, stdev=291.74
    clat (msec): min=3, max=121, avg=64.85, stdev=11.71
     lat (msec): min=4, max=122, avg=66.05, stdev=11.72
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   50], 10.00th=[   52], 20.00th=[   56],
     | 30.00th=[   59], 40.00th=[   63], 50.00th=[   65], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   74], 90.00th=[   80], 95.00th=[   83],
     | 99.00th=[   95], 99.50th=[  108], 99.90th=[  120], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  184, max=  472, per=25.14%, avg=308.75, stdev=61.93, samples=32
   iops        : min=   46, max=  118, avg=77.19, stdev=15.48, samples=32
  write: IOPS=79, BW=318KiB/s (326kB/s)(5192KiB/16314msec)
    slat (usec): min=15, max=205, avg=55.45, stdev=25.44
    clat (msec): min=60, max=224, avg=136.57, stdev=30.32
     lat (msec): min=60, max=224, avg=136.63, stdev=30.32
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[   87], 10.00th=[   97], 20.00th=[  110],
     | 30.00th=[  120], 40.00th=[  130], 50.00th=[  138], 60.00th=[  146],
     | 70.00th=[  153], 80.00th=[  163], 90.00th=[  176], 95.00th=[  188],
     | 99.00th=[  203], 99.50th=[  209], 99.90th=[  220], 99.95th=[  226],
     | 99.99th=[  226]
   bw (  KiB/s): min=  264, max=  376, per=24.77%, avg=316.75, stdev=32.50, samples=32
   iops        : min=   66, max=   94, avg=79.19, stdev= 8.13, samples=32
  lat (msec)   : 4=0.04%, 20=0.23%, 50=3.28%, 100=51.56%, 250=44.88%
  cpu          : usr=0.23%, sys=1.47%, ctx=3209, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23919: Tue Aug 27 10:26:26 2024
   read: IOPS=76, BW=305KiB/s (312kB/s)(4976KiB/16334msec)
    slat (usec): min=703, max=3884, avg=1206.33, stdev=287.74
    clat (msec): min=3, max=151, avg=64.76, stdev=12.83
     lat (msec): min=4, max=152, avg=65.97, stdev=12.84
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   47], 10.00th=[   52], 20.00th=[   56],
     | 30.00th=[   59], 40.00th=[   62], 50.00th=[   65], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   74], 90.00th=[   80], 95.00th=[   85],
     | 99.00th=[   94], 99.50th=[  110], 99.90th=[  138], 99.95th=[  153],
     | 99.99th=[  153]
   bw (  KiB/s): min=  184, max=  440, per=24.77%, avg=304.12, stdev=66.36, samples=32
   iops        : min=   46, max=  110, avg=75.94, stdev=16.62, samples=32
  write: IOPS=80, BW=322KiB/s (330kB/s)(5264KiB/16334msec)
    slat (usec): min=14, max=277, avg=55.10, stdev=25.98
    clat (msec): min=56, max=226, avg=135.87, stdev=30.49
     lat (msec): min=56, max=226, avg=135.93, stdev=30.49
    clat percentiles (msec):
     |  1.00th=[   63],  5.00th=[   84], 10.00th=[   96], 20.00th=[  110],
     | 30.00th=[  121], 40.00th=[  129], 50.00th=[  138], 60.00th=[  144],
     | 70.00th=[  153], 80.00th=[  163], 90.00th=[  176], 95.00th=[  186],
     | 99.00th=[  199], 99.50th=[  209], 99.90th=[  211], 99.95th=[  226],
     | 99.99th=[  226]
   bw (  KiB/s): min=  216, max=  359, per=25.01%, avg=319.88, stdev=32.21, samples=32
   iops        : min=   54, max=   89, avg=79.88, stdev= 8.02, samples=32
  lat (msec)   : 4=0.04%, 10=0.12%, 20=0.27%, 50=3.32%, 100=51.33%
  lat (msec)   : 250=44.92%
  cpu          : usr=0.18%, sys=1.51%, ctx=3116, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23920: Tue Aug 27 10:26:26 2024
   read: IOPS=76, BW=305KiB/s (312kB/s)(4984KiB/16334msec)
    slat (usec): min=707, max=3575, avg=1197.25, stdev=279.41
    clat (usec): min=1718, max=138636, avg=65074.71, stdev=12294.04
     lat (msec): min=2, max=140, avg=66.27, stdev=12.31
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   48], 10.00th=[   52], 20.00th=[   56],
     | 30.00th=[   60], 40.00th=[   63], 50.00th=[   66], 60.00th=[   69],
     | 70.00th=[   71], 80.00th=[   74], 90.00th=[   80], 95.00th=[   83],
     | 99.00th=[   94], 99.50th=[  108], 99.90th=[  138], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  176, max=  440, per=24.91%, avg=305.88, stdev=57.35, samples=32
   iops        : min=   44, max=  110, avg=76.38, stdev=14.28, samples=32
  write: IOPS=80, BW=322KiB/s (330kB/s)(5256KiB/16334msec)
    slat (usec): min=13, max=966, avg=57.77, stdev=37.81
    clat (msec): min=59, max=226, avg=135.70, stdev=30.17
     lat (msec): min=59, max=226, avg=135.76, stdev=30.17
    clat percentiles (msec):
     |  1.00th=[   66],  5.00th=[   85], 10.00th=[   96], 20.00th=[  111],
     | 30.00th=[  121], 40.00th=[  130], 50.00th=[  136], 60.00th=[  144],
     | 70.00th=[  153], 80.00th=[  161], 90.00th=[  174], 95.00th=[  186],
     | 99.00th=[  207], 99.50th=[  211], 99.90th=[  226], 99.95th=[  226],
     | 99.99th=[  226]
   bw (  KiB/s): min=  232, max=  392, per=24.93%, avg=318.88, stdev=37.24, samples=32
   iops        : min=   58, max=   98, avg=79.62, stdev= 9.36, samples=32
  lat (msec)   : 2=0.04%, 10=0.08%, 20=0.31%, 50=3.36%, 100=51.13%
  lat (msec)   : 250=45.08%
  cpu          : usr=0.19%, sys=1.51%, ctx=3232, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23921: Tue Aug 27 10:26:26 2024
   read: IOPS=78, BW=312KiB/s (320kB/s)(5060KiB/16194msec)
    slat (usec): min=12, max=3688, avg=1185.28, stdev=286.28
    clat (usec): min=1083, max=177332, avg=65102.04, stdev=14001.62
     lat (usec): min=1472, max=178779, avg=66289.05, stdev=14048.75
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   47], 10.00th=[   52], 20.00th=[   56],
     | 30.00th=[   60], 40.00th=[   63], 50.00th=[   66], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   74], 90.00th=[   80], 95.00th=[   84],
     | 99.00th=[   96], 99.50th=[  108], 99.90th=[  178], 99.95th=[  178],
     | 99.99th=[  178]
   bw (  KiB/s): min=  176, max=  416, per=25.58%, avg=314.12, stdev=57.98, samples=32
   iops        : min=   44, max=  104, avg=78.44, stdev=14.52, samples=32
  write: IOPS=79, BW=320KiB/s (328kB/s)(5180KiB/16194msec)
    slat (usec): min=12, max=335, avg=54.90, stdev=26.30
    clat (msec): min=16, max=231, avg=135.12, stdev=32.17
     lat (msec): min=16, max=231, avg=135.18, stdev=32.17
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   84], 10.00th=[   93], 20.00th=[  108],
     | 30.00th=[  120], 40.00th=[  129], 50.00th=[  136], 60.00th=[  144],
     | 70.00th=[  153], 80.00th=[  161], 90.00th=[  176], 95.00th=[  188],
     | 99.00th=[  209], 99.50th=[  222], 99.90th=[  232], 99.95th=[  232],
     | 99.99th=[  232]
   bw (  KiB/s): min=  248, max=  424, per=24.91%, avg=318.62, stdev=43.60, samples=32
   iops        : min=   62, max=  106, avg=79.56, stdev=10.91, samples=32
  lat (msec)   : 2=0.12%, 4=0.12%, 10=0.04%, 20=0.59%, 50=3.67%
  lat (msec)   : 100=51.68%, 250=43.79%
  cpu          : usr=0.25%, sys=1.46%, ctx=3185, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1229KiB/s (1258kB/s), 305KiB/s-312KiB/s (312kB/s-320kB/s), io=19.6MiB (20.5MB), run=16194-16334msec
  WRITE: bw=1279KiB/s (1310kB/s), 318KiB/s-322KiB/s (326kB/s-330kB/s), io=20.4MiB (21.4MB), run=16194-16334msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23938: Tue Aug 27 10:26:27 2024
   read: IOPS=13, BW=533MiB/s (559MB/s)(40.0MiB/75msec)
    slat (nsec): min=5657.2k, max=5657.2k, avg=5657247.00, stdev= 0.00
    clat (nsec): min=64717k, max=64717k, avg=64716561.00, stdev= 0.00
     lat (nsec): min=70385k, max=70385k, avg=70385035.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[64750],  5.00th=[64750], 10.00th=[64750], 20.00th=[64750],
     | 30.00th=[64750], 40.00th=[64750], 50.00th=[64750], 60.00th=[64750],
     | 70.00th=[64750], 80.00th=[64750], 90.00th=[64750], 95.00th=[64750],
     | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.76%, ctx=21, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=533MiB/s (559MB/s), 533MiB/s-533MiB/s (559MB/s-559MB/s), io=40.0MiB (41.9MB), run=75-75msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout