== sanity test 398c: run fio to test AIO ================= 19:26:28 (1736987188)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0892367 s, 470 MB/s
osc.lustre-OST0000-osc-ffff88012aab8800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23629: Wed Jan 15 19:26:56 2025
  write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26219msec)
    slat (usec): min=11, max=497, avg=68.67, stdev=41.37
    clat (msec): min=11, max=338, avg=163.72, stdev=37.65
     lat (msec): min=11, max=338, avg=163.79, stdev=37.66
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[  107], 10.00th=[  117], 20.00th=[  134],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  163], 60.00th=[  171],
     | 70.00th=[  182], 80.00th=[  192], 90.00th=[  211], 95.00th=[  228],
     | 99.00th=[  259], 99.50th=[  268], 99.90th=[  330], 99.95th=[  330],
     | 99.99th=[  338]
   bw (  KiB/s): min=  263, max=  744, per=25.07%, avg=390.02, stdev=80.81, samples=52
   iops        : min=   65, max=  186, avg=97.42, stdev=20.25, samples=52
  lat (msec)   : 20=0.08%, 50=0.08%, 100=3.32%, 250=94.92%, 500=1.60%
  cpu          : usr=0.13%, sys=0.82%, ctx=1990, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23630: Wed Jan 15 19:26:56 2025
  write: IOPS=97, BW=389KiB/s (399kB/s)(10.0MiB/26305msec)
    slat (usec): min=11, max=1327, avg=70.91, stdev=51.73
    clat (msec): min=52, max=290, avg=164.25, stdev=36.55
     lat (msec): min=52, max=290, avg=164.32, stdev=36.56
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[  107], 10.00th=[  118], 20.00th=[  134],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  163], 60.00th=[  171],
     | 70.00th=[  182], 80.00th=[  194], 90.00th=[  213], 95.00th=[  230],
     | 99.00th=[  255], 99.50th=[  262], 99.90th=[  288], 99.95th=[  288],
     | 99.99th=[  292]
   bw (  KiB/s): min=  248, max=  640, per=24.98%, avg=388.65, stdev=74.30, samples=52
   iops        : min=   62, max=  160, avg=97.10, stdev=18.58, samples=52
  lat (msec)   : 100=3.32%, 250=95.27%, 500=1.41%
  cpu          : usr=0.19%, sys=0.72%, ctx=2351, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23631: Wed Jan 15 19:26:56 2025
  write: IOPS=97, BW=389KiB/s (399kB/s)(10.0MiB/26305msec)
    slat (usec): min=11, max=471, avg=68.90, stdev=39.98
    clat (msec): min=56, max=290, avg=164.24, stdev=36.46
     lat (msec): min=57, max=290, avg=164.31, stdev=36.46
    clat percentiles (msec):
     |  1.00th=[   82],  5.00th=[  107], 10.00th=[  118], 20.00th=[  134],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  163], 60.00th=[  174],
     | 70.00th=[  182], 80.00th=[  194], 90.00th=[  211], 95.00th=[  228],
     | 99.00th=[  255], 99.50th=[  266], 99.90th=[  288], 99.95th=[  292],
     | 99.99th=[  292]
   bw (  KiB/s): min=  272, max=  640, per=24.98%, avg=388.75, stdev=74.51, samples=52
   iops        : min=   68, max=  160, avg=97.17, stdev=18.63, samples=52
  lat (msec)   : 100=3.36%, 250=95.20%, 500=1.45%
  cpu          : usr=0.14%, sys=0.82%, ctx=2080, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23632: Wed Jan 15 19:26:56 2025
  write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26323msec)
    slat (usec): min=10, max=989, avg=69.50, stdev=45.56
    clat (msec): min=68, max=286, avg=164.36, stdev=36.15
     lat (msec): min=68, max=286, avg=164.43, stdev=36.16
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[  108], 10.00th=[  118], 20.00th=[  134],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  163], 60.00th=[  171],
     | 70.00th=[  182], 80.00th=[  192], 90.00th=[  211], 95.00th=[  230],
     | 99.00th=[  255], 99.50th=[  264], 99.90th=[  288], 99.95th=[  288],
     | 99.99th=[  288]
   bw (  KiB/s): min=  255, max=  640, per=24.95%, avg=388.17, stdev=75.18, samples=52
   iops        : min=   63, max=  160, avg=96.96, stdev=18.85, samples=52
  lat (msec)   : 100=3.01%, 250=95.35%, 500=1.64%
  cpu          : usr=0.16%, sys=0.78%, ctx=2166, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1556KiB/s (1593kB/s), 389KiB/s-391KiB/s (398kB/s-400kB/s), io=40.0MiB (41.9MB), run=26219-26323msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23662: Wed Jan 15 19:27:11 2025
   read: IOPS=86, BW=345KiB/s (353kB/s)(5048KiB/14632msec)
    slat (usec): min=487, max=2519, avg=839.32, stdev=222.52
    clat (msec): min=9, max=101, avg=59.75, stdev=11.22
     lat (msec): min=10, max=102, avg=60.59, stdev=11.22
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   44], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   55], 40.00th=[   58], 50.00th=[   61], 60.00th=[   64],
     | 70.00th=[   66], 80.00th=[   69], 90.00th=[   74], 95.00th=[   78],
     | 99.00th=[   84], 99.50th=[   88], 99.90th=[   92], 99.95th=[  103],
     | 99.99th=[  103]
   bw (  KiB/s): min=  216, max=  456, per=25.26%, avg=345.03, stdev=62.04, samples=29
   iops        : min=   54, max=  114, avg=86.21, stdev=15.46, samples=29
  write: IOPS=88, BW=355KiB/s (363kB/s)(5192KiB/14632msec)
    slat (usec): min=12, max=941, avg=59.01, stdev=49.75
    clat (msec): min=51, max=191, avg=121.16, stdev=26.43
     lat (msec): min=51, max=191, avg=121.22, stdev=26.44
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   80], 10.00th=[   87], 20.00th=[   99],
     | 30.00th=[  107], 40.00th=[  113], 50.00th=[  122], 60.00th=[  127],
     | 70.00th=[  134], 80.00th=[  146], 90.00th=[  159], 95.00th=[  167],
     | 99.00th=[  184], 99.50th=[  186], 99.90th=[  192], 99.95th=[  192],
     | 99.99th=[  192]
   bw (  KiB/s): min=  304, max=  392, per=24.85%, avg=353.31, stdev=27.11, samples=29
   iops        : min=   76, max=   98, avg=88.28, stdev= 6.82, samples=29
  lat (msec)   : 10=0.12%, 20=0.31%, 50=8.48%, 100=51.60%, 250=39.49%
  cpu          : usr=0.27%, sys=1.72%, ctx=2899, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23663: Wed Jan 15 19:27:11 2025
   read: IOPS=84, BW=339KiB/s (347kB/s)(4976KiB/14671msec)
    slat (usec): min=513, max=3487, avg=835.81, stdev=249.97
    clat (usec): min=896, max=122659, avg=59917.42, stdev=11545.42
     lat (msec): min=2, max=123, avg=60.76, stdev=11.55
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   44], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   57], 50.00th=[   61], 60.00th=[   63],
     | 70.00th=[   66], 80.00th=[   70], 90.00th=[   74], 95.00th=[   78],
     | 99.00th=[   87], 99.50th=[   89], 99.90th=[  106], 99.95th=[  123],
     | 99.99th=[  123]
   bw (  KiB/s): min=  208, max=  440, per=24.68%, avg=337.07, stdev=66.35, samples=29
   iops        : min=   52, max=  110, avg=84.24, stdev=16.57, samples=29
  write: IOPS=89, BW=359KiB/s (367kB/s)(5264KiB/14671msec)
    slat (usec): min=11, max=338, avg=58.76, stdev=36.60
    clat (msec): min=55, max=200, avg=120.68, stdev=27.01
     lat (msec): min=55, max=200, avg=120.74, stdev=27.02
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   78], 10.00th=[   87], 20.00th=[   96],
     | 30.00th=[  107], 40.00th=[  113], 50.00th=[  121], 60.00th=[  127],
     | 70.00th=[  136], 80.00th=[  144], 90.00th=[  157], 95.00th=[  167],
     | 99.00th=[  188], 99.50th=[  194], 99.90th=[  197], 99.95th=[  201],
     | 99.99th=[  201]
   bw (  KiB/s): min=  296, max=  440, per=25.12%, avg=357.21, stdev=39.40, samples=29
   iops        : min=   74, max=  110, avg=89.28, stdev= 9.82, samples=29
  lat (usec)   : 1000=0.04%
  lat (msec)   : 10=0.08%, 20=0.23%, 50=9.38%, 100=50.78%, 250=39.49%
  cpu          : usr=0.25%, sys=1.69%, ctx=2995, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23664: Wed Jan 15 19:27:11 2025
   read: IOPS=84, BW=339KiB/s (348kB/s)(4984KiB/14685msec)
    slat (usec): min=483, max=3349, avg=849.36, stdev=247.61
    clat (usec): min=1742, max=99085, avg=59542.89, stdev=11125.30
     lat (usec): min=2574, max=99780, avg=60394.11, stdev=11132.60
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[43254], 10.00th=[46924], 20.00th=[50070],
     | 30.00th=[53740], 40.00th=[56886], 50.00th=[59507], 60.00th=[62653],
     | 70.00th=[65799], 80.00th=[68682], 90.00th=[72877], 95.00th=[77071],
     | 99.00th=[83362], 99.50th=[87557], 99.90th=[98042], 99.95th=[99091],
     | 99.99th=[99091]
   bw (  KiB/s): min=  264, max=  448, per=24.92%, avg=340.38, stdev=50.43, samples=29
   iops        : min=   66, max=  112, avg=85.07, stdev=12.57, samples=29
  write: IOPS=89, BW=358KiB/s (367kB/s)(5256KiB/14685msec)
    slat (usec): min=11, max=1026, avg=59.47, stdev=43.83
    clat (msec): min=51, max=198, avg=121.27, stdev=27.16
     lat (msec): min=51, max=198, avg=121.33, stdev=27.16
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   80], 10.00th=[   88], 20.00th=[  100],
     | 30.00th=[  107], 40.00th=[  113], 50.00th=[  121], 60.00th=[  128],
     | 70.00th=[  136], 80.00th=[  144], 90.00th=[  159], 95.00th=[  169],
     | 99.00th=[  186], 99.50th=[  190], 99.90th=[  199], 99.95th=[  199],
     | 99.99th=[  199]
   bw (  KiB/s): min=  248, max=  456, per=24.89%, avg=353.90, stdev=42.85, samples=29
   iops        : min=   62, max=  114, avg=88.45, stdev=10.72, samples=29
  lat (msec)   : 2=0.04%, 10=0.20%, 20=0.16%, 50=8.98%, 100=50.74%
  lat (msec)   : 250=39.88%
  cpu          : usr=0.25%, sys=1.75%, ctx=3006, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23665: Wed Jan 15 19:27:11 2025
   read: IOPS=86, BW=347KiB/s (355kB/s)(5060KiB/14603msec)
    slat (usec): min=22, max=3673, avg=843.57, stdev=257.65
    clat (usec): min=819, max=111422, avg=59531.87, stdev=11600.70
     lat (usec): min=1713, max=112370, avg=60377.17, stdev=11619.64
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   44], 10.00th=[   46], 20.00th=[   51],
     | 30.00th=[   55], 40.00th=[   58], 50.00th=[   61], 60.00th=[   63],
     | 70.00th=[   66], 80.00th=[   69], 90.00th=[   73], 95.00th=[   78],
     | 99.00th=[   86], 99.50th=[   91], 99.90th=[  101], 99.95th=[  112],
     | 99.99th=[  112]
   bw (  KiB/s): min=  200, max=  440, per=25.42%, avg=347.28, stdev=55.22, samples=29
   iops        : min=   50, max=  110, avg=86.79, stdev=13.82, samples=29
  write: IOPS=88, BW=355KiB/s (363kB/s)(5180KiB/14603msec)
    slat (usec): min=11, max=422, avg=56.24, stdev=35.34
    clat (msec): min=18, max=214, avg=121.27, stdev=27.94
     lat (msec): min=18, max=214, avg=121.33, stdev=27.94
    clat percentiles (msec):
     |  1.00th=[   56],  5.00th=[   78], 10.00th=[   87], 20.00th=[   99],
     | 30.00th=[  107], 40.00th=[  113], 50.00th=[  122], 60.00th=[  129],
     | 70.00th=[  136], 80.00th=[  144], 90.00th=[  157], 95.00th=[  169],
     | 99.00th=[  190], 99.50th=[  192], 99.90th=[  215], 99.95th=[  215],
     | 99.99th=[  215]
   bw (  KiB/s): min=  272, max=  400, per=24.81%, avg=352.79, stdev=39.59, samples=29
   iops        : min=   68, max=  100, avg=88.17, stdev= 9.88, samples=29
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.08%, 4=0.08%, 10=0.12%, 20=0.20%, 50=8.79%
  lat (msec)   : 100=51.56%, 250=39.14%
  cpu          : usr=0.33%, sys=1.62%, ctx=2864, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1367KiB/s (1399kB/s), 339KiB/s-347KiB/s (347kB/s-355kB/s), io=19.6MiB (20.5MB), run=14603-14685msec
  WRITE: bw=1423KiB/s (1457kB/s), 355KiB/s-359KiB/s (363kB/s-367kB/s), io=20.4MiB (21.4MB), run=14603-14685msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23681: Wed Jan 15 19:27:13 2025
   read: IOPS=13, BW=541MiB/s (567MB/s)(40.0MiB/74msec)
    slat (nsec): min=9913.2k, max=9913.2k, avg=9913173.00, stdev= 0.00
    clat (nsec): min=60251k, max=60251k, avg=60251137.00, stdev= 0.00
     lat (nsec): min=70187k, max=70187k, avg=70186663.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[60031],  5.00th=[60031], 10.00th=[60031], 20.00th=[60031],
     | 30.00th=[60031], 40.00th=[60031], 50.00th=[60031], 60.00th=[60031],
     | 70.00th=[60031], 80.00th=[60031], 90.00th=[60031], 95.00th=[60031],
     | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=12.33%, ctx=14, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=541MiB/s (567MB/s), 541MiB/s-541MiB/s (567MB/s-567MB/s), io=40.0MiB (41.9MB), run=74-74msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck