== sanity test 398c: run fio to test AIO ================= 06:49:34 (1726310974)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.10022 s, 419 MB/s
osc.lustre-OST0000-osc-ffff8800a8267000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=25282: Sat Sep 14 06:50:06 2024
  write: IOPS=85, BW=341KiB/s (349kB/s)(10.0MiB/30044msec)
    slat (usec): min=14, max=591, avg=75.87, stdev=48.02
    clat (msec): min=25, max=307, avg=187.52, stdev=37.93
     lat (msec): min=25, max=307, avg=187.60, stdev=37.93
    clat percentiles (msec):
     |  1.00th=[  105],  5.00th=[  129], 10.00th=[  142], 20.00th=[  157],
     | 30.00th=[  167], 40.00th=[  178], 50.00th=[  188], 60.00th=[  197],
     | 70.00th=[  207], 80.00th=[  220], 90.00th=[  234], 95.00th=[  251],
     | 99.00th=[  284], 99.50th=[  292], 99.90th=[  309], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  KiB/s): min=  240, max=  504, per=24.95%, avg=339.30, stdev=54.51, samples=60
   iops        : min=   60, max=  126, avg=84.80, stdev=13.65, samples=60
  lat (msec)   : 50=0.27%, 100=0.51%, 250=93.63%, 500=5.59%
  cpu          : usr=0.13%, sys=0.81%, ctx=2611, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25283: Sat Sep 14 06:50:06 2024
  write: IOPS=85, BW=340KiB/s (349kB/s)(10.0MiB/30087msec)
    slat (usec): min=13, max=740, avg=71.92, stdev=45.62
    clat (msec): min=70, max=313, avg=187.76, stdev=37.20
     lat (msec): min=70, max=313, avg=187.84, stdev=37.20
    clat percentiles (msec):
     |  1.00th=[  105],  5.00th=[  129], 10.00th=[  142], 20.00th=[  157],
     | 30.00th=[  167], 40.00th=[  178], 50.00th=[  188], 60.00th=[  194],
     | 70.00th=[  207], 80.00th=[  218], 90.00th=[  236], 95.00th=[  253],
     | 99.00th=[  284], 99.50th=[  292], 99.90th=[  309], 99.95th=[  313],
     | 99.99th=[  313]
   bw (  KiB/s): min=  247, max=  512, per=24.94%, avg=339.25, stdev=52.81, samples=60
   iops        : min=   61, max=  128, avg=84.75, stdev=13.22, samples=60
  lat (msec)   : 100=0.74%, 250=93.83%, 500=5.43%
  cpu          : usr=0.14%, sys=0.76%, ctx=2229, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25284: Sat Sep 14 06:50:06 2024
  write: IOPS=85, BW=340KiB/s (348kB/s)(10.0MiB/30098msec)
    slat (usec): min=13, max=467, avg=73.52, stdev=44.79
    clat (msec): min=73, max=325, avg=187.73, stdev=37.22
     lat (msec): min=73, max=325, avg=187.80, stdev=37.22
    clat percentiles (msec):
     |  1.00th=[  105],  5.00th=[  132], 10.00th=[  142], 20.00th=[  157],
     | 30.00th=[  167], 40.00th=[  176], 50.00th=[  188], 60.00th=[  197],
     | 70.00th=[  207], 80.00th=[  218], 90.00th=[  239], 95.00th=[  253],
     | 99.00th=[  279], 99.50th=[  292], 99.90th=[  326], 99.95th=[  326],
     | 99.99th=[  326]
   bw (  KiB/s): min=  240, max=  512, per=24.94%, avg=339.25, stdev=51.36, samples=60
   iops        : min=   60, max=  128, avg=84.75, stdev=12.88, samples=60
  lat (msec)   : 100=0.62%, 250=93.48%, 500=5.90%
  cpu          : usr=0.16%, sys=0.79%, ctx=2372, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25285: Sat Sep 14 06:50:06 2024
  write: IOPS=85, BW=340KiB/s (348kB/s)(10.0MiB/30116msec)
    slat (usec): min=13, max=303, avg=68.28, stdev=40.32
    clat (msec): min=72, max=325, avg=187.77, stdev=36.96
     lat (msec): min=72, max=325, avg=187.84, stdev=36.97
    clat percentiles (msec):
     |  1.00th=[  114],  5.00th=[  128], 10.00th=[  140], 20.00th=[  157],
     | 30.00th=[  167], 40.00th=[  178], 50.00th=[  188], 60.00th=[  197],
     | 70.00th=[  207], 80.00th=[  218], 90.00th=[  234], 95.00th=[  253],
     | 99.00th=[  288], 99.50th=[  296], 99.90th=[  309], 99.95th=[  309],
     | 99.99th=[  326]
   bw (  KiB/s): min=  208, max=  416, per=24.87%, avg=338.23, stdev=49.97, samples=60
   iops        : min=   52, max=  104, avg=84.53, stdev=12.50, samples=60
  lat (msec)   : 100=0.31%, 250=94.26%, 500=5.43%
  cpu          : usr=0.14%, sys=0.74%, ctx=2120, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1360KiB/s (1393kB/s), 340KiB/s-341KiB/s (348kB/s-349kB/s), io=40.0MiB (41.9MB), run=30044-30116msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=25319: Sat Sep 14 06:50:23 2024
   read: IOPS=77, BW=309KiB/s (317kB/s)(5048KiB/16332msec)
    slat (usec): min=655, max=3236, avg=1210.03, stdev=325.61
    clat (usec): min=1771, max=161628, avg=65611.55, stdev=14120.69
     lat (msec): min=2, max=162, avg=66.82, stdev=14.19
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   48], 10.00th=[   52], 20.00th=[   56],
     | 30.00th=[   59], 40.00th=[   63], 50.00th=[   65], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   75], 90.00th=[   82], 95.00th=[   87],
     | 99.00th=[  112], 99.50th=[  127], 99.90th=[  150], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  191, max=  424, per=25.15%, avg=307.59, stdev=55.65, samples=32
   iops        : min=   47, max=  106, avg=76.78, stdev=13.97, samples=32
  write: IOPS=79, BW=318KiB/s (326kB/s)(5192KiB/16332msec)
    slat (usec): min=14, max=232, avg=57.45, stdev=29.30
    clat (msec): min=71, max=234, avg=135.87, stdev=29.64
     lat (msec): min=71, max=234, avg=135.92, stdev=29.64
    clat percentiles (msec):
     |  1.00th=[   80],  5.00th=[   91], 10.00th=[  100], 20.00th=[  110],
     | 30.00th=[  118], 40.00th=[  127], 50.00th=[  134], 60.00th=[  142],
     | 70.00th=[  153], 80.00th=[  161], 90.00th=[  176], 95.00th=[  186],
     | 99.00th=[  218], 99.50th=[  228], 99.90th=[  234], 99.95th=[  236],
     | 99.99th=[  236]
   bw (  KiB/s): min=  224, max=  392, per=24.83%, avg=316.09, stdev=43.01, samples=32
   iops        : min=   56, max=   98, avg=78.91, stdev=10.76, samples=32
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.04%, 20=0.20%, 50=3.52%
  lat (msec)   : 100=49.92%, 250=46.17%
  cpu          : usr=0.17%, sys=1.59%, ctx=3559, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25320: Sat Sep 14 06:50:23 2024
   read: IOPS=75, BW=304KiB/s (311kB/s)(4976KiB/16370msec)
    slat (usec): min=37, max=4701, avg=1196.35, stdev=333.00
    clat (usec): min=1743, max=119026, avg=64730.07, stdev=13489.72
     lat (msec): min=3, max=123, avg=65.93, stdev=13.57
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   47], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   59], 40.00th=[   62], 50.00th=[   65], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   75], 90.00th=[   81], 95.00th=[   85],
     | 99.00th=[  107], 99.50th=[  111], 99.90th=[  118], 99.95th=[  120],
     | 99.99th=[  120]
   bw (  KiB/s): min=  160, max=  448, per=24.61%, avg=300.94, stdev=60.90, samples=32
   iops        : min=   40, max=  112, avg=75.19, stdev=15.27, samples=32
  write: IOPS=80, BW=322KiB/s (329kB/s)(5264KiB/16370msec)
    slat (usec): min=15, max=290, avg=58.85, stdev=30.72
    clat (msec): min=29, max=232, avg=136.50, stdev=30.08
     lat (msec): min=29, max=232, avg=136.56, stdev=30.09
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   91], 10.00th=[   99], 20.00th=[  111],
     | 30.00th=[  120], 40.00th=[  128], 50.00th=[  136], 60.00th=[  146],
     | 70.00th=[  155], 80.00th=[  163], 90.00th=[  174], 95.00th=[  186],
     | 99.00th=[  211], 99.50th=[  218], 99.90th=[  230], 99.95th=[  232],
     | 99.99th=[  232]
   bw (  KiB/s): min=  224, max=  400, per=25.05%, avg=318.94, stdev=39.25, samples=32
   iops        : min=   56, max=  100, avg=79.69, stdev= 9.83, samples=32
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.16%, 20=0.20%, 50=4.53%
  lat (msec)   : 100=48.67%, 250=46.29%
  cpu          : usr=0.31%, sys=1.43%, ctx=3677, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25321: Sat Sep 14 06:50:23 2024
   read: IOPS=75, BW=304KiB/s (311kB/s)(4984KiB/16402msec)
    slat (usec): min=673, max=3134, avg=1209.57, stdev=328.54
    clat (usec): min=1413, max=117268, avg=64968.20, stdev=12952.05
     lat (msec): min=2, max=119, avg=66.18, stdev=13.04
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   46], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   59], 40.00th=[   63], 50.00th=[   65], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   75], 90.00th=[   82], 95.00th=[   86],
     | 99.00th=[  103], 99.50th=[  105], 99.90th=[  117], 99.95th=[  117],
     | 99.99th=[  117]
   bw (  KiB/s): min=  176, max=  504, per=24.75%, avg=302.69, stdev=68.03, samples=32
   iops        : min=   44, max=  126, avg=75.63, stdev=17.03, samples=32
  write: IOPS=80, BW=320KiB/s (328kB/s)(5256KiB/16402msec)
    slat (usec): min=14, max=465, avg=56.14, stdev=30.98
    clat (msec): min=32, max=230, avg=136.51, stdev=29.97
     lat (msec): min=32, max=230, avg=136.57, stdev=29.97
    clat percentiles (msec):
     |  1.00th=[   80],  5.00th=[   91], 10.00th=[  100], 20.00th=[  110],
     | 30.00th=[  120], 40.00th=[  127], 50.00th=[  136], 60.00th=[  144],
     | 70.00th=[  155], 80.00th=[  165], 90.00th=[  176], 95.00th=[  188],
     | 99.00th=[  211], 99.50th=[  218], 99.90th=[  230], 99.95th=[  232],
     | 99.99th=[  232]
   bw (  KiB/s): min=  216, max=  368, per=24.72%, avg=314.69, stdev=36.50, samples=32
   iops        : min=   54, max=   92, avg=78.63, stdev= 9.17, samples=32
  lat (msec)   : 2=0.04%, 4=0.04%, 20=0.20%, 50=4.69%, 100=48.59%
  lat (msec)   : 250=46.45%
  cpu          : usr=0.15%, sys=1.62%, ctx=3515, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25322: Sat Sep 14 06:50:23 2024
   read: IOPS=77, BW=309KiB/s (317kB/s)(5060KiB/16359msec)
    slat (usec): min=616, max=3575, avg=1197.98, stdev=329.69
    clat (usec): min=1332, max=117294, avg=65100.21, stdev=13235.09
     lat (msec): min=2, max=118, avg=66.30, stdev=13.32
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   47], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   59], 40.00th=[   63], 50.00th=[   66], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   75], 90.00th=[   81], 95.00th=[   86],
     | 99.00th=[  106], 99.50th=[  111], 99.90th=[  115], 99.95th=[  117],
     | 99.99th=[  117]
   bw (  KiB/s): min=  208, max=  464, per=25.28%, avg=309.12, stdev=67.96, samples=32
   iops        : min=   52, max=  116, avg=77.19, stdev=17.06, samples=32
  write: IOPS=79, BW=317KiB/s (324kB/s)(5180KiB/16359msec)
    slat (usec): min=13, max=938, avg=55.76, stdev=38.79
    clat (msec): min=56, max=230, avg=136.78, stdev=29.37
     lat (msec): min=56, max=230, avg=136.83, stdev=29.37
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[   92], 10.00th=[  101], 20.00th=[  110],
     | 30.00th=[  120], 40.00th=[  128], 50.00th=[  136], 60.00th=[  144],
     | 70.00th=[  155], 80.00th=[  163], 90.00th=[  176], 95.00th=[  184],
     | 99.00th=[  207], 99.50th=[  218], 99.90th=[  230], 99.95th=[  232],
     | 99.99th=[  232]
   bw (  KiB/s): min=  232, max=  376, per=24.64%, avg=313.63, stdev=37.88, samples=32
   iops        : min=   58, max=   94, avg=78.31, stdev= 9.46, samples=32
  lat (msec)   : 2=0.04%, 4=0.04%, 20=0.27%, 50=4.10%, 100=49.34%
  lat (msec)   : 250=46.21%
  cpu          : usr=0.17%, sys=1.57%, ctx=3642, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1224KiB/s (1253kB/s), 304KiB/s-309KiB/s (311kB/s-317kB/s), io=19.6MiB (20.5MB), run=16332-16402msec
  WRITE: bw=1274KiB/s (1304kB/s), 317KiB/s-322KiB/s (324kB/s-329kB/s), io=20.4MiB (21.4MB), run=16332-16402msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=25339: Sat Sep 14 06:50:25 2024
   read: IOPS=6, BW=268MiB/s (281MB/s)(40.0MiB/149msec)
    slat (nsec): min=6011.9k, max=6011.9k, avg=6011891.00, stdev= 0.00
    clat (nsec): min=135414k, max=135414k, avg=135413565.00, stdev= 0.00
     lat (nsec): min=141435k, max=141435k, avg=141435201.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  136], 10.00th=[  136], 20.00th=[  136],
     | 30.00th=[  136], 40.00th=[  136], 50.00th=[  136], 60.00th=[  136],
     | 70.00th=[  136], 80.00th=[  136], 90.00th=[  136], 95.00th=[  136],
     | 99.00th=[  136], 99.50th=[  136], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.05%, ctx=17, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=40.0MiB (41.9MB), run=149-149msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout