== sanity test 398c: run fio to test AIO ================= 15:46:21 (1726861581)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.118645 s, 354 MB/s
osc.lustre-OST0000-osc-ffff88012a4c9000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=11145: Fri Sep 20 15:46:57 2024
  write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34441msec)
    slat (usec): min=14, max=641, avg=85.15, stdev=50.87
    clat (msec): min=17, max=378, avg=214.80, stdev=53.53
     lat (msec): min=17, max=378, avg=214.88, stdev=53.52
    clat percentiles (msec):
     |  1.00th=[  114],  5.00th=[  140], 10.00th=[  148], 20.00th=[  165],
     | 30.00th=[  180], 40.00th=[  194], 50.00th=[  211], 60.00th=[  226],
     | 70.00th=[  245], 80.00th=[  264], 90.00th=[  292], 95.00th=[  309],
     | 99.00th=[  338], 99.50th=[  347], 99.90th=[  380], 99.95th=[  380],
     | 99.99th=[  380]
   bw (  KiB/s): min=  176, max=  448, per=24.94%, avg=296.25, stdev=67.99, samples=68
   iops        : min=   44, max=  112, avg=73.99, stdev=16.97, samples=68
  lat (msec)   : 20=0.04%, 100=0.66%, 250=72.23%, 500=27.07%
  cpu          : usr=0.09%, sys=0.78%, ctx=2250, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11146: Fri Sep 20 15:46:57 2024
  write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34460msec)
    slat (usec): min=15, max=571, avg=87.30, stdev=52.74
    clat (msec): min=18, max=378, avg=214.95, stdev=53.48
     lat (msec): min=18, max=378, avg=215.04, stdev=53.47
    clat percentiles (msec):
     |  1.00th=[  114],  5.00th=[  140], 10.00th=[  148], 20.00th=[  167],
     | 30.00th=[  180], 40.00th=[  194], 50.00th=[  211], 60.00th=[  228],
     | 70.00th=[  245], 80.00th=[  264], 90.00th=[  292], 95.00th=[  309],
     | 99.00th=[  342], 99.50th=[  347], 99.90th=[  363], 99.95th=[  368],
     | 99.99th=[  380]
   bw (  KiB/s): min=  184, max=  448, per=24.85%, avg=295.22, stdev=65.55, samples=68
   iops        : min=   46, max=  112, avg=73.75, stdev=16.37, samples=68
  lat (msec)   : 20=0.04%, 50=0.04%, 100=0.70%, 250=72.19%, 500=27.03%
  cpu          : usr=0.13%, sys=0.73%, ctx=2526, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11147: Fri Sep 20 15:46:57 2024
  write: IOPS=74, BW=298KiB/s (305kB/s)(10.0MiB/34420msec)
    slat (usec): min=14, max=658, avg=83.68, stdev=49.79
    clat (msec): min=39, max=374, avg=214.81, stdev=55.17
     lat (msec): min=39, max=374, avg=214.89, stdev=55.16
    clat percentiles (msec):
     |  1.00th=[  105],  5.00th=[  138], 10.00th=[  148], 20.00th=[  165],
     | 30.00th=[  180], 40.00th=[  194], 50.00th=[  211], 60.00th=[  228],
     | 70.00th=[  245], 80.00th=[  264], 90.00th=[  292], 95.00th=[  309],
     | 99.00th=[  347], 99.50th=[  359], 99.90th=[  368], 99.95th=[  368],
     | 99.99th=[  376]
   bw (  KiB/s): min=  160, max=  504, per=25.00%, avg=296.99, stdev=70.64, samples=68
   iops        : min=   40, max=  126, avg=74.19, stdev=17.66, samples=68
  lat (msec)   : 50=0.47%, 100=0.43%, 250=71.37%, 500=27.73%
  cpu          : usr=0.14%, sys=0.73%, ctx=2393, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11148: Fri Sep 20 15:46:57 2024
  write: IOPS=74, BW=297KiB/s (304kB/s)(10.0MiB/34460msec)
    slat (usec): min=14, max=422, avg=82.06, stdev=46.05
    clat (msec): min=85, max=378, avg=215.04, stdev=53.13
     lat (msec): min=85, max=378, avg=215.12, stdev=53.12
    clat percentiles (msec):
     |  1.00th=[  110],  5.00th=[  140], 10.00th=[  150], 20.00th=[  167],
     | 30.00th=[  180], 40.00th=[  194], 50.00th=[  209], 60.00th=[  228],
     | 70.00th=[  247], 80.00th=[  266], 90.00th=[  288], 95.00th=[  309],
     | 99.00th=[  342], 99.50th=[  351], 99.90th=[  376], 99.95th=[  376],
     | 99.99th=[  380]
   bw (  KiB/s): min=  176, max=  416, per=24.93%, avg=296.19, stdev=65.12, samples=68
   iops        : min=   44, max=  104, avg=74.01, stdev=16.25, samples=68
  lat (msec)   : 100=0.08%, 250=72.70%, 500=27.23%
  cpu          : usr=0.10%, sys=0.76%, ctx=2091, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1189KiB/s (1217kB/s), 297KiB/s-298KiB/s (304kB/s-305kB/s), io=40.0MiB (41.9MB), run=34420-34460msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=11186: Fri Sep 20 15:47:16 2024
   read: IOPS=72, BW=289KiB/s (296kB/s)(5048KiB/17471msec)
    slat (usec): min=882, max=4426, avg=1470.07, stdev=331.84
    clat (usec): min=1971, max=155238, avg=72820.95, stdev=14314.12
     lat (msec): min=3, max=156, avg=74.29, stdev=14.36
    clat percentiles (msec):
     |  1.00th=[   28],  5.00th=[   54], 10.00th=[   59], 20.00th=[   63],
     | 30.00th=[   66], 40.00th=[   70], 50.00th=[   73], 60.00th=[   77],
     | 70.00th=[   80], 80.00th=[   83], 90.00th=[   89], 95.00th=[   92],
     | 99.00th=[  116], 99.50th=[  131], 99.90th=[  144], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  200, max=  392, per=25.12%, avg=287.88, stdev=58.28, samples=34
   iops        : min=   50, max=   98, avg=71.88, stdev=14.57, samples=34
  write: IOPS=74, BW=297KiB/s (304kB/s)(5192KiB/17471msec)
    slat (usec): min=15, max=1870, avg=90.14, stdev=81.07
    clat (msec): min=67, max=249, avg=142.52, stdev=32.20
     lat (msec): min=67, max=249, avg=142.61, stdev=32.21
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[   99], 10.00th=[  105], 20.00th=[  114],
     | 30.00th=[  124], 40.00th=[  131], 50.00th=[  138], 60.00th=[  148],
     | 70.00th=[  157], 80.00th=[  171], 90.00th=[  190], 95.00th=[  203],
     | 99.00th=[  224], 99.50th=[  226], 99.90th=[  239], 99.95th=[  249],
     | 99.99th=[  249]
   bw (  KiB/s): min=  208, max=  360, per=24.66%, avg=294.24, stdev=33.35, samples=34
   iops        : min=   52, max=   90, avg=73.47, stdev= 8.34, samples=34
  lat (msec)   : 2=0.04%, 4=0.04%, 20=0.39%, 50=1.13%, 100=50.00%
  lat (msec)   : 250=48.40%
  cpu          : usr=0.36%, sys=2.40%, ctx=3053, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11187: Fri Sep 20 15:47:16 2024
   read: IOPS=71, BW=285KiB/s (291kB/s)(4976KiB/17489msec)
    slat (usec): min=725, max=4439, avg=1470.97, stdev=319.04
    clat (msec): min=2, max=150, avg=72.47, stdev=14.42
     lat (msec): min=3, max=152, avg=73.95, stdev=14.46
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   52], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   66], 40.00th=[   69], 50.00th=[   72], 60.00th=[   75],
     | 70.00th=[   79], 80.00th=[   83], 90.00th=[   88], 95.00th=[   93],
     | 99.00th=[  124], 99.50th=[  138], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  184, max=  424, per=24.63%, avg=282.24, stdev=58.72, samples=34
   iops        : min=   46, max=  106, avg=70.47, stdev=14.74, samples=34
  write: IOPS=75, BW=301KiB/s (308kB/s)(5264KiB/17489msec)
    slat (usec): min=15, max=343, avg=87.41, stdev=41.36
    clat (msec): min=68, max=248, avg=142.17, stdev=33.11
     lat (msec): min=68, max=248, avg=142.26, stdev=33.11
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[   94], 10.00th=[  104], 20.00th=[  113],
     | 30.00th=[  122], 40.00th=[  130], 50.00th=[  138], 60.00th=[  148],
     | 70.00th=[  157], 80.00th=[  171], 90.00th=[  190], 95.00th=[  203],
     | 99.00th=[  226], 99.50th=[  230], 99.90th=[  241], 99.95th=[  249],
     | 99.99th=[  249]
   bw (  KiB/s): min=  216, max=  352, per=25.06%, avg=298.94, stdev=32.33, samples=34
   iops        : min=   54, max=   88, avg=74.65, stdev= 8.02, samples=34
  lat (msec)   : 4=0.04%, 20=0.27%, 50=1.80%, 100=49.45%, 250=48.44%
  cpu          : usr=0.39%, sys=2.38%, ctx=2926, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11188: Fri Sep 20 15:47:16 2024
   read: IOPS=71, BW=285KiB/s (292kB/s)(4984KiB/17500msec)
    slat (usec): min=816, max=4380, avg=1481.76, stdev=329.83
    clat (usec): min=1628, max=141115, avg=73477.85, stdev=13845.03
     lat (msec): min=3, max=143, avg=74.96, stdev=13.92
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   54], 10.00th=[   59], 20.00th=[   64],
     | 30.00th=[   67], 40.00th=[   70], 50.00th=[   73], 60.00th=[   77],
     | 70.00th=[   80], 80.00th=[   84], 90.00th=[   90], 95.00th=[   94],
     | 99.00th=[  125], 99.50th=[  129], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  192, max=  432, per=24.79%, avg=284.12, stdev=57.83, samples=34
   iops        : min=   48, max=  108, avg=70.94, stdev=14.44, samples=34
  write: IOPS=75, BW=300KiB/s (308kB/s)(5256KiB/17500msec)
    slat (usec): min=16, max=1441, avg=88.13, stdev=60.53
    clat (msec): min=50, max=249, avg=141.58, stdev=33.56
     lat (msec): min=50, max=249, avg=141.67, stdev=33.56
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[   94], 10.00th=[  102], 20.00th=[  112],
     | 30.00th=[  121], 40.00th=[  129], 50.00th=[  140], 60.00th=[  148],
     | 70.00th=[  159], 80.00th=[  171], 90.00th=[  188], 95.00th=[  203],
     | 99.00th=[  226], 99.50th=[  236], 99.90th=[  249], 99.95th=[  249],
     | 99.99th=[  249]
   bw (  KiB/s): min=  223, max=  392, per=24.74%, avg=295.18, stdev=40.90, samples=34
   iops        : min=   55, max=   98, avg=73.71, stdev=10.26, samples=34
  lat (msec)   : 2=0.04%, 20=0.20%, 50=1.25%, 100=50.66%, 250=47.85%
  cpu          : usr=0.34%, sys=2.42%, ctx=2869, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11189: Fri Sep 20 15:47:16 2024
   read: IOPS=72, BW=291KiB/s (298kB/s)(5060KiB/17364msec)
    slat (usec): min=28, max=4584, avg=1466.01, stdev=341.56
    clat (usec): min=1987, max=147082, avg=72472.40, stdev=14761.54
     lat (msec): min=3, max=148, avg=73.94, stdev=14.83
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   52], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   66], 40.00th=[   69], 50.00th=[   72], 60.00th=[   77],
     | 70.00th=[   80], 80.00th=[   84], 90.00th=[   90], 95.00th=[   94],
     | 99.00th=[  112], 99.50th=[  122], 99.90th=[  142], 99.95th=[  148],
     | 99.99th=[  148]
   bw (  KiB/s): min=  176, max=  391, per=25.45%, avg=291.65, stdev=52.86, samples=34
   iops        : min=   44, max=   97, avg=72.82, stdev=13.17, samples=34
  write: IOPS=74, BW=298KiB/s (305kB/s)(5180KiB/17364msec)
    slat (usec): min=18, max=357, avg=88.38, stdev=40.08
    clat (msec): min=27, max=241, avg=142.04, stdev=33.95
     lat (msec): min=27, max=241, avg=142.13, stdev=33.96
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[   94], 10.00th=[  102], 20.00th=[  113],
     | 30.00th=[  123], 40.00th=[  130], 50.00th=[  138], 60.00th=[  148],
     | 70.00th=[  159], 80.00th=[  174], 90.00th=[  190], 95.00th=[  205],
     | 99.00th=[  226], 99.50th=[  228], 99.90th=[  239], 99.95th=[  241],
     | 99.99th=[  241]
   bw (  KiB/s): min=  247, max=  376, per=24.86%, avg=296.59, stdev=31.02, samples=34
   iops        : min=   61, max=   94, avg=74.06, stdev= 7.84, samples=34
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.12%, 20=0.23%, 50=1.56%
  lat (msec)   : 100=50.35%, 250=47.58%
  cpu          : usr=0.30%, sys=2.52%, ctx=3070, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1147KiB/s (1174kB/s), 285KiB/s-291KiB/s (291kB/s-298kB/s), io=19.6MiB (20.5MB), run=17364-17500msec
  WRITE: bw=1194KiB/s (1222kB/s), 297KiB/s-301KiB/s (304kB/s-308kB/s), io=20.4MiB (21.4MB), run=17364-17500msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=11207: Fri Sep 20 15:47:17 2024
   read: IOPS=6, BW=272MiB/s (285MB/s)(40.0MiB/147msec)
    slat (nsec): min=5468.3k, max=5468.3k, avg=5468297.00, stdev= 0.00
    clat (nsec): min=134893k, max=134893k, avg=134893429.00, stdev= 0.00
     lat (nsec): min=140371k, max=140371k, avg=140371476.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  136], 10.00th=[  136], 20.00th=[  136],
     | 30.00th=[  136], 40.00th=[  136], 50.00th=[  136], 60.00th=[  136],
     | 70.00th=[  136], 80.00th=[  136], 90.00th=[  136], 95.00th=[  136],
     | 99.00th=[  136], 99.50th=[  136], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.11%, ctx=17, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=40.0MiB (41.9MB), run=147-147msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout