== sanity test 398c: run fio to test AIO ================= 02:40:54 (1715582454)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0841995 s, 498 MB/s
osc.lustre-OST0000-osc-ffff8800a87b9000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=2007: Mon May 13 02:41:23 2024
  write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27648msec)
    slat (usec): min=13, max=298, avg=55.58, stdev=31.51
    clat (msec): min=47, max=267, avg=172.59, stdev=32.29
     lat (msec): min=47, max=267, avg=172.65, stdev=32.29
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[  120], 10.00th=[  136], 20.00th=[  146],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  176], 60.00th=[  182],
     | 70.00th=[  188], 80.00th=[  199], 90.00th=[  211], 95.00th=[  226],
     | 99.00th=[  243], 99.50th=[  247], 99.90th=[  257], 99.95th=[  268],
     | 99.99th=[  268]
   bw (  KiB/s): min=  288, max=  640, per=24.94%, avg=368.93, stdev=55.95, samples=55
   iops        : min=   72, max=  160, avg=92.16, stdev=14.03, samples=55
  lat (msec)   : 50=0.27%, 100=1.84%, 250=97.66%, 500=0.23%
  cpu          : usr=0.14%, sys=0.55%, ctx=2507, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2008: Mon May 13 02:41:23 2024
  write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27659msec)
    slat (usec): min=14, max=345, avg=55.99, stdev=31.75
    clat (msec): min=35, max=278, avg=172.66, stdev=31.60
     lat (msec): min=35, max=278, avg=172.72, stdev=31.60
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[  122], 10.00th=[  136], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  176], 60.00th=[  182],
     | 70.00th=[  190], 80.00th=[  199], 90.00th=[  211], 95.00th=[  222],
     | 99.00th=[  241], 99.50th=[  245], 99.90th=[  257], 99.95th=[  268],
     | 99.99th=[  279]
   bw (  KiB/s): min=  296, max=  640, per=24.95%, avg=369.07, stdev=54.59, samples=55
   iops        : min=   74, max=  160, avg=92.20, stdev=13.68, samples=55
  lat (msec)   : 50=0.12%, 100=2.03%, 250=97.70%, 500=0.16%
  cpu          : usr=0.10%, sys=0.59%, ctx=2478, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2009: Mon May 13 02:41:23 2024
  write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27680msec)
    slat (usec): min=14, max=417, avg=55.69, stdev=33.76
    clat (msec): min=35, max=254, avg=172.80, stdev=32.20
     lat (msec): min=35, max=254, avg=172.86, stdev=32.20
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[  120], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  174], 60.00th=[  180],
     | 70.00th=[  190], 80.00th=[  201], 90.00th=[  213], 95.00th=[  224],
     | 99.00th=[  241], 99.50th=[  247], 99.90th=[  255], 99.95th=[  255],
     | 99.99th=[  255]
   bw (  KiB/s): min=  272, max=  640, per=24.88%, avg=367.91, stdev=57.76, samples=55
   iops        : min=   68, max=  160, avg=91.91, stdev=14.48, samples=55
  lat (msec)   : 50=0.08%, 100=2.11%, 250=97.54%, 500=0.27%
  cpu          : usr=0.09%, sys=0.59%, ctx=2433, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2010: Mon May 13 02:41:23 2024
  write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27615msec)
    slat (usec): min=13, max=257, avg=56.44, stdev=30.02
    clat (msec): min=18, max=294, avg=172.42, stdev=32.96
     lat (msec): min=18, max=294, avg=172.48, stdev=32.96
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[  121], 10.00th=[  136], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  176], 60.00th=[  182],
     | 70.00th=[  190], 80.00th=[  199], 90.00th=[  213], 95.00th=[  222],
     | 99.00th=[  241], 99.50th=[  247], 99.90th=[  271], 99.95th=[  271],
     | 99.99th=[  296]
   bw (  KiB/s): min=  272, max=  704, per=25.02%, avg=370.11, stdev=62.78, samples=55
   iops        : min=   68, max=  176, avg=92.47, stdev=15.74, samples=55
  lat (msec)   : 20=0.04%, 50=0.70%, 100=1.76%, 250=97.23%, 500=0.27%
  cpu          : usr=0.14%, sys=0.55%, ctx=2486, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1480KiB/s (1515kB/s), 370KiB/s-371KiB/s (379kB/s-380kB/s), io=40.0MiB (41.9MB), run=27615-27680msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=2042: Mon May 13 02:41:42 2024
   read: IOPS=71, BW=287KiB/s (294kB/s)(5048KiB/17576msec)
    slat (usec): min=709, max=3008, avg=1215.03, stdev=272.51
    clat (usec): min=1807, max=130057, avg=72798.88, stdev=13072.43
     lat (msec): min=3, max=131, avg=74.02, stdev=13.08
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   55], 10.00th=[   61], 20.00th=[   64],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   74], 60.00th=[   77],
     | 70.00th=[   80], 80.00th=[   83], 90.00th=[   88], 95.00th=[   91],
     | 99.00th=[  103], 99.50th=[  107], 99.90th=[  118], 99.95th=[  131],
     | 99.99th=[  131]
   bw (  KiB/s): min=  192, max=  408, per=25.15%, avg=286.26, stdev=53.17, samples=35
   iops        : min=   48, max=  102, avg=71.46, stdev=13.38, samples=35
  write: IOPS=73, BW=295KiB/s (302kB/s)(5192KiB/17576msec)
    slat (usec): min=15, max=588, avg=52.80, stdev=27.87
    clat (msec): min=53, max=251, avg=144.41, stdev=34.25
     lat (msec): min=53, max=251, avg=144.47, stdev=34.26
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[   93], 10.00th=[  103], 20.00th=[  115],
     | 30.00th=[  125], 40.00th=[  134], 50.00th=[  142], 60.00th=[  153],
     | 70.00th=[  161], 80.00th=[  174], 90.00th=[  192], 95.00th=[  205],
     | 99.00th=[  241], 99.50th=[  241], 99.90th=[  251], 99.95th=[  251],
     | 99.99th=[  251]
   bw (  KiB/s): min=  232, max=  368, per=24.83%, avg=294.26, stdev=38.54, samples=35
   iops        : min=   58, max=   92, avg=73.46, stdev= 9.60, samples=35
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.08%, 20=0.43%, 50=0.90%
  lat (msec)   : 100=51.76%, 250=46.72%, 500=0.04%
  cpu          : usr=0.19%, sys=1.27%, ctx=3407, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2043: Mon May 13 02:41:42 2024
   read: IOPS=70, BW=283KiB/s (290kB/s)(4976KiB/17599msec)
    slat (usec): min=13, max=3203, avg=1216.17, stdev=295.75
    clat (usec): min=1621, max=117874, avg=72259.49, stdev=12864.92
     lat (usec): min=1944, max=119044, avg=73477.50, stdev=12885.75
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   53], 10.00th=[   59], 20.00th=[   63],
     | 30.00th=[   66], 40.00th=[   70], 50.00th=[   73], 60.00th=[   75],
     | 70.00th=[   79], 80.00th=[   83], 90.00th=[   89], 95.00th=[   92],
     | 99.00th=[  102], 99.50th=[  103], 99.90th=[  108], 99.95th=[  118],
     | 99.99th=[  118]
   bw (  KiB/s): min=  160, max=  416, per=24.69%, avg=281.03, stdev=63.99, samples=35
   iops        : min=   40, max=  104, avg=70.17, stdev=16.03, samples=35
  write: IOPS=74, BW=299KiB/s (306kB/s)(5264KiB/17599msec)
    slat (usec): min=12, max=223, avg=53.04, stdev=24.66
    clat (msec): min=14, max=254, avg=144.34, stdev=35.32
     lat (msec): min=14, max=254, avg=144.40, stdev=35.32
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   93], 10.00th=[  101], 20.00th=[  114],
     | 30.00th=[  127], 40.00th=[  136], 50.00th=[  142], 60.00th=[  153],
     | 70.00th=[  161], 80.00th=[  174], 90.00th=[  190], 95.00th=[  203],
     | 99.00th=[  239], 99.50th=[  243], 99.90th=[  255], 99.95th=[  255],
     | 99.99th=[  255]
   bw (  KiB/s): min=  232, max=  360, per=25.14%, avg=297.94, stdev=36.72, samples=35
   iops        : min=   58, max=   90, avg=74.40, stdev= 9.14, samples=35
  lat (msec)   : 2=0.08%, 4=0.04%, 20=0.35%, 50=1.60%, 100=51.02%
  lat (msec)   : 250=46.80%, 500=0.12%
  cpu          : usr=0.16%, sys=1.28%, ctx=3718, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2044: Mon May 13 02:41:42 2024
   read: IOPS=70, BW=283KiB/s (289kB/s)(4984KiB/17630msec)
    slat (usec): min=632, max=3017, avg=1227.00, stdev=283.24
    clat (msec): min=10, max=108, avg=72.56, stdev=12.21
     lat (msec): min=12, max=109, avg=73.78, stdev=12.23
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   55], 10.00th=[   60], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   70], 50.00th=[   73], 60.00th=[   75],
     | 70.00th=[   79], 80.00th=[   82], 90.00th=[   88], 95.00th=[   92],
     | 99.00th=[  102], 99.50th=[  103], 99.90th=[  107], 99.95th=[  109],
     | 99.99th=[  109]
   bw (  KiB/s): min=  183, max=  416, per=24.69%, avg=281.00, stdev=59.21, samples=35
   iops        : min=   45, max=  104, avg=70.14, stdev=14.86, samples=35
  write: IOPS=74, BW=298KiB/s (305kB/s)(5256KiB/17630msec)
    slat (usec): min=16, max=1241, avg=54.53, stdev=44.80
    clat (msec): min=32, max=259, avg=144.49, stdev=35.87
     lat (msec): min=32, max=259, avg=144.54, stdev=35.87
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   89], 10.00th=[  101], 20.00th=[  112],
     | 30.00th=[  126], 40.00th=[  136], 50.00th=[  144], 60.00th=[  155],
     | 70.00th=[  163], 80.00th=[  174], 90.00th=[  192], 95.00th=[  207],
     | 99.00th=[  239], 99.50th=[  243], 99.90th=[  259], 99.95th=[  259],
     | 99.99th=[  259]
   bw (  KiB/s): min=  216, max=  384, per=24.89%, avg=294.94, stdev=36.92, samples=35
   iops        : min=   54, max=   96, avg=73.63, stdev= 9.21, samples=35
  lat (msec)   : 20=0.35%, 50=0.90%, 100=52.11%, 250=46.56%, 500=0.08%
  cpu          : usr=0.16%, sys=1.31%, ctx=3584, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2045: Mon May 13 02:41:42 2024
   read: IOPS=71, BW=288KiB/s (294kB/s)(5060KiB/17599msec)
    slat (usec): min=689, max=3310, avg=1224.52, stdev=300.92
    clat (usec): min=1466, max=111110, avg=72918.47, stdev=11869.40
     lat (msec): min=2, max=112, avg=74.14, stdev=11.87
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   56], 10.00th=[   61], 20.00th=[   64],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   73], 60.00th=[   77],
     | 70.00th=[   79], 80.00th=[   83], 90.00th=[   88], 95.00th=[   92],
     | 99.00th=[  102], 99.50th=[  103], 99.90th=[  106], 99.95th=[  111],
     | 99.99th=[  111]
   bw (  KiB/s): min=  192, max=  400, per=25.24%, avg=287.23, stdev=45.00, samples=35
   iops        : min=   48, max=  100, avg=71.74, stdev=11.21, samples=35
  write: IOPS=73, BW=294KiB/s (301kB/s)(5180KiB/17599msec)
    slat (usec): min=15, max=1450, avg=54.14, stdev=47.33
    clat (msec): min=22, max=251, avg=144.75, stdev=34.38
     lat (msec): min=22, max=251, avg=144.80, stdev=34.38
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   91], 10.00th=[  101], 20.00th=[  115],
     | 30.00th=[  127], 40.00th=[  136], 50.00th=[  142], 60.00th=[  153],
     | 70.00th=[  163], 80.00th=[  171], 90.00th=[  192], 95.00th=[  207],
     | 99.00th=[  234], 99.50th=[  243], 99.90th=[  251], 99.95th=[  251],
     | 99.99th=[  251]
   bw (  KiB/s): min=  224, max=  368, per=24.66%, avg=292.26, stdev=40.72, samples=35
   iops        : min=   56, max=   92, avg=73.00, stdev=10.26, samples=35
  lat (msec)   : 2=0.04%, 20=0.23%, 50=0.98%, 100=52.34%, 250=46.33%
  lat (msec)   : 500=0.08%
  cpu          : usr=0.18%, sys=1.28%, ctx=3491, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1138KiB/s (1166kB/s), 283KiB/s-288KiB/s (289kB/s-294kB/s), io=19.6MiB (20.5MB), run=17576-17630msec
  WRITE: bw=1185KiB/s (1213kB/s), 294KiB/s-299KiB/s (301kB/s-306kB/s), io=20.4MiB (21.4MB), run=17576-17630msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=2063: Mon May 13 02:41:43 2024
   read: IOPS=14, BW=580MiB/s (608MB/s)(40.0MiB/69msec)
    slat (nsec): min=5404.4k, max=5404.4k, avg=5404368.00, stdev= 0.00
    clat (nsec): min=59594k, max=59594k, avg=59593500.00, stdev= 0.00
     lat (nsec): min=65007k, max=65007k, avg=65006782.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[59507],  5.00th=[59507], 10.00th=[59507], 20.00th=[59507],
     | 30.00th=[59507], 40.00th=[59507], 50.00th=[59507], 60.00th=[59507],
     | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507],
     | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.35%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s), io=40.0MiB (41.9MB), run=69-69msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout