== sanity test 398c: run fio to test AIO ================= 02:01:47 (1721887307)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.107426 s, 390 MB/s
osc.lustre-OST0000-osc-ffff8800b0032000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23355: Thu Jul 25 02:02:24 2024
  write: IOPS=72, BW=292KiB/s (299kB/s)(10.0MiB/35119msec)
    slat (usec): min=15, max=600, avg=85.97, stdev=59.87
    clat (msec): min=80, max=399, avg=219.07, stdev=51.17
     lat (msec): min=81, max=399, avg=219.16, stdev=51.18
    clat percentiles (msec):
     |  1.00th=[  111],  5.00th=[  144], 10.00th=[  159], 20.00th=[  176],
     | 30.00th=[  188], 40.00th=[  203], 50.00th=[  218], 60.00th=[  230],
     | 70.00th=[  245], 80.00th=[  262], 90.00th=[  288], 95.00th=[  313],
     | 99.00th=[  351], 99.50th=[  363], 99.90th=[  380], 99.95th=[  380],
     | 99.99th=[  401]
   bw (  KiB/s): min=  192, max=  512, per=24.96%, avg=290.79, stdev=59.97, samples=70
   iops        : min=   48, max=  128, avg=72.64, stdev=14.99, samples=70
  lat (msec)   : 100=0.74%, 250=72.97%, 500=26.29%
  cpu          : usr=0.13%, sys=0.71%, ctx=2285, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23356: Thu Jul 25 02:02:24 2024
  write: IOPS=73, BW=292KiB/s (299kB/s)(10.0MiB/35067msec)
    slat (usec): min=13, max=632, avg=84.69, stdev=57.64
    clat (msec): min=26, max=382, avg=218.89, stdev=52.73
     lat (msec): min=26, max=383, avg=218.98, stdev=52.74
    clat percentiles (msec):
     |  1.00th=[   94],  5.00th=[  142], 10.00th=[  159], 20.00th=[  176],
     | 30.00th=[  190], 40.00th=[  203], 50.00th=[  215], 60.00th=[  230],
     | 70.00th=[  243], 80.00th=[  262], 90.00th=[  292], 95.00th=[  313],
     | 99.00th=[  347], 99.50th=[  359], 99.90th=[  384], 99.95th=[  384],
     | 99.99th=[  384]
   bw (  KiB/s): min=  168, max=  488, per=24.96%, avg=290.81, stdev=61.26, samples=70
   iops        : min=   42, max=  122, avg=72.67, stdev=15.32, samples=70
  lat (msec)   : 50=0.27%, 100=0.78%, 250=73.05%, 500=25.90%
  cpu          : usr=0.14%, sys=0.68%, ctx=2229, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23357: Thu Jul 25 02:02:24 2024
  write: IOPS=72, BW=292KiB/s (299kB/s)(10.0MiB/35099msec)
    slat (usec): min=13, max=663, avg=86.42, stdev=59.42
    clat (msec): min=86, max=396, avg=218.95, stdev=50.77
     lat (msec): min=86, max=396, avg=219.04, stdev=50.78
    clat percentiles (msec):
     |  1.00th=[  111],  5.00th=[  146], 10.00th=[  159], 20.00th=[  176],
     | 30.00th=[  190], 40.00th=[  203], 50.00th=[  218], 60.00th=[  228],
     | 70.00th=[  245], 80.00th=[  262], 90.00th=[  288], 95.00th=[  305],
     | 99.00th=[  351], 99.50th=[  363], 99.90th=[  368], 99.95th=[  384],
     | 99.99th=[  397]
   bw (  KiB/s): min=  168, max=  520, per=24.96%, avg=290.73, stdev=61.12, samples=70
   iops        : min=   42, max=  130, avg=72.59, stdev=15.29, samples=70
  lat (msec)   : 100=0.78%, 250=72.34%, 500=26.88%
  cpu          : usr=0.14%, sys=0.71%, ctx=2247, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23358: Thu Jul 25 02:02:24 2024
  write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35129msec)
    slat (usec): min=14, max=762, avg=81.93, stdev=56.31
    clat (msec): min=81, max=399, avg=219.11, stdev=51.15
     lat (msec): min=82, max=399, avg=219.19, stdev=51.16
    clat percentiles (msec):
     |  1.00th=[  111],  5.00th=[  144], 10.00th=[  159], 20.00th=[  176],
     | 30.00th=[  190], 40.00th=[  203], 50.00th=[  215], 60.00th=[  228],
     | 70.00th=[  245], 80.00th=[  262], 90.00th=[  288], 95.00th=[  309],
     | 99.00th=[  351], 99.50th=[  368], 99.90th=[  380], 99.95th=[  397],
     | 99.99th=[  401]
   bw (  KiB/s): min=  176, max=  512, per=24.96%, avg=290.81, stdev=61.35, samples=70
   iops        : min=   44, max=  128, avg=72.67, stdev=15.35, samples=70
  lat (msec)   : 100=0.62%, 250=72.81%, 500=26.56%
  cpu          : usr=0.08%, sys=0.73%, ctx=2106, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1166KiB/s (1194kB/s), 291KiB/s-292KiB/s (298kB/s-299kB/s), io=40.0MiB (41.9MB), run=35067-35129msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23396: Thu Jul 25 02:02:44 2024
   read: IOPS=67, BW=272KiB/s (278kB/s)(5048KiB/18582msec)
    slat (usec): min=689, max=3026, avg=1218.17, stdev=316.63
    clat (msec): min=11, max=153, avg=76.04, stdev=13.73
     lat (msec): min=12, max=154, avg=77.26, stdev=13.77
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   58], 10.00th=[   63], 20.00th=[   66],
     | 30.00th=[   69], 40.00th=[   73], 50.00th=[   77], 60.00th=[   80],
     | 70.00th=[   83], 80.00th=[   86], 90.00th=[   93], 95.00th=[   96],
     | 99.00th=[  113], 99.50th=[  127], 99.90th=[  142], 99.95th=[  155],
     | 99.99th=[  155]
   bw (  KiB/s): min=  192, max=  392, per=25.13%, avg=271.11, stdev=54.09, samples=37
   iops        : min=   48, max=   98, avg=67.76, stdev=13.50, samples=37
  write: IOPS=69, BW=279KiB/s (286kB/s)(5192KiB/18582msec)
    slat (usec): min=13, max=1131, avg=57.47, stdev=41.71
    clat (msec): min=73, max=253, avg=153.36, stdev=33.73
     lat (msec): min=73, max=253, avg=153.42, stdev=33.73
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  101], 10.00th=[  113], 20.00th=[  125],
     | 30.00th=[  134], 40.00th=[  142], 50.00th=[  150], 60.00th=[  159],
     | 70.00th=[  169], 80.00th=[  182], 90.00th=[  201], 95.00th=[  218],
     | 99.00th=[  241], 99.50th=[  243], 99.90th=[  255], 99.95th=[  255],
     | 99.99th=[  255]
   bw (  KiB/s): min=  200, max=  344, per=24.83%, avg=279.11, stdev=33.69, samples=37
   iops        : min=   50, max=   86, avg=69.76, stdev= 8.44, samples=37
  lat (msec)   : 20=0.39%, 50=0.51%, 100=49.61%, 250=49.41%, 500=0.08%
  cpu          : usr=0.23%, sys=1.42%, ctx=2959, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23397: Thu Jul 25 02:02:44 2024
   read: IOPS=66, BW=268KiB/s (274kB/s)(4976KiB/18582msec)
    slat (usec): min=52, max=3796, avg=1191.22, stdev=308.33
    clat (msec): min=2, max=154, avg=76.55, stdev=15.15
     lat (msec): min=2, max=155, avg=77.74, stdev=15.20
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   55], 10.00th=[   62], 20.00th=[   67],
     | 30.00th=[   70], 40.00th=[   74], 50.00th=[   78], 60.00th=[   81],
     | 70.00th=[   84], 80.00th=[   87], 90.00th=[   94], 95.00th=[   97],
     | 99.00th=[  115], 99.50th=[  127], 99.90th=[  142], 99.95th=[  155],
     | 99.99th=[  155]
   bw (  KiB/s): min=  159, max=  376, per=24.76%, avg=267.19, stdev=53.56, samples=37
   iops        : min=   39, max=   94, avg=66.76, stdev=13.43, samples=37
  write: IOPS=70, BW=283KiB/s (290kB/s)(5264KiB/18582msec)
    slat (usec): min=14, max=874, avg=57.17, stdev=37.06
    clat (msec): min=37, max=263, avg=152.19, stdev=34.31
     lat (msec): min=37, max=263, avg=152.25, stdev=34.30
    clat percentiles (msec):
     |  1.00th=[   82],  5.00th=[  102], 10.00th=[  112], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  142], 50.00th=[  148], 60.00th=[  159],
     | 70.00th=[  169], 80.00th=[  180], 90.00th=[  199], 95.00th=[  215],
     | 99.00th=[  243], 99.50th=[  253], 99.90th=[  257], 99.95th=[  264],
     | 99.99th=[  264]
   bw (  KiB/s): min=  200, max=  360, per=25.18%, avg=282.97, stdev=38.06, samples=37
   iops        : min=   50, max=   90, avg=70.70, stdev= 9.52, samples=37
  lat (msec)   : 4=0.12%, 10=0.04%, 20=0.43%, 50=0.78%, 100=47.73%
  lat (msec)   : 250=50.62%, 500=0.27%
  cpu          : usr=0.15%, sys=1.51%, ctx=2845, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23398: Thu Jul 25 02:02:44 2024
   read: IOPS=67, BW=269KiB/s (275kB/s)(4984KiB/18545msec)
    slat (usec): min=629, max=4085, avg=1199.85, stdev=310.14
    clat (msec): min=12, max=139, avg=76.28, stdev=13.58
     lat (msec): min=13, max=142, avg=77.49, stdev=13.63
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   57], 10.00th=[   63], 20.00th=[   67],
     | 30.00th=[   69], 40.00th=[   73], 50.00th=[   78], 60.00th=[   80],
     | 70.00th=[   83], 80.00th=[   86], 90.00th=[   92], 95.00th=[   95],
     | 99.00th=[  112], 99.50th=[  131], 99.90th=[  138], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  160, max=  400, per=24.88%, avg=268.43, stdev=53.39, samples=37
   iops        : min=   40, max=  100, avg=67.03, stdev=13.36, samples=37
  write: IOPS=70, BW=283KiB/s (290kB/s)(5256KiB/18545msec)
    slat (usec): min=13, max=241, avg=57.27, stdev=28.26
    clat (msec): min=73, max=266, avg=151.89, stdev=32.64
     lat (msec): min=73, max=266, avg=151.95, stdev=32.64
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  104], 10.00th=[  112], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  148], 60.00th=[  159],
     | 70.00th=[  167], 80.00th=[  178], 90.00th=[  194], 95.00th=[  215],
     | 99.00th=[  245], 99.50th=[  253], 99.90th=[  259], 99.95th=[  266],
     | 99.99th=[  266]
   bw (  KiB/s): min=  223, max=  368, per=25.06%, avg=281.62, stdev=32.28, samples=37
   iops        : min=   55, max=   92, avg=70.32, stdev= 8.11, samples=37
  lat (msec)   : 20=0.27%, 50=0.59%, 100=47.73%, 250=51.09%, 500=0.31%
  cpu          : usr=0.26%, sys=1.39%, ctx=2941, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23399: Thu Jul 25 02:02:44 2024
   read: IOPS=68, BW=273KiB/s (279kB/s)(5060KiB/18545msec)
    slat (usec): min=651, max=2948, avg=1192.10, stdev=290.80
    clat (msec): min=2, max=144, avg=76.86, stdev=13.50
     lat (msec): min=4, max=145, avg=78.05, stdev=13.51
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   57], 10.00th=[   63], 20.00th=[   67],
     | 30.00th=[   70], 40.00th=[   74], 50.00th=[   78], 60.00th=[   81],
     | 70.00th=[   83], 80.00th=[   87], 90.00th=[   93], 95.00th=[   96],
     | 99.00th=[  117], 99.50th=[  131], 99.90th=[  140], 99.95th=[  144],
     | 99.99th=[  144]
   bw (  KiB/s): min=  160, max=  392, per=25.18%, avg=271.68, stdev=52.32, samples=37
   iops        : min=   40, max=   98, avg=67.84, stdev=13.09, samples=37
  write: IOPS=69, BW=279KiB/s (286kB/s)(5180KiB/18545msec)
    slat (usec): min=13, max=1303, avg=58.26, stdev=46.89
    clat (msec): min=78, max=265, avg=152.29, stdev=33.36
     lat (msec): min=78, max=265, avg=152.35, stdev=33.36
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  103], 10.00th=[  111], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  142], 50.00th=[  150], 60.00th=[  159],
     | 70.00th=[  169], 80.00th=[  180], 90.00th=[  199], 95.00th=[  211],
     | 99.00th=[  241], 99.50th=[  247], 99.90th=[  259], 99.95th=[  266],
     | 99.99th=[  266]
   bw (  KiB/s): min=  216, max=  376, per=24.77%, avg=278.38, stdev=38.43, samples=37
   iops        : min=   54, max=   94, avg=69.51, stdev= 9.63, samples=37
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.12%, 50=0.51%, 100=49.22%
  lat (msec)   : 250=49.88%, 500=0.20%
  cpu          : usr=0.15%, sys=1.51%, ctx=3009, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1080KiB/s (1106kB/s), 268KiB/s-273KiB/s (274kB/s-279kB/s), io=19.6MiB (20.5MB), run=18545-18582msec
  WRITE: bw=1124KiB/s (1151kB/s), 279KiB/s-283KiB/s (286kB/s-290kB/s), io=20.4MiB (21.4MB), run=18545-18582msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23418: Thu Jul 25 02:02:45 2024
   read: IOPS=6, BW=263MiB/s (276MB/s)(40.0MiB/152msec)
    slat (nsec): min=5659.3k, max=5659.3k, avg=5659251.00, stdev= 0.00
    clat (nsec): min=139743k, max=139743k, avg=139743242.00, stdev= 0.00
     lat (nsec): min=145412k, max=145412k, avg=145411996.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  140],  5.00th=[  140], 10.00th=[  140], 20.00th=[  140],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.64%, ctx=17, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=263MiB/s (276MB/s), 263MiB/s-263MiB/s (276MB/s-276MB/s), io=40.0MiB (41.9MB), run=152-152msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout