== sanity test 398c: run fio to test AIO ================= 06:49:34 (1721386174)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.078277 s, 536 MB/s
osc.lustre-OST0000-osc-ffff8800b0717800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=24233: Fri Jul 19 06:50:05 2024
  write: IOPS=86, BW=348KiB/s (356kB/s)(10.0MiB/29445msec)
    slat (usec): min=12, max=419, avg=74.69, stdev=44.64
    clat (msec): min=16, max=326, avg=183.86, stdev=36.54
     lat (msec): min=16, max=326, avg=183.94, stdev=36.55
    clat percentiles (msec):
     |  1.00th=[   99],  5.00th=[  130], 10.00th=[  140], 20.00th=[  155],
     | 30.00th=[  165], 40.00th=[  174], 50.00th=[  184], 60.00th=[  192],
     | 70.00th=[  203], 80.00th=[  215], 90.00th=[  230], 95.00th=[  245],
     | 99.00th=[  268], 99.50th=[  279], 99.90th=[  313], 99.95th=[  326],
     | 99.99th=[  326]
   bw (  KiB/s): min=  240, max=  544, per=24.97%, avg=346.83, stdev=52.45, samples=58
   iops        : min=   60, max=  136, avg=86.66, stdev=13.12, samples=58
  lat (msec)   : 20=0.04%, 100=1.41%, 250=95.35%, 500=3.20%
  cpu          : usr=0.20%, sys=0.73%, ctx=2333, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24234: Fri Jul 19 06:50:05 2024
  write: IOPS=86, BW=347KiB/s (356kB/s)(10.0MiB/29481msec)
    slat (usec): min=12, max=533, avg=73.54, stdev=45.29
    clat (msec): min=50, max=288, avg=184.08, stdev=36.44
     lat (msec): min=50, max=289, avg=184.15, stdev=36.45
    clat percentiles (msec):
     |  1.00th=[   95],  5.00th=[  129], 10.00th=[  140], 20.00th=[  153],
     | 30.00th=[  163], 40.00th=[  174], 50.00th=[  184], 60.00th=[  194],
     | 70.00th=[  205], 80.00th=[  218], 90.00th=[  232], 95.00th=[  243],
     | 99.00th=[  264], 99.50th=[  279], 99.90th=[  288], 99.95th=[  288],
     | 99.99th=[  288]
   bw (  KiB/s): min=  240, max=  544, per=24.89%, avg=345.72, stdev=53.27, samples=58
   iops        : min=   60, max=  136, avg=86.38, stdev=13.33, samples=58
  lat (msec)   : 100=1.45%, 250=95.82%, 500=2.73%
  cpu          : usr=0.16%, sys=0.77%, ctx=2218, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24235: Fri Jul 19 06:50:05 2024
  write: IOPS=87, BW=348KiB/s (357kB/s)(10.0MiB/29398msec)
    slat (usec): min=11, max=389, avg=73.08, stdev=43.85
    clat (msec): min=52, max=330, avg=183.57, stdev=37.99
     lat (msec): min=52, max=330, avg=183.65, stdev=38.00
    clat percentiles (msec):
     |  1.00th=[   82],  5.00th=[  124], 10.00th=[  140], 20.00th=[  155],
     | 30.00th=[  165], 40.00th=[  174], 50.00th=[  184], 60.00th=[  194],
     | 70.00th=[  205], 80.00th=[  218], 90.00th=[  232], 95.00th=[  243],
     | 99.00th=[  271], 99.50th=[  288], 99.90th=[  330], 99.95th=[  330],
     | 99.99th=[  330]
   bw (  KiB/s): min=  248, max=  648, per=25.02%, avg=347.52, stdev=61.08, samples=58
   iops        : min=   62, max=  162, avg=86.83, stdev=15.26, samples=58
  lat (msec)   : 100=2.50%, 250=95.20%, 500=2.30%
  cpu          : usr=0.16%, sys=0.74%, ctx=2409, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24236: Fri Jul 19 06:50:05 2024
  write: IOPS=86, BW=347KiB/s (356kB/s)(10.0MiB/29480msec)
    slat (usec): min=13, max=418, avg=70.39, stdev=43.04
    clat (msec): min=67, max=289, avg=184.09, stdev=35.90
     lat (msec): min=67, max=289, avg=184.16, stdev=35.91
    clat percentiles (msec):
     |  1.00th=[   99],  5.00th=[  128], 10.00th=[  142], 20.00th=[  155],
     | 30.00th=[  165], 40.00th=[  174], 50.00th=[  184], 60.00th=[  192],
     | 70.00th=[  203], 80.00th=[  215], 90.00th=[  232], 95.00th=[  243],
     | 99.00th=[  268], 99.50th=[  279], 99.90th=[  288], 99.95th=[  292],
     | 99.99th=[  292]
   bw (  KiB/s): min=  232, max=  528, per=24.90%, avg=345.86, stdev=54.02, samples=58
   iops        : min=   58, max=  132, avg=86.41, stdev=13.52, samples=58
  lat (msec)   : 100=1.17%, 250=96.13%, 500=2.70%
  cpu          : usr=0.18%, sys=0.72%, ctx=2058, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1389KiB/s (1423kB/s), 347KiB/s-348KiB/s (356kB/s-357kB/s), io=40.0MiB (41.9MB), run=29398-29481msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=24270: Fri Jul 19 06:50:24 2024
   read: IOPS=70, BW=283KiB/s (290kB/s)(5048KiB/17847msec)
    slat (usec): min=485, max=3404, avg=1005.75, stdev=319.35
    clat (msec): min=3, max=146, avg=77.12, stdev=14.74
     lat (msec): min=4, max=147, avg=78.13, stdev=14.80
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   57], 10.00th=[   62], 20.00th=[   66],
     | 30.00th=[   70], 40.00th=[   74], 50.00th=[   77], 60.00th=[   80],
     | 70.00th=[   84], 80.00th=[   89], 90.00th=[   96], 95.00th=[  102],
     | 99.00th=[  121], 99.50th=[  124], 99.90th=[  146], 99.95th=[  146],
     | 99.99th=[  146]
   bw (  KiB/s): min=  200, max=  392, per=25.14%, avg=281.57, stdev=47.23, samples=35
   iops        : min=   50, max=   98, avg=70.37, stdev=11.83, samples=35
  write: IOPS=72, BW=291KiB/s (298kB/s)(5192KiB/17847msec)
    slat (usec): min=13, max=233, avg=65.74, stdev=34.32
    clat (msec): min=70, max=223, avg=143.64, stdev=30.98
     lat (msec): min=70, max=223, avg=143.71, stdev=30.98
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[   95], 10.00th=[  104], 20.00th=[  116],
     | 30.00th=[  125], 40.00th=[  136], 50.00th=[  142], 60.00th=[  150],
     | 70.00th=[  161], 80.00th=[  174], 90.00th=[  188], 95.00th=[  199],
     | 99.00th=[  209], 99.50th=[  213], 99.90th=[  220], 99.95th=[  224],
     | 99.99th=[  224]
   bw (  KiB/s): min=  224, max=  416, per=24.81%, avg=289.34, stdev=40.68, samples=35
   iops        : min=   56, max=  104, avg=72.31, stdev=10.14, samples=35
  lat (msec)   : 4=0.08%, 20=0.16%, 50=0.78%, 100=49.38%, 250=49.61%
  cpu          : usr=0.31%, sys=1.75%, ctx=2803, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24271: Fri Jul 19 06:50:24 2024
   read: IOPS=69, BW=278KiB/s (284kB/s)(4976KiB/17917msec)
    slat (usec): min=461, max=3228, avg=994.55, stdev=300.65
    clat (msec): min=11, max=147, avg=76.93, stdev=15.04
     lat (msec): min=11, max=148, avg=77.93, stdev=15.11
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   57], 10.00th=[   61], 20.00th=[   66],
     | 30.00th=[   69], 40.00th=[   73], 50.00th=[   77], 60.00th=[   80],
     | 70.00th=[   84], 80.00th=[   88], 90.00th=[   96], 95.00th=[  102],
     | 99.00th=[  118], 99.50th=[  132], 99.90th=[  148], 99.95th=[  148],
     | 99.99th=[  148]
   bw (  KiB/s): min=  136, max=  416, per=24.69%, avg=276.54, stdev=56.43, samples=35
   iops        : min=   34, max=  104, avg=69.11, stdev=14.12, samples=35
  write: IOPS=73, BW=294KiB/s (301kB/s)(5264KiB/17917msec)
    slat (usec): min=14, max=302, avg=65.97, stdev=35.67
    clat (msec): min=23, max=227, avg=143.90, stdev=32.28
     lat (msec): min=23, max=227, avg=143.97, stdev=32.28
    clat percentiles (msec):
     |  1.00th=[   82],  5.00th=[   94], 10.00th=[  103], 20.00th=[  114],
     | 30.00th=[  125], 40.00th=[  133], 50.00th=[  142], 60.00th=[  153],
     | 70.00th=[  163], 80.00th=[  174], 90.00th=[  190], 95.00th=[  199],
     | 99.00th=[  211], 99.50th=[  215], 99.90th=[  224], 99.95th=[  228],
     | 99.99th=[  228]
   bw (  KiB/s): min=  224, max=  384, per=25.01%, avg=291.63, stdev=41.91, samples=35
   iops        : min=   56, max=   96, avg=72.89, stdev=10.45, samples=35
  lat (msec)   : 20=0.20%, 50=1.05%, 100=49.18%, 250=49.57%
  cpu          : usr=0.27%, sys=1.72%, ctx=2807, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24272: Fri Jul 19 06:50:24 2024
   read: IOPS=69, BW=278KiB/s (285kB/s)(4984KiB/17917msec)
    slat (usec): min=449, max=2732, avg=995.57, stdev=303.69
    clat (msec): min=12, max=149, avg=76.42, stdev=15.14
     lat (msec): min=13, max=150, avg=77.41, stdev=15.21
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   55], 10.00th=[   59], 20.00th=[   64],
     | 30.00th=[   68], 40.00th=[   73], 50.00th=[   77], 60.00th=[   80],
     | 70.00th=[   84], 80.00th=[   88], 90.00th=[   96], 95.00th=[  102],
     | 99.00th=[  118], 99.50th=[  126], 99.90th=[  148], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  176, max=  456, per=24.90%, avg=278.83, stdev=63.85, samples=35
   iops        : min=   44, max=  114, avg=69.69, stdev=15.96, samples=35
  write: IOPS=73, BW=293KiB/s (300kB/s)(5256KiB/17917msec)
    slat (usec): min=13, max=1174, avg=68.09, stdev=49.58
    clat (msec): min=62, max=220, avg=144.42, stdev=31.22
     lat (msec): min=62, max=220, avg=144.49, stdev=31.23
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[   95], 10.00th=[  104], 20.00th=[  116],
     | 30.00th=[  127], 40.00th=[  136], 50.00th=[  144], 60.00th=[  150],
     | 70.00th=[  163], 80.00th=[  171], 90.00th=[  188], 95.00th=[  199],
     | 99.00th=[  211], 99.50th=[  211], 99.90th=[  220], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  192, max=  392, per=24.85%, avg=289.80, stdev=34.21, samples=35
   iops        : min=   48, max=   98, avg=72.43, stdev= 8.54, samples=35
  lat (msec)   : 20=0.16%, 50=1.17%, 100=48.28%, 250=50.39%
  cpu          : usr=0.34%, sys=1.69%, ctx=2796, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24273: Fri Jul 19 06:50:24 2024
   read: IOPS=71, BW=285KiB/s (292kB/s)(5060KiB/17767msec)
    slat (usec): min=22, max=2470, avg=989.62, stdev=305.94
    clat (msec): min=2, max=182, avg=76.43, stdev=15.94
     lat (msec): min=2, max=183, avg=77.42, stdev=16.02
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   56], 10.00th=[   61], 20.00th=[   65],
     | 30.00th=[   68], 40.00th=[   73], 50.00th=[   77], 60.00th=[   80],
     | 70.00th=[   84], 80.00th=[   88], 90.00th=[   94], 95.00th=[  102],
     | 99.00th=[  123], 99.50th=[  131], 99.90th=[  171], 99.95th=[  184],
     | 99.99th=[  184]
   bw (  KiB/s): min=  216, max=  392, per=25.53%, avg=285.91, stdev=45.80, samples=35
   iops        : min=   54, max=   98, avg=71.46, stdev=11.42, samples=35
  write: IOPS=72, BW=292KiB/s (299kB/s)(5180KiB/17767msec)
    slat (usec): min=13, max=296, avg=65.83, stdev=36.08
    clat (msec): min=25, max=232, avg=143.66, stdev=32.26
     lat (msec): min=25, max=232, avg=143.73, stdev=32.26
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   95], 10.00th=[  102], 20.00th=[  115],
     | 30.00th=[  125], 40.00th=[  136], 50.00th=[  144], 60.00th=[  153],
     | 70.00th=[  163], 80.00th=[  171], 90.00th=[  188], 95.00th=[  199],
     | 99.00th=[  213], 99.50th=[  213], 99.90th=[  226], 99.95th=[  232],
     | 99.99th=[  232]
   bw (  KiB/s): min=  224, max=  440, per=24.91%, avg=290.49, stdev=40.19, samples=35
   iops        : min=   56, max=  110, avg=72.60, stdev=10.05, samples=35
  lat (msec)   : 4=0.16%, 10=0.04%, 20=0.20%, 50=1.02%, 100=49.96%
  lat (msec)   : 250=48.63%
  cpu          : usr=0.25%, sys=1.78%, ctx=2822, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1120KiB/s (1147kB/s), 278KiB/s-285KiB/s (284kB/s-292kB/s), io=19.6MiB (20.5MB), run=17767-17917msec
  WRITE: bw=1166KiB/s (1194kB/s), 291KiB/s-294KiB/s (298kB/s-301kB/s), io=20.4MiB (21.4MB), run=17767-17917msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=24291: Fri Jul 19 06:50:25 2024
   read: IOPS=8, BW=339MiB/s (355MB/s)(40.0MiB/118msec)
    slat (nsec): min=7246.4k, max=7246.4k, avg=7246408.00, stdev= 0.00
    clat (nsec): min=106163k, max=106163k, avg=106162695.00, stdev= 0.00
     lat (nsec): min=113418k, max=113418k, avg=113418143.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  107], 10.00th=[  107], 20.00th=[  107],
     | 30.00th=[  107], 40.00th=[  107], 50.00th=[  107], 60.00th=[  107],
     | 70.00th=[  107], 80.00th=[  107], 90.00th=[  107], 95.00th=[  107],
     | 99.00th=[  107], 99.50th=[  107], 99.90th=[  107], 99.95th=[  107],
     | 99.99th=[  107]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.13%, ctx=15, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=40.0MiB (41.9MB), run=118-118msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck