== sanity test 398c: run fio to test AIO ================= 06:26:09 (1747218369)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0911261 s, 460 MB/s
osc.lustre-OST0000-osc-ffff8800a8361800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=11638: Wed May 14 06:26:36 2025
  write: IOPS=99, BW=399KiB/s (408kB/s)(10.0MiB/25693msec)
    slat (usec): min=13, max=637, avg=76.10, stdev=44.85
    clat (msec): min=61, max=380, avg=160.32, stdev=31.41
     lat (msec): min=61, max=380, avg=160.40, stdev=31.41
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  111], 10.00th=[  124], 20.00th=[  136],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  161], 60.00th=[  169],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  197], 95.00th=[  207],
     | 99.00th=[  234], 99.50th=[  279], 99.90th=[  368], 99.95th=[  368],
     | 99.99th=[  380]
   bw (  KiB/s): min=  319, max=  640, per=25.04%, avg=399.10, stdev=56.02, samples=51
   iops        : min=   79, max=  160, avg=99.69, stdev=14.03, samples=51
  lat (msec)   : 100=1.45%, 250=97.93%, 500=0.62%
  cpu          : usr=0.19%, sys=0.80%, ctx=2354, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11639: Wed May 14 06:26:36 2025
  write: IOPS=99, BW=399KiB/s (408kB/s)(10.0MiB/25693msec)
    slat (usec): min=13, max=484, avg=71.70, stdev=45.30
    clat (msec): min=65, max=304, avg=160.29, stdev=29.53
     lat (msec): min=65, max=304, avg=160.36, stdev=29.53
    clat percentiles (msec):
     |  1.00th=[   94],  5.00th=[  112], 10.00th=[  124], 20.00th=[  136],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  161], 60.00th=[  169],
     | 70.00th=[  176], 80.00th=[  184], 90.00th=[  197], 95.00th=[  209],
     | 99.00th=[  232], 99.50th=[  234], 99.90th=[  255], 99.95th=[  279],
     | 99.99th=[  305]
   bw (  KiB/s): min=  312, max=  624, per=24.97%, avg=398.00, stdev=53.41, samples=51
   iops        : min=   78, max=  156, avg=99.41, stdev=13.41, samples=51
  lat (msec)   : 100=1.48%, 250=98.32%, 500=0.20%
  cpu          : usr=0.14%, sys=0.80%, ctx=2084, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11640: Wed May 14 06:26:36 2025
  write: IOPS=99, BW=399KiB/s (408kB/s)(10.0MiB/25682msec)
    slat (usec): min=13, max=416, avg=74.08, stdev=44.63
    clat (msec): min=14, max=369, avg=160.27, stdev=32.07
     lat (msec): min=14, max=369, avg=160.35, stdev=32.07
    clat percentiles (msec):
     |  1.00th=[   70],  5.00th=[  111], 10.00th=[  123], 20.00th=[  136],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  161], 60.00th=[  169],
     | 70.00th=[  178], 80.00th=[  186], 90.00th=[  199], 95.00th=[  207],
     | 99.00th=[  234], 99.50th=[  257], 99.90th=[  321], 99.95th=[  334],
     | 99.99th=[  372]
   bw (  KiB/s): min=  288, max=  640, per=25.04%, avg=399.10, stdev=58.43, samples=51
   iops        : min=   72, max=  160, avg=99.69, stdev=14.65, samples=51
  lat (msec)   : 20=0.04%, 50=0.35%, 100=1.37%, 250=97.70%, 500=0.55%
  cpu          : usr=0.17%, sys=0.79%, ctx=2316, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=11641: Wed May 14 06:26:36 2025
  write: IOPS=99, BW=399KiB/s (408kB/s)(10.0MiB/25682msec)
    slat (usec): min=14, max=473, avg=75.44, stdev=45.72
    clat (msec): min=57, max=381, avg=160.23, stdev=31.07
     lat (msec): min=57, max=381, avg=160.31, stdev=31.08
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  112], 10.00th=[  123], 20.00th=[  136],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  161], 60.00th=[  169],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  199], 95.00th=[  209],
     | 99.00th=[  232], 99.50th=[  243], 99.90th=[  359], 99.95th=[  372],
     | 99.99th=[  380]
   bw (  KiB/s): min=  303, max=  640, per=25.04%, avg=399.10, stdev=55.54, samples=51
   iops        : min=   75, max=  160, avg=99.69, stdev=13.92, samples=51
  lat (msec)   : 100=1.56%, 250=98.01%, 500=0.43%
  cpu          : usr=0.18%, sys=0.79%, ctx=2346, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1594KiB/s (1632kB/s), 399KiB/s-399KiB/s (408kB/s-408kB/s), io=40.0MiB (41.9MB), run=25682-25693msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=11671: Wed May 14 06:26:53 2025
   read: IOPS=79, BW=318KiB/s (326kB/s)(5048KiB/15857msec)
    slat (usec): min=735, max=4634, avg=1242.20, stdev=288.12
    clat (msec): min=3, max=109, avg=67.11, stdev=12.56
     lat (msec): min=4, max=111, avg=68.36, stdev=12.57
    clat percentiles (msec):
     |  1.00th=[   29],  5.00th=[   48], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   68], 60.00th=[   71],
     | 70.00th=[   73], 80.00th=[   78], 90.00th=[   83], 95.00th=[   86],
     | 99.00th=[   94], 99.50th=[   99], 99.90th=[  109], 99.95th=[  110],
     | 99.99th=[  110]
   bw (  KiB/s): min=  192, max=  408, per=25.15%, avg=317.90, stdev=48.58, samples=31
   iops        : min=   48, max=  102, avg=79.45, stdev=12.12, samples=31
  write: IOPS=81, BW=327KiB/s (335kB/s)(5192KiB/15857msec)
    slat (usec): min=13, max=310, avg=66.76, stdev=35.48
    clat (msec): min=52, max=216, avg=128.60, stdev=29.71
     lat (msec): min=52, max=216, avg=128.67, stdev=29.71
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   84], 10.00th=[   92], 20.00th=[  104],
     | 30.00th=[  112], 40.00th=[  120], 50.00th=[  126], 60.00th=[  133],
     | 70.00th=[  144], 80.00th=[  155], 90.00th=[  171], 95.00th=[  184],
     | 99.00th=[  199], 99.50th=[  203], 99.90th=[  213], 99.95th=[  218],
     | 99.99th=[  218]
   bw (  KiB/s): min=  264, max=  392, per=24.90%, avg=327.71, stdev=32.48, samples=31
   iops        : min=   66, max=   98, avg=81.90, stdev= 8.11, samples=31
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.39%, 50=2.54%, 100=54.88%
  lat (msec)   : 250=42.11%
  cpu          : usr=0.18%, sys=1.75%, ctx=3754, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11672: Wed May 14 06:26:53 2025
   read: IOPS=78, BW=314KiB/s (321kB/s)(4976KiB/15857msec)
    slat (usec): min=727, max=3137, avg=1251.36, stdev=282.74
    clat (msec): min=9, max=122, avg=67.21, stdev=12.49
     lat (msec): min=11, max=123, avg=68.46, stdev=12.50
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   50], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   61], 40.00th=[   64], 50.00th=[   68], 60.00th=[   71],
     | 70.00th=[   73], 80.00th=[   78], 90.00th=[   83], 95.00th=[   87],
     | 99.00th=[   97], 99.50th=[  101], 99.90th=[  122], 99.95th=[  123],
     | 99.99th=[  123]
   bw (  KiB/s): min=  224, max=  432, per=24.86%, avg=314.26, stdev=55.63, samples=31
   iops        : min=   56, max=  108, avg=78.52, stdev=13.88, samples=31
  write: IOPS=82, BW=332KiB/s (340kB/s)(5264KiB/15857msec)
    slat (usec): min=15, max=268, avg=65.95, stdev=35.32
    clat (msec): min=55, max=215, avg=127.68, stdev=29.26
     lat (msec): min=55, max=215, avg=127.75, stdev=29.26
    clat percentiles (msec):
     |  1.00th=[   61],  5.00th=[   84], 10.00th=[   92], 20.00th=[  103],
     | 30.00th=[  111], 40.00th=[  118], 50.00th=[  127], 60.00th=[  134],
     | 70.00th=[  144], 80.00th=[  153], 90.00th=[  169], 95.00th=[  182],
     | 99.00th=[  197], 99.50th=[  199], 99.90th=[  201], 99.95th=[  215],
     | 99.99th=[  215]
   bw (  KiB/s): min=  240, max=  408, per=25.23%, avg=332.06, stdev=36.09, samples=31
   iops        : min=   60, max=  102, avg=82.97, stdev= 9.04, samples=31
  lat (msec)   : 10=0.08%, 20=0.20%, 50=2.30%, 100=54.88%, 250=42.54%
  cpu          : usr=0.17%, sys=1.75%, ctx=3601, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11673: Wed May 14 06:26:53 2025
   read: IOPS=78, BW=314KiB/s (322kB/s)(4984KiB/15874msec)
    slat (usec): min=18, max=3573, avg=1258.06, stdev=312.20
    clat (usec): min=1721, max=126565, avg=67298.07, stdev=12463.93
     lat (msec): min=2, max=128, avg=68.56, stdev=12.50
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   48], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   61], 40.00th=[   66], 50.00th=[   68], 60.00th=[   71],
     | 70.00th=[   74], 80.00th=[   78], 90.00th=[   83], 95.00th=[   86],
     | 99.00th=[   94], 99.50th=[   99], 99.90th=[  117], 99.95th=[  127],
     | 99.99th=[  127]
   bw (  KiB/s): min=  224, max=  400, per=24.92%, avg=315.03, stdev=46.76, samples=31
   iops        : min=   56, max=  100, avg=78.71, stdev=11.65, samples=31
  write: IOPS=82, BW=331KiB/s (339kB/s)(5256KiB/15874msec)
    slat (usec): min=13, max=304, avg=68.04, stdev=36.83
    clat (msec): min=19, max=212, avg=128.06, stdev=30.71
     lat (msec): min=20, max=212, avg=128.13, stdev=30.71
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   82], 10.00th=[   91], 20.00th=[  103],
     | 30.00th=[  112], 40.00th=[  120], 50.00th=[  128], 60.00th=[  134],
     | 70.00th=[  144], 80.00th=[  155], 90.00th=[  169], 95.00th=[  182],
     | 99.00th=[  199], 99.50th=[  203], 99.90th=[  211], 99.95th=[  213],
     | 99.99th=[  213]
   bw (  KiB/s): min=  272, max=  400, per=25.02%, avg=329.23, stdev=36.58, samples=31
   iops        : min=   68, max=  100, avg=82.26, stdev= 9.13, samples=31
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.20%, 50=3.12%
  lat (msec)   : 100=53.95%, 250=42.58%
  cpu          : usr=0.22%, sys=1.73%, ctx=3607, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=11674: Wed May 14 06:26:53 2025
   read: IOPS=80, BW=321KiB/s (328kB/s)(5060KiB/15780msec)
    slat (usec): min=694, max=3637, avg=1258.38, stdev=279.06
    clat (msec): min=4, max=186, avg=67.49, stdev=13.05
     lat (msec): min=5, max=187, avg=68.75, stdev=13.06
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   49], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   69], 60.00th=[   71],
     | 70.00th=[   74], 80.00th=[   78], 90.00th=[   83], 95.00th=[   87],
     | 99.00th=[   93], 99.50th=[   99], 99.90th=[  184], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  200, max=  416, per=25.54%, avg=322.81, stdev=53.72, samples=31
   iops        : min=   50, max=  104, avg=80.68, stdev=13.39, samples=31
  write: IOPS=82, BW=328KiB/s (336kB/s)(5180KiB/15780msec)
    slat (usec): min=15, max=1454, avg=65.53, stdev=51.23
    clat (msec): min=45, max=271, avg=127.32, stdev=30.65
     lat (msec): min=46, max=271, avg=127.39, stdev=30.65
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   83], 10.00th=[   91], 20.00th=[  102],
     | 30.00th=[  112], 40.00th=[  118], 50.00th=[  126], 60.00th=[  132],
     | 70.00th=[  142], 80.00th=[  153], 90.00th=[  167], 95.00th=[  178],
     | 99.00th=[  199], 99.50th=[  257], 99.90th=[  271], 99.95th=[  271],
     | 99.99th=[  271]
   bw (  KiB/s): min=  256, max=  416, per=24.96%, avg=328.48, stdev=43.87, samples=31
   iops        : min=   64, max=  104, avg=82.10, stdev=10.97, samples=31
  lat (msec)   : 10=0.08%, 20=0.23%, 50=2.54%, 100=56.17%, 250=40.66%
  lat (msec)   : 500=0.31%
  cpu          : usr=0.29%, sys=1.66%, ctx=3453, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1264KiB/s (1295kB/s), 314KiB/s-321KiB/s (321kB/s-328kB/s), io=19.6MiB (20.5MB), run=15780-15874msec
  WRITE: bw=1316KiB/s (1348kB/s), 327KiB/s-332KiB/s (335kB/s-340kB/s), io=20.4MiB (21.4MB), run=15780-15874msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=11691: Wed May 14 06:26:54 2025
   read: IOPS=9, BW=370MiB/s (388MB/s)(40.0MiB/108msec)
    slat (nsec): min=7120.1k, max=7120.1k, avg=7120121.00, stdev= 0.00
    clat (nsec): min=94987k, max=94987k, avg=94987007.00, stdev= 0.00
     lat (nsec): min=102117k, max=102117k, avg=102117115.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[94897],  5.00th=[94897], 10.00th=[94897], 20.00th=[94897],
     | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897],
     | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897],
     | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.54%, ctx=17, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=370MiB/s (388MB/s), 370MiB/s-370MiB/s (388MB/s-388MB/s), io=40.0MiB (41.9MB), run=108-108msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout