== sanity test 398c: run fio to test AIO ================= 13:36:23 (1736793383)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.130757 s, 321 MB/s
osc.lustre-OST0000-osc-ffff88009dc8b800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=3524: Mon Jan 13 13:36:50 2025
  write: IOPS=101, BW=405KiB/s (414kB/s)(10.0MiB/25303msec)
    slat (usec): min=13, max=498, avg=61.81, stdev=36.78
    clat (msec): min=53, max=255, avg=157.90, stdev=28.32
     lat (msec): min=53, max=255, avg=157.97, stdev=28.32
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  107], 10.00th=[  121], 20.00th=[  134],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  159], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  184], 90.00th=[  194], 95.00th=[  201],
     | 99.00th=[  215], 99.50th=[  222], 99.90th=[  239], 99.95th=[  239],
     | 99.99th=[  255]
   bw (  KiB/s): min=  296, max=  592, per=24.99%, avg=404.26, stdev=54.86, samples=50
   iops        : min=   74, max=  148, avg=101.02, stdev=13.72, samples=50
  lat (msec)   : 100=3.24%, 250=96.72%, 500=0.04%
  cpu          : usr=0.15%, sys=0.71%, ctx=2430, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=3525: Mon Jan 13 13:36:50 2025
  write: IOPS=101, BW=405KiB/s (415kB/s)(10.0MiB/25280msec)
    slat (usec): min=12, max=282, avg=60.52, stdev=35.19
    clat (msec): min=28, max=273, avg=157.86, stdev=28.89
     lat (msec): min=28, max=273, avg=157.92, stdev=28.89
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  108], 10.00th=[  122], 20.00th=[  134],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  184], 90.00th=[  192], 95.00th=[  201],
     | 99.00th=[  213], 99.50th=[  220], 99.90th=[  264], 99.95th=[  264],
     | 99.99th=[  275]
   bw (  KiB/s): min=  328, max=  648, per=25.00%, avg=404.56, stdev=55.39, samples=50
   iops        : min=   82, max=  162, avg=101.08, stdev=13.84, samples=50
  lat (msec)   : 50=0.31%, 100=3.24%, 250=96.33%, 500=0.12%
  cpu          : usr=0.10%, sys=0.74%, ctx=2248, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=3526: Mon Jan 13 13:36:50 2025
  write: IOPS=101, BW=405KiB/s (414kB/s)(10.0MiB/25303msec)
    slat (usec): min=12, max=387, avg=60.70, stdev=37.91
    clat (msec): min=59, max=245, avg=157.94, stdev=27.84
     lat (msec): min=59, max=245, avg=158.00, stdev=27.84
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  110], 10.00th=[  121], 20.00th=[  136],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  182], 90.00th=[  192], 95.00th=[  201],
     | 99.00th=[  215], 99.50th=[  222], 99.90th=[  236], 99.95th=[  236],
     | 99.99th=[  245]
   bw (  KiB/s): min=  312, max=  608, per=24.91%, avg=403.12, stdev=53.34, samples=50
   iops        : min=   78, max=  152, avg=100.76, stdev=13.45, samples=50
  lat (msec)   : 100=3.09%, 250=96.91%
  cpu          : usr=0.13%, sys=0.69%, ctx=2284, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=3527: Mon Jan 13 13:36:50 2025
  write: IOPS=101, BW=405KiB/s (414kB/s)(10.0MiB/25310msec)
    slat (usec): min=12, max=464, avg=63.23, stdev=39.29
    clat (msec): min=58, max=243, avg=157.92, stdev=27.93
     lat (msec): min=59, max=243, avg=157.99, stdev=27.93
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  110], 10.00th=[  122], 20.00th=[  134],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  182], 90.00th=[  192], 95.00th=[  201],
     | 99.00th=[  215], 99.50th=[  224], 99.90th=[  234], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  304, max=  576, per=24.95%, avg=403.66, stdev=49.56, samples=50
   iops        : min=   76, max=  144, avg=100.90, stdev=12.38, samples=50
  lat (msec)   : 100=3.83%, 250=96.17%
  cpu          : usr=0.14%, sys=0.72%, ctx=2449, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1618KiB/s (1657kB/s), 405KiB/s-405KiB/s (414kB/s-415kB/s), io=40.0MiB (41.9MB), run=25280-25310msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=3557: Mon Jan 13 13:37:06 2025
   read: IOPS=86, BW=345KiB/s (353kB/s)(5048KiB/14637msec)
    slat (usec): min=676, max=2893, avg=1177.35, stdev=317.58
    clat (usec): min=1268, max=127217, avg=61312.79, stdev=12595.36
     lat (msec): min=2, max=129, avg=62.49, stdev=12.68
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   45], 10.00th=[   48], 20.00th=[   53],
     | 30.00th=[   56], 40.00th=[   58], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   66], 80.00th=[   70], 90.00th=[   77], 95.00th=[   82],
     | 99.00th=[  101], 99.50th=[  111], 99.90th=[  125], 99.95th=[  128],
     | 99.99th=[  128]
   bw (  KiB/s): min=  232, max=  440, per=24.91%, avg=341.48, stdev=58.15, samples=29
   iops        : min=   58, max=  110, avg=85.34, stdev=14.55, samples=29
  write: IOPS=88, BW=355KiB/s (363kB/s)(5192KiB/14637msec)
    slat (usec): min=14, max=247, avg=57.35, stdev=30.87
    clat (msec): min=30, max=205, avg=119.22, stdev=29.14
     lat (msec): min=30, max=205, avg=119.28, stdev=29.14
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   75], 10.00th=[   82], 20.00th=[   93],
     | 30.00th=[  102], 40.00th=[  110], 50.00th=[  120], 60.00th=[  127],
     | 70.00th=[  134], 80.00th=[  144], 90.00th=[  159], 95.00th=[  171],
     | 99.00th=[  188], 99.50th=[  201], 99.90th=[  205], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=  256, max=  480, per=24.68%, avg=352.24, stdev=46.88, samples=29
   iops        : min=   64, max=  120, avg=88.03, stdev=11.73, samples=29
  lat (msec)   : 2=0.08%, 4=0.12%, 20=0.16%, 50=6.33%, 100=56.64%
  lat (msec)   : 250=36.68%
  cpu          : usr=0.31%, sys=1.63%, ctx=3654, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=3558: Mon Jan 13 13:37:06 2025
   read: IOPS=85, BW=340KiB/s (349kB/s)(4976KiB/14621msec)
    slat (usec): min=37, max=2817, avg=1148.89, stdev=304.11
    clat (msec): min=2, max=125, avg=61.06, stdev=12.88
     lat (msec): min=3, max=127, avg=62.21, stdev=12.95
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   44], 10.00th=[   48], 20.00th=[   52],
     | 30.00th=[   56], 40.00th=[   58], 50.00th=[   61], 60.00th=[   64],
     | 70.00th=[   66], 80.00th=[   70], 90.00th=[   75], 95.00th=[   82],
     | 99.00th=[  106], 99.50th=[  112], 99.90th=[  124], 99.95th=[  126],
     | 99.99th=[  126]
   bw (  KiB/s): min=  264, max=  432, per=24.67%, avg=338.21, stdev=49.36, samples=29
   iops        : min=   66, max=  108, avg=84.55, stdev=12.34, samples=29
  write: IOPS=90, BW=360KiB/s (369kB/s)(5264KiB/14621msec)
    slat (usec): min=13, max=298, avg=56.43, stdev=31.27
    clat (msec): min=29, max=210, avg=118.74, stdev=28.81
     lat (msec): min=29, max=210, avg=118.80, stdev=28.81
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   74], 10.00th=[   82], 20.00th=[   94],
     | 30.00th=[  103], 40.00th=[  111], 50.00th=[  118], 60.00th=[  127],
     | 70.00th=[  133], 80.00th=[  144], 90.00th=[  157], 95.00th=[  167],
     | 99.00th=[  192], 99.50th=[  203], 99.90th=[  207], 99.95th=[  211],
     | 99.99th=[  211]
   bw (  KiB/s): min=  296, max=  464, per=25.19%, avg=359.45, stdev=39.25, samples=29
   iops        : min=   74, max=  116, avg=89.86, stdev= 9.81, samples=29
  lat (msec)   : 4=0.12%, 20=0.35%, 50=7.07%, 100=54.45%, 250=38.01%
  cpu          : usr=0.23%, sys=1.68%, ctx=3436, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=3559: Mon Jan 13 13:37:06 2025
   read: IOPS=85, BW=342KiB/s (350kB/s)(4984KiB/14588msec)
    slat (usec): min=592, max=2724, avg=1156.90, stdev=297.63
    clat (usec): min=1845, max=127442, avg=60714.13, stdev=12817.46
     lat (msec): min=3, max=129, avg=61.87, stdev=12.90
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   44], 10.00th=[   48], 20.00th=[   52],
     | 30.00th=[   55], 40.00th=[   58], 50.00th=[   61], 60.00th=[   63],
     | 70.00th=[   66], 80.00th=[   70], 90.00th=[   75], 95.00th=[   81],
     | 99.00th=[  106], 99.50th=[  112], 99.90th=[  127], 99.95th=[  128],
     | 99.99th=[  128]
   bw (  KiB/s): min=  240, max=  472, per=24.93%, avg=341.72, stdev=57.64, samples=29
   iops        : min=   60, max=  118, avg=85.38, stdev=14.47, samples=29
  write: IOPS=90, BW=360KiB/s (369kB/s)(5256KiB/14588msec)
    slat (usec): min=13, max=1216, avg=57.50, stdev=46.93
    clat (msec): min=38, max=205, avg=118.46, stdev=28.21
     lat (msec): min=38, max=205, avg=118.52, stdev=28.22
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   75], 10.00th=[   84], 20.00th=[   92],
     | 30.00th=[  103], 40.00th=[  111], 50.00th=[  117], 60.00th=[  126],
     | 70.00th=[  133], 80.00th=[  142], 90.00th=[  155], 95.00th=[  169],
     | 99.00th=[  186], 99.50th=[  197], 99.90th=[  205], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=  271, max=  472, per=25.11%, avg=358.28, stdev=43.92, samples=29
   iops        : min=   67, max=  118, avg=89.52, stdev=11.04, samples=29
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.23%, 50=7.11%
  lat (msec)   : 100=55.12%, 250=37.38%
  cpu          : usr=0.27%, sys=1.69%, ctx=3529, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=3560: Mon Jan 13 13:37:06 2025
   read: IOPS=86, BW=346KiB/s (355kB/s)(5060KiB/14613msec)
    slat (usec): min=661, max=3274, avg=1147.04, stdev=299.16
    clat (usec): min=1871, max=128674, avg=60873.16, stdev=13007.97
     lat (msec): min=3, max=131, avg=62.02, stdev=13.09
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   44], 10.00th=[   48], 20.00th=[   52],
     | 30.00th=[   55], 40.00th=[   58], 50.00th=[   61], 60.00th=[   64],
     | 70.00th=[   66], 80.00th=[   69], 90.00th=[   75], 95.00th=[   82],
     | 99.00th=[  108], 99.50th=[  116], 99.90th=[  127], 99.95th=[  129],
     | 99.99th=[  129]
   bw (  KiB/s): min=  208, max=  424, per=25.23%, avg=345.93, stdev=57.32, samples=29
   iops        : min=   52, max=  106, avg=86.48, stdev=14.33, samples=29
  write: IOPS=88, BW=354KiB/s (363kB/s)(5180KiB/14613msec)
    slat (usec): min=13, max=964, avg=57.39, stdev=40.94
    clat (msec): min=47, max=214, avg=119.36, stdev=28.65
     lat (msec): min=47, max=214, avg=119.42, stdev=28.65
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   75], 10.00th=[   84], 20.00th=[   92],
     | 30.00th=[  103], 40.00th=[  112], 50.00th=[  120], 60.00th=[  127],
     | 70.00th=[  134], 80.00th=[  144], 90.00th=[  159], 95.00th=[  169],
     | 99.00th=[  188], 99.50th=[  197], 99.90th=[  207], 99.95th=[  215],
     | 99.99th=[  215]
   bw (  KiB/s): min=  248, max=  480, per=24.73%, avg=352.83, stdev=52.52, samples=29
   iops        : min=   62, max=  120, avg=88.21, stdev=13.13, samples=29
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.08%, 20=0.39%, 50=6.64%
  lat (msec)   : 100=55.74%, 250=37.07%
  cpu          : usr=0.23%, sys=1.70%, ctx=3460, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1371KiB/s (1404kB/s), 340KiB/s-346KiB/s (349kB/s-355kB/s), io=19.6MiB (20.5MB), run=14588-14637msec
  WRITE: bw=1427KiB/s (1462kB/s), 354KiB/s-360KiB/s (363kB/s-369kB/s), io=20.4MiB (21.4MB), run=14588-14637msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=3575: Mon Jan 13 13:37:07 2025
   read: IOPS=7, BW=284MiB/s (297MB/s)(40.0MiB/141msec)
    slat (nsec): min=5629.5k, max=5629.5k, avg=5629482.00, stdev= 0.00
    clat (nsec): min=129480k, max=129480k, avg=129479780.00, stdev= 0.00
     lat (nsec): min=135119k, max=135119k, avg=135118529.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  130],  5.00th=[  130], 10.00th=[  130], 20.00th=[  130],
     | 30.00th=[  130], 40.00th=[  130], 50.00th=[  130], 60.00th=[  130],
     | 70.00th=[  130], 80.00th=[  130], 90.00th=[  130], 95.00th=[  130],
     | 99.00th=[  130], 99.50th=[  130], 99.90th=[  130], 99.95th=[  130],
     | 99.99th=[  130]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.29%, ctx=17, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=284MiB/s (297MB/s), 284MiB/s-284MiB/s (297MB/s-297MB/s), io=40.0MiB (41.9MB), run=141-141msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout