== sanity test 398c: run fio to test AIO ================= 16:28:20 (1709069300)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.224461 s, 187 MB/s
osc.lustre-OST0000-osc-ffff8800b6fa2800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=2134: Tue Feb 27 16:28:38 2024
  write: IOPS=166, BW=665KiB/s (681kB/s)(10.0MiB/15403msec)
    slat (usec): min=10, max=3549, avg=74.03, stdev=181.15
    clat (msec): min=18, max=184, avg=96.00, stdev=26.58
     lat (msec): min=18, max=184, avg=96.07, stdev=26.58
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   61], 10.00th=[   66], 20.00th=[   75],
     | 30.00th=[   81], 40.00th=[   86], 50.00th=[   91], 60.00th=[   99],
     | 70.00th=[  108], 80.00th=[  120], 90.00th=[  133], 95.00th=[  148],
     | 99.00th=[  169], 99.50th=[  171], 99.90th=[  180], 99.95th=[  184],
     | 99.99th=[  186]
   bw (  KiB/s): min=  455, max=  896, per=24.97%, avg=661.47, stdev=120.70, samples=30
   iops        : min=  113, max=  224, avg=165.13, stdev=30.29, samples=30
  lat (msec)   : 20=0.04%, 50=1.41%, 100=60.94%, 250=37.62%
  cpu          : usr=0.29%, sys=2.10%, ctx=962, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2135: Tue Feb 27 16:28:38 2024
  write: IOPS=165, BW=662KiB/s (678kB/s)(10.0MiB/15459msec)
    slat (usec): min=11, max=8491, avg=106.22, stdev=338.67
    clat (msec): min=10, max=184, avg=96.29, stdev=26.53
     lat (msec): min=10, max=184, avg=96.40, stdev=26.53
    clat percentiles (msec):
     |  1.00th=[   52],  5.00th=[   62], 10.00th=[   66], 20.00th=[   74],
     | 30.00th=[   81], 40.00th=[   86], 50.00th=[   91], 60.00th=[   99],
     | 70.00th=[  108], 80.00th=[  117], 90.00th=[  133], 95.00th=[  146],
     | 99.00th=[  174], 99.50th=[  182], 99.90th=[  186], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  407, max=  896, per=24.89%, avg=659.40, stdev=116.86, samples=30
   iops        : min=  101, max=  224, avg=164.63, stdev=29.31, samples=30
  lat (msec)   : 20=0.04%, 50=0.39%, 100=61.64%, 250=37.93%
  cpu          : usr=0.87%, sys=1.91%, ctx=1014, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2136: Tue Feb 27 16:28:38 2024
  write: IOPS=165, BW=664KiB/s (679kB/s)(10.0MiB/15433msec)
    slat (usec): min=11, max=2876, avg=88.63, stdev=227.01
    clat (msec): min=9, max=191, avg=96.15, stdev=27.26
     lat (msec): min=9, max=191, avg=96.24, stdev=27.26
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   59], 10.00th=[   67], 20.00th=[   75],
     | 30.00th=[   81], 40.00th=[   86], 50.00th=[   92], 60.00th=[   99],
     | 70.00th=[  107], 80.00th=[  118], 90.00th=[  133], 95.00th=[  150],
     | 99.00th=[  169], 99.50th=[  176], 99.90th=[  182], 99.95th=[  188],
     | 99.99th=[  192]
   bw (  KiB/s): min=  488, max=  896, per=25.00%, avg=662.37, stdev=115.21, samples=30
   iops        : min=  122, max=  224, avg=165.47, stdev=28.83, samples=30
  lat (msec)   : 10=0.04%, 20=0.12%, 50=1.37%, 100=60.51%, 250=37.97%
  cpu          : usr=0.42%, sys=2.24%, ctx=973, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2137: Tue Feb 27 16:28:38 2024
  write: IOPS=165, BW=662KiB/s (678kB/s)(10.0MiB/15461msec)
    slat (usec): min=10, max=2964, avg=75.23, stdev=182.73
    clat (msec): min=33, max=193, avg=96.34, stdev=25.99
     lat (msec): min=33, max=195, avg=96.42, stdev=25.98
    clat percentiles (msec):
     |  1.00th=[   50],  5.00th=[   62], 10.00th=[   67], 20.00th=[   75],
     | 30.00th=[   81], 40.00th=[   87], 50.00th=[   92], 60.00th=[  100],
     | 70.00th=[  109], 80.00th=[  118], 90.00th=[  131], 95.00th=[  146],
     | 99.00th=[  171], 99.50th=[  184], 99.90th=[  194], 99.95th=[  194],
     | 99.99th=[  194]
   bw (  KiB/s): min=  399, max= 1000, per=24.95%, avg=660.93, stdev=122.74, samples=30
   iops        : min=   99, max=  250, avg=165.00, stdev=30.82, samples=30
  lat (msec)   : 50=1.37%, 100=59.49%, 250=39.14%
  cpu          : usr=0.00%, sys=2.28%, ctx=775, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2649KiB/s (2713kB/s), 662KiB/s-665KiB/s (678kB/s-681kB/s), io=40.0MiB (41.9MB), run=15403-15461msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=2160: Tue Feb 27 16:28:53 2024
   read: IOPS=90, BW=362KiB/s (371kB/s)(5048KiB/13943msec)
    slat (usec): min=26, max=71162, avg=5552.50, stdev=3533.56
    clat (msec): min=13, max=225, avg=83.23, stdev=26.88
     lat (msec): min=13, max=233, avg=88.79, stdev=27.77
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   51], 10.00th=[   56], 20.00th=[   63],
     | 30.00th=[   69], 40.00th=[   74], 50.00th=[   79], 60.00th=[   85],
     | 70.00th=[   92], 80.00th=[  102], 90.00th=[  115], 95.00th=[  130],
     | 99.00th=[  186], 99.50th=[  192], 99.90th=[  226], 99.95th=[  226],
     | 99.99th=[  226]
   bw (  KiB/s): min=  175, max=  455, per=25.41%, avg=363.33, stdev=65.36, samples=27
   iops        : min=   43, max=  113, avg=90.63, stdev=16.37, samples=27
  write: IOPS=93, BW=372KiB/s (381kB/s)(5192KiB/13943msec)
    slat (usec): min=13, max=5542, avg=95.81, stdev=243.52
    clat (msec): min=14, max=213, avg=85.29, stdev=24.92
     lat (msec): min=14, max=213, avg=85.39, stdev=24.92
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   55], 10.00th=[   59], 20.00th=[   66],
     | 30.00th=[   70], 40.00th=[   75], 50.00th=[   81], 60.00th=[   88],
     | 70.00th=[   94], 80.00th=[  102], 90.00th=[  115], 95.00th=[  128],
     | 99.00th=[  184], 99.50th=[  192], 99.90th=[  207], 99.95th=[  213],
     | 99.99th=[  213]
   bw (  KiB/s): min=  199, max=  504, per=25.15%, avg=374.56, stdev=70.06, samples=27
   iops        : min=   49, max=  126, avg=93.44, stdev=17.58, samples=27
  lat (msec)   : 20=0.20%, 50=3.36%, 100=75.31%, 250=21.13%
  cpu          : usr=1.16%, sys=4.38%, ctx=2265, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2161: Tue Feb 27 16:28:53 2024
   read: IOPS=88, BW=355KiB/s (363kB/s)(4976KiB/14027msec)
    slat (usec): min=16, max=55643, avg=5510.46, stdev=3348.95
    clat (msec): min=13, max=225, avg=81.76, stdev=25.28
     lat (msec): min=22, max=230, avg=87.27, stdev=26.23
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   50], 10.00th=[   55], 20.00th=[   62],
     | 30.00th=[   67], 40.00th=[   73], 50.00th=[   79], 60.00th=[   85],
     | 70.00th=[   91], 80.00th=[  100], 90.00th=[  112], 95.00th=[  127],
     | 99.00th=[  174], 99.50th=[  199], 99.90th=[  213], 99.95th=[  226],
     | 99.99th=[  226]
   bw (  KiB/s): min=  136, max=  496, per=24.64%, avg=352.36, stdev=84.15, samples=28
   iops        : min=   34, max=  124, avg=87.96, stdev=20.98, samples=28
  write: IOPS=93, BW=375KiB/s (384kB/s)(5264KiB/14027msec)
    slat (usec): min=14, max=1805, avg=91.71, stdev=180.68
    clat (msec): min=42, max=232, avg=87.84, stdev=26.56
     lat (msec): min=42, max=232, avg=87.93, stdev=26.56
    clat percentiles (msec):
     |  1.00th=[   48],  5.00th=[   56], 10.00th=[   61], 20.00th=[   68],
     | 30.00th=[   73], 40.00th=[   79], 50.00th=[   84], 60.00th=[   90],
     | 70.00th=[   96], 80.00th=[  104], 90.00th=[  118], 95.00th=[  136],
     | 99.00th=[  192], 99.50th=[  209], 99.90th=[  232], 99.95th=[  232],
     | 99.99th=[  232]
   bw (  KiB/s): min=  224, max=  560, per=25.14%, avg=374.36, stdev=68.52, samples=28
   iops        : min=   56, max=  140, avg=93.46, stdev=17.13, samples=28
  lat (msec)   : 20=0.08%, 50=3.48%, 100=74.26%, 250=22.19%
  cpu          : usr=1.45%, sys=3.94%, ctx=2218, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2162: Tue Feb 27 16:28:53 2024
   read: IOPS=88, BW=355KiB/s (364kB/s)(4984KiB/14026msec)
    slat (usec): min=19, max=80479, avg=5496.03, stdev=3694.26
    clat (msec): min=35, max=226, avg=82.13, stdev=24.88
     lat (msec): min=40, max=234, avg=87.63, stdev=25.93
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   51], 10.00th=[   56], 20.00th=[   63],
     | 30.00th=[   68], 40.00th=[   73], 50.00th=[   79], 60.00th=[   84],
     | 70.00th=[   90], 80.00th=[  100], 90.00th=[  110], 95.00th=[  127],
     | 99.00th=[  176], 99.50th=[  190], 99.90th=[  226], 99.95th=[  228],
     | 99.99th=[  228]
   bw (  KiB/s): min=  208, max=  576, per=24.80%, avg=354.61, stdev=79.49, samples=28
   iops        : min=   52, max=  144, avg=88.50, stdev=19.83, samples=28
  write: IOPS=93, BW=375KiB/s (384kB/s)(5256KiB/14026msec)
    slat (usec): min=12, max=3381, avg=89.06, stdev=192.90
    clat (msec): min=40, max=219, avg=87.49, stdev=26.99
     lat (msec): min=40, max=219, avg=87.58, stdev=26.99
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   55], 10.00th=[   60], 20.00th=[   67],
     | 30.00th=[   72], 40.00th=[   77], 50.00th=[   83], 60.00th=[   89],
     | 70.00th=[   97], 80.00th=[  105], 90.00th=[  121], 95.00th=[  133],
     | 99.00th=[  190], 99.50th=[  197], 99.90th=[  209], 99.95th=[  220],
     | 99.99th=[  220]
   bw (  KiB/s): min=  184, max=  493, per=24.98%, avg=372.00, stdev=75.42, samples=28
   iops        : min=   46, max=  123, avg=92.86, stdev=18.82, samples=28
  lat (msec)   : 50=3.32%, 100=73.71%, 250=22.97%
  cpu          : usr=0.50%, sys=5.07%, ctx=2245, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2163: Tue Feb 27 16:28:53 2024
   read: IOPS=90, BW=361KiB/s (369kB/s)(5060KiB/14026msec)
    slat (usec): min=21, max=58079, avg=5444.80, stdev=2955.46
    clat (msec): min=13, max=222, avg=83.09, stdev=26.43
     lat (msec): min=15, max=234, avg=88.54, stdev=27.31
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   50], 10.00th=[   55], 20.00th=[   64],
     | 30.00th=[   69], 40.00th=[   74], 50.00th=[   79], 60.00th=[   85],
     | 70.00th=[   91], 80.00th=[  101], 90.00th=[  114], 95.00th=[  128],
     | 99.00th=[  186], 99.50th=[  194], 99.90th=[  220], 99.95th=[  224],
     | 99.99th=[  224]
   bw (  KiB/s): min=  184, max=  528, per=25.08%, avg=358.61, stdev=80.15, samples=28
   iops        : min=   46, max=  132, avg=89.36, stdev=19.94, samples=28
  write: IOPS=92, BW=369KiB/s (378kB/s)(5180KiB/14026msec)
    slat (usec): min=13, max=2843, avg=96.85, stdev=216.78
    clat (msec): min=41, max=229, avg=86.59, stdev=25.76
     lat (msec): min=41, max=229, avg=86.69, stdev=25.77
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   55], 10.00th=[   60], 20.00th=[   67],
     | 30.00th=[   72], 40.00th=[   77], 50.00th=[   82], 60.00th=[   89],
     | 70.00th=[   96], 80.00th=[  103], 90.00th=[  118], 95.00th=[  131],
     | 99.00th=[  192], 99.50th=[  199], 99.90th=[  218], 99.95th=[  230],
     | 99.99th=[  230]
   bw (  KiB/s): min=  216, max=  504, per=24.68%, avg=367.46, stdev=74.70, samples=28
   iops        : min=   54, max=  126, avg=91.57, stdev=18.74, samples=28
  lat (msec)   : 20=0.08%, 50=3.63%, 100=74.53%, 250=21.76%
  cpu          : usr=1.03%, sys=4.59%, ctx=2257, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1431KiB/s (1465kB/s), 355KiB/s-362KiB/s (363kB/s-371kB/s), io=19.6MiB (20.5MB), run=13943-14027msec
  WRITE: bw=1489KiB/s (1525kB/s), 369KiB/s-375KiB/s (378kB/s-384kB/s), io=20.4MiB (21.4MB), run=13943-14027msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=2178: Tue Feb 27 16:28:56 2024
   read: IOPS=1, BW=53.6MiB/s (56.2MB/s)(40.0MiB/746msec)
    slat (nsec): min=13300k, max=13300k, avg=13299752.00, stdev= 0.00
    clat (nsec): min=678999k, max=678999k, avg=678998760.00, stdev= 0.00
     lat (nsec): min=692326k, max=692326k, avg=692325909.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  676],  5.00th=[  676], 10.00th=[  676], 20.00th=[  676],
     | 30.00th=[  676], 40.00th=[  676], 50.00th=[  676], 60.00th=[  676],
     | 70.00th=[  676], 80.00th=[  676], 90.00th=[  676], 95.00th=[  676],
     | 99.00th=[  676], 99.50th=[  676], 99.90th=[  676], 99.95th=[  676],
     | 99.99th=[  676]
  lat (msec)   : 750=100.00%
  cpu          : usr=0.81%, sys=1.74%, ctx=3, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=53.6MiB/s (56.2MB/s), 53.6MiB/s-53.6MiB/s (56.2MB/s-56.2MB/s), io=40.0MiB (41.9MB), run=746-746msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout