== sanity test 398c: run fio to test AIO ================= 02:40:31 (1735371631)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0964346 s, 435 MB/s
osc.lustre-OST0000-osc-ffff88012ab96000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=6396: Sat Dec 28 02:40:57 2024
  write: IOPS=101, BW=404KiB/s (414kB/s)(10.0MiB/25318msec)
    slat (usec): min=12, max=657, avg=81.95, stdev=53.53
    clat (msec): min=9, max=273, avg=158.00, stdev=34.27
     lat (msec): min=9, max=273, avg=158.09, stdev=34.27
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   97], 10.00th=[  117], 20.00th=[  136],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  159], 60.00th=[  167],
     | 70.00th=[  174], 80.00th=[  184], 90.00th=[  199], 95.00th=[  211],
     | 99.00th=[  247], 99.50th=[  262], 99.90th=[  268], 99.95th=[  271],
     | 99.99th=[  275]
   bw (  KiB/s): min=  311, max=  816, per=24.93%, avg=402.58, stdev=75.94, samples=50
   iops        : min=   77, max=  204, avg=100.56, stdev=18.99, samples=50
  lat (msec)   : 10=0.04%, 20=0.04%, 50=0.16%, 100=5.62%, 250=93.44%
  lat (msec)   : 500=0.70%
  cpu          : usr=0.20%, sys=0.96%, ctx=2271, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6397: Sat Dec 28 02:40:57 2024
  write: IOPS=100, BW=404KiB/s (414kB/s)(10.0MiB/25356msec)
    slat (usec): min=12, max=546, avg=79.00, stdev=52.84
    clat (msec): min=51, max=275, avg=158.07, stdev=33.76
     lat (msec): min=51, max=275, avg=158.15, stdev=33.76
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   99], 10.00th=[  120], 20.00th=[  134],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  159], 60.00th=[  167],
     | 70.00th=[  174], 80.00th=[  184], 90.00th=[  197], 95.00th=[  211],
     | 99.00th=[  249], 99.50th=[  259], 99.90th=[  271], 99.95th=[  275],
     | 99.99th=[  275]
   bw (  KiB/s): min=  304, max=  768, per=24.85%, avg=401.40, stdev=70.92, samples=50
   iops        : min=   76, max=  192, avg=100.32, stdev=17.74, samples=50
  lat (msec)   : 100=5.47%, 250=93.79%, 500=0.74%
  cpu          : usr=0.15%, sys=0.95%, ctx=2120, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6398: Sat Dec 28 02:40:57 2024
  write: IOPS=101, BW=404KiB/s (414kB/s)(10.0MiB/25346msec)
    slat (usec): min=11, max=597, avg=79.18, stdev=53.96
    clat (msec): min=43, max=275, avg=158.11, stdev=33.32
     lat (msec): min=43, max=275, avg=158.19, stdev=33.32
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[  100], 10.00th=[  121], 20.00th=[  136],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  159], 60.00th=[  167],
     | 70.00th=[  174], 80.00th=[  182], 90.00th=[  199], 95.00th=[  213],
     | 99.00th=[  243], 99.50th=[  255], 99.90th=[  275], 99.95th=[  275],
     | 99.99th=[  275]
   bw (  KiB/s): min=  264, max=  768, per=24.91%, avg=402.36, stdev=73.56, samples=50
   iops        : min=   66, max=  192, avg=100.56, stdev=18.40, samples=50
  lat (msec)   : 50=0.08%, 100=4.96%, 250=94.30%, 500=0.66%
  cpu          : usr=0.17%, sys=0.94%, ctx=2001, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6399: Sat Dec 28 02:40:57 2024
  write: IOPS=100, BW=404KiB/s (414kB/s)(10.0MiB/25356msec)
    slat (usec): min=12, max=560, avg=79.46, stdev=54.81
    clat (msec): min=41, max=275, avg=158.21, stdev=33.23
     lat (msec): min=41, max=275, avg=158.30, stdev=33.23
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   97], 10.00th=[  120], 20.00th=[  134],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  159], 60.00th=[  167],
     | 70.00th=[  174], 80.00th=[  182], 90.00th=[  199], 95.00th=[  213],
     | 99.00th=[  243], 99.50th=[  262], 99.90th=[  275], 99.95th=[  275],
     | 99.99th=[  275]
   bw (  KiB/s): min=  288, max=  768, per=24.90%, avg=402.06, stdev=72.72, samples=50
   iops        : min=   72, max=  192, avg=100.50, stdev=18.19, samples=50
  lat (msec)   : 50=0.08%, 100=5.47%, 250=93.71%, 500=0.74%
  cpu          : usr=0.19%, sys=0.92%, ctx=2031, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1615KiB/s (1654kB/s), 404KiB/s-404KiB/s (414kB/s-414kB/s), io=40.0MiB (41.9MB), run=25318-25356msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=6429: Sat Dec 28 02:41:14 2024
   read: IOPS=83, BW=334KiB/s (342kB/s)(5048KiB/15131msec)
    slat (usec): min=733, max=3593, avg=1246.00, stdev=306.01
    clat (msec): min=6, max=139, avg=61.15, stdev=12.11
     lat (msec): min=8, max=140, avg=62.40, stdev=12.14
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   43], 10.00th=[   49], 20.00th=[   53],
     | 30.00th=[   56], 40.00th=[   58], 50.00th=[   62], 60.00th=[   65],
     | 70.00th=[   67], 80.00th=[   70], 90.00th=[   75], 95.00th=[   80],
     | 99.00th=[   93], 99.50th=[   97], 99.90th=[  129], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  264, max=  416, per=25.17%, avg=333.47, stdev=46.27, samples=30
   iops        : min=   66, max=  104, avg=83.27, stdev=11.57, samples=30
  write: IOPS=85, BW=343KiB/s (351kB/s)(5192KiB/15131msec)
    slat (usec): min=12, max=631, avg=73.09, stdev=50.57
    clat (msec): min=65, max=218, avg=125.33, stdev=25.31
     lat (msec): min=65, max=218, avg=125.40, stdev=25.31
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   85], 10.00th=[   94], 20.00th=[  105],
     | 30.00th=[  112], 40.00th=[  118], 50.00th=[  126], 60.00th=[  132],
     | 70.00th=[  138], 80.00th=[  146], 90.00th=[  159], 95.00th=[  169],
     | 99.00th=[  194], 99.50th=[  215], 99.90th=[  218], 99.95th=[  220],
     | 99.99th=[  220]
   bw (  KiB/s): min=  248, max=  440, per=24.84%, avg=342.80, stdev=45.30, samples=30
   iops        : min=   62, max=  110, avg=85.60, stdev=11.30, samples=30
  lat (msec)   : 10=0.08%, 20=0.35%, 50=5.82%, 100=50.55%, 250=43.20%
  cpu          : usr=0.33%, sys=2.23%, ctx=3027, majf=0, minf=34
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6430: Sat Dec 28 02:41:14 2024
   read: IOPS=82, BW=329KiB/s (337kB/s)(4976KiB/15122msec)
    slat (usec): min=693, max=3232, avg=1230.97, stdev=278.79
    clat (msec): min=9, max=102, avg=60.90, stdev=11.33
     lat (msec): min=11, max=104, avg=62.14, stdev=11.35
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   44], 10.00th=[   49], 20.00th=[   53],
     | 30.00th=[   55], 40.00th=[   59], 50.00th=[   62], 60.00th=[   65],
     | 70.00th=[   67], 80.00th=[   69], 90.00th=[   75], 95.00th=[   79],
     | 99.00th=[   90], 99.50th=[   95], 99.90th=[  103], 99.95th=[  104],
     | 99.99th=[  104]
   bw (  KiB/s): min=  240, max=  416, per=24.59%, avg=325.80, stdev=48.84, samples=30
   iops        : min=   60, max=  104, avg=81.40, stdev=12.19, samples=30
  write: IOPS=87, BW=348KiB/s (356kB/s)(5264KiB/15122msec)
    slat (usec): min=12, max=701, avg=74.30, stdev=48.22
    clat (msec): min=18, max=220, avg=124.62, stdev=25.83
     lat (msec): min=18, max=220, avg=124.69, stdev=25.83
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   85], 10.00th=[   93], 20.00th=[  103],
     | 30.00th=[  111], 40.00th=[  118], 50.00th=[  124], 60.00th=[  130],
     | 70.00th=[  138], 80.00th=[  146], 90.00th=[  159], 95.00th=[  167],
     | 99.00th=[  201], 99.50th=[  205], 99.90th=[  218], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  272, max=  456, per=25.15%, avg=347.13, stdev=48.75, samples=30
   iops        : min=   68, max=  114, avg=86.73, stdev=12.19, samples=30
  lat (msec)   : 10=0.04%, 20=0.35%, 50=5.98%, 100=50.98%, 250=42.66%
  cpu          : usr=0.21%, sys=2.39%, ctx=2895, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6431: Sat Dec 28 02:41:14 2024
   read: IOPS=82, BW=330KiB/s (338kB/s)(4984KiB/15104msec)
    slat (usec): min=34, max=4046, avg=1223.44, stdev=295.53
    clat (msec): min=2, max=155, avg=60.86, stdev=12.32
     lat (msec): min=3, max=157, avg=62.08, stdev=12.37
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   45], 10.00th=[   49], 20.00th=[   53],
     | 30.00th=[   55], 40.00th=[   58], 50.00th=[   61], 60.00th=[   64],
     | 70.00th=[   67], 80.00th=[   70], 90.00th=[   74], 95.00th=[   79],
     | 99.00th=[   90], 99.50th=[   97], 99.90th=[  155], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  264, max=  456, per=24.98%, avg=331.03, stdev=51.23, samples=30
   iops        : min=   66, max=  114, avg=82.63, stdev=12.87, samples=30
  write: IOPS=86, BW=348KiB/s (356kB/s)(5256KiB/15104msec)
    slat (usec): min=12, max=1358, avg=74.23, stdev=58.94
    clat (msec): min=28, max=218, avg=124.79, stdev=25.53
     lat (msec): min=28, max=218, avg=124.86, stdev=25.53
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   85], 10.00th=[   95], 20.00th=[  104],
     | 30.00th=[  111], 40.00th=[  118], 50.00th=[  125], 60.00th=[  130],
     | 70.00th=[  136], 80.00th=[  146], 90.00th=[  159], 95.00th=[  169],
     | 99.00th=[  188], 99.50th=[  205], 99.90th=[  218], 99.95th=[  220],
     | 99.99th=[  220]
   bw (  KiB/s): min=  272, max=  456, per=25.15%, avg=347.03, stdev=40.81, samples=30
   iops        : min=   68, max=  114, avg=86.63, stdev=10.24, samples=30
  lat (msec)   : 4=0.04%, 10=0.20%, 20=0.12%, 50=5.94%, 100=50.16%
  lat (msec)   : 250=43.55%
  cpu          : usr=0.23%, sys=2.37%, ctx=3072, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6432: Sat Dec 28 02:41:14 2024
   read: IOPS=83, BW=334KiB/s (342kB/s)(5060KiB/15139msec)
    slat (usec): min=704, max=4004, avg=1237.24, stdev=298.21
    clat (msec): min=2, max=115, avg=60.99, stdev=11.61
     lat (msec): min=3, max=116, avg=62.23, stdev=11.64
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   44], 10.00th=[   49], 20.00th=[   53],
     | 30.00th=[   55], 40.00th=[   58], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   67], 80.00th=[   70], 90.00th=[   75], 95.00th=[   79],
     | 99.00th=[   91], 99.50th=[   95], 99.90th=[  109], 99.95th=[  115],
     | 99.99th=[  115]
   bw (  KiB/s): min=  216, max=  480, per=25.29%, avg=335.13, stdev=58.67, samples=30
   iops        : min=   54, max=  120, avg=83.73, stdev=14.63, samples=30
  write: IOPS=85, BW=342KiB/s (350kB/s)(5180KiB/15139msec)
    slat (usec): min=13, max=641, avg=74.43, stdev=49.35
    clat (msec): min=55, max=220, avg=125.73, stdev=25.18
     lat (msec): min=56, max=220, avg=125.81, stdev=25.18
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[   86], 10.00th=[   95], 20.00th=[  106],
     | 30.00th=[  113], 40.00th=[  120], 50.00th=[  125], 60.00th=[  131],
     | 70.00th=[  138], 80.00th=[  146], 90.00th=[  159], 95.00th=[  169],
     | 99.00th=[  207], 99.50th=[  211], 99.90th=[  220], 99.95th=[  220],
     | 99.99th=[  220]
   bw (  KiB/s): min=  264, max=  432, per=24.71%, avg=341.00, stdev=34.98, samples=30
   iops        : min=   66, max=  108, avg=85.20, stdev= 8.74, samples=30
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.23%, 50=5.82%, 100=50.74%
  lat (msec)   : 250=43.09%
  cpu          : usr=0.29%, sys=2.31%, ctx=3004, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1326KiB/s (1357kB/s), 329KiB/s-334KiB/s (337kB/s-342kB/s), io=19.6MiB (20.5MB), run=15104-15139msec
  WRITE: bw=1380KiB/s (1413kB/s), 342KiB/s-348KiB/s (350kB/s-356kB/s), io=20.4MiB (21.4MB), run=15104-15139msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=6448: Sat Dec 28 02:41:15 2024
   read: IOPS=8, BW=331MiB/s (347MB/s)(40.0MiB/121msec)
    slat (nsec): min=9555.5k, max=9555.5k, avg=9555481.00, stdev= 0.00
    clat (nsec): min=105290k, max=105290k, avg=105290208.00, stdev= 0.00
     lat (nsec): min=114855k, max=114855k, avg=114854750.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  106],  5.00th=[  106], 10.00th=[  106], 20.00th=[  106],
     | 30.00th=[  106], 40.00th=[  106], 50.00th=[  106], 60.00th=[  106],
     | 70.00th=[  106], 80.00th=[  106], 90.00th=[  106], 95.00th=[  106],
     | 99.00th=[  106], 99.50th=[  106], 99.90th=[  106], 99.95th=[  106],
     | 99.99th=[  106]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=8.33%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=40.0MiB (41.9MB), run=121-121msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout