== sanity test 398c: run fio to test AIO ================= 07:02:12 (1715338932)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.110375 s, 380 MB/s
osc.lustre-OST0000-osc-ffff8800a7d7c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=1147: Fri May 10 07:02:45 2024
  write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31022msec)
    slat (usec): min=13, max=697, avg=89.85, stdev=54.40
    clat (msec): min=78, max=310, avg=193.64, stdev=33.61
     lat (msec): min=78, max=310, avg=193.74, stdev=33.61
    clat percentiles (msec):
     |  1.00th=[  105],  5.00th=[  136], 10.00th=[  150], 20.00th=[  167],
     | 30.00th=[  180], 40.00th=[  188], 50.00th=[  194], 60.00th=[  203],
     | 70.00th=[  211], 80.00th=[  222], 90.00th=[  236], 95.00th=[  247],
     | 99.00th=[  271], 99.50th=[  275], 99.90th=[  309], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  KiB/s): min=  264, max=  504, per=24.89%, avg=328.32, stdev=40.93, samples=62
   iops        : min=   66, max=  126, avg=82.03, stdev=10.26, samples=62
  lat (msec)   : 100=0.16%, 250=95.66%, 500=4.18%
  cpu          : usr=0.15%, sys=0.81%, ctx=2708, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=1148: Fri May 10 07:02:45 2024
  write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/30999msec)
    slat (usec): min=13, max=736, avg=93.17, stdev=58.27
    clat (msec): min=21, max=298, avg=193.51, stdev=35.06
     lat (msec): min=21, max=298, avg=193.60, stdev=35.06
    clat percentiles (msec):
     |  1.00th=[  100],  5.00th=[  133], 10.00th=[  148], 20.00th=[  169],
     | 30.00th=[  180], 40.00th=[  188], 50.00th=[  197], 60.00th=[  203],
     | 70.00th=[  211], 80.00th=[  224], 90.00th=[  236], 95.00th=[  247],
     | 99.00th=[  271], 99.50th=[  275], 99.90th=[  288], 99.95th=[  300],
     | 99.99th=[  300]
   bw (  KiB/s): min=  263, max=  512, per=24.98%, avg=329.49, stdev=48.42, samples=61
   iops        : min=   65, max=  128, avg=82.31, stdev=12.13, samples=61
  lat (msec)   : 50=0.20%, 100=1.02%, 250=95.00%, 500=3.79%
  cpu          : usr=0.14%, sys=0.82%, ctx=2736, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=1149: Fri May 10 07:02:45 2024
  write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31023msec)
    slat (usec): min=14, max=585, avg=90.62, stdev=54.37
    clat (msec): min=74, max=310, avg=193.58, stdev=33.69
     lat (msec): min=75, max=310, avg=193.68, stdev=33.70
    clat percentiles (msec):
     |  1.00th=[  110],  5.00th=[  136], 10.00th=[  150], 20.00th=[  167],
     | 30.00th=[  178], 40.00th=[  188], 50.00th=[  194], 60.00th=[  203],
     | 70.00th=[  211], 80.00th=[  222], 90.00th=[  236], 95.00th=[  249],
     | 99.00th=[  271], 99.50th=[  275], 99.90th=[  300], 99.95th=[  300],
     | 99.99th=[  309]
   bw (  KiB/s): min=  272, max=  472, per=24.89%, avg=328.31, stdev=40.61, samples=62
   iops        : min=   68, max=  118, avg=82.02, stdev=10.16, samples=62
  lat (msec)   : 100=0.78%, 250=95.16%, 500=4.06%
  cpu          : usr=0.16%, sys=0.79%, ctx=2705, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=1150: Fri May 10 07:02:45 2024
  write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31033msec)
    slat (usec): min=14, max=1063, avg=91.27, stdev=59.77
    clat (msec): min=20, max=320, avg=193.71, stdev=33.80
     lat (msec): min=21, max=320, avg=193.80, stdev=33.80
    clat percentiles (msec):
     |  1.00th=[  112],  5.00th=[  134], 10.00th=[  150], 20.00th=[  167],
     | 30.00th=[  180], 40.00th=[  188], 50.00th=[  194], 60.00th=[  203],
     | 70.00th=[  211], 80.00th=[  222], 90.00th=[  236], 95.00th=[  247],
     | 99.00th=[  268], 99.50th=[  275], 99.90th=[  300], 99.95th=[  309],
     | 99.99th=[  321]
   bw (  KiB/s): min=  271, max=  464, per=24.89%, avg=328.32, stdev=41.12, samples=62
   iops        : min=   67, max=  116, avg=82.03, stdev=10.30, samples=62
  lat (msec)   : 50=0.04%, 100=0.16%, 250=96.21%, 500=3.59%
  cpu          : usr=0.17%, sys=0.78%, ctx=2790, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1320KiB/s (1352kB/s), 330KiB/s-330KiB/s (338kB/s-338kB/s), io=40.0MiB (41.9MB), run=30999-31033msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=1185: Fri May 10 07:03:05 2024
   read: IOPS=66, BW=266KiB/s (272kB/s)(5048KiB/18976msec)
    slat (usec): min=16, max=2770, avg=1301.88, stdev=320.99
    clat (usec): min=1340, max=163009, avg=79592.02, stdev=14607.10
     lat (msec): min=2, max=164, avg=80.90, stdev=14.62
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   58], 10.00th=[   65], 20.00th=[   69],
     | 30.00th=[   72], 40.00th=[   77], 50.00th=[   80], 60.00th=[   83],
     | 70.00th=[   86], 80.00th=[   90], 90.00th=[   97], 95.00th=[  102],
     | 99.00th=[  114], 99.50th=[  118], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  152, max=  376, per=25.15%, avg=265.86, stdev=52.85, samples=37
   iops        : min=   38, max=   94, avg=66.41, stdev=13.22, samples=37
  write: IOPS=68, BW=274KiB/s (280kB/s)(5192KiB/18976msec)
    slat (usec): min=14, max=1407, avg=68.84, stdev=70.37
    clat (msec): min=67, max=261, avg=155.08, stdev=35.84
     lat (msec): min=67, max=261, avg=155.15, stdev=35.84
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[   99], 10.00th=[  110], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  144], 50.00th=[  153], 60.00th=[  165],
     | 70.00th=[  176], 80.00th=[  188], 90.00th=[  205], 95.00th=[  215],
     | 99.00th=[  236], 99.50th=[  245], 99.90th=[  251], 99.95th=[  262],
     | 99.99th=[  262]
   bw (  KiB/s): min=  176, max=  352, per=24.86%, avg=273.43, stdev=39.04, samples=37
   iops        : min=   44, max=   88, avg=68.30, stdev= 9.76, samples=37
  lat (msec)   : 2=0.04%, 20=0.31%, 50=0.39%, 100=48.16%, 250=50.98%
  lat (msec)   : 500=0.12%
  cpu          : usr=0.23%, sys=1.43%, ctx=3564, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=1186: Fri May 10 07:03:05 2024
   read: IOPS=65, BW=263KiB/s (269kB/s)(4976KiB/18956msec)
    slat (usec): min=14, max=2644, avg=1313.33, stdev=306.02
    clat (usec): min=1812, max=187824, avg=79381.21, stdev=15615.23
     lat (msec): min=3, max=189, avg=80.70, stdev=15.63
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   58], 10.00th=[   64], 20.00th=[   69],
     | 30.00th=[   73], 40.00th=[   77], 50.00th=[   81], 60.00th=[   84],
     | 70.00th=[   86], 80.00th=[   91], 90.00th=[   96], 95.00th=[  101],
     | 99.00th=[  115], 99.50th=[  132], 99.90th=[  188], 99.95th=[  188],
     | 99.99th=[  188]
   bw (  KiB/s): min=  168, max=  384, per=24.91%, avg=263.27, stdev=57.98, samples=37
   iops        : min=   42, max=   96, avg=65.76, stdev=14.48, samples=37
  write: IOPS=69, BW=278KiB/s (284kB/s)(5264KiB/18956msec)
    slat (usec): min=14, max=442, avg=65.22, stdev=39.13
    clat (msec): min=17, max=255, avg=154.01, stdev=37.83
     lat (msec): min=17, max=255, avg=154.07, stdev=37.83
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   95], 10.00th=[  106], 20.00th=[  123],
     | 30.00th=[  132], 40.00th=[  144], 50.00th=[  153], 60.00th=[  163],
     | 70.00th=[  176], 80.00th=[  188], 90.00th=[  205], 95.00th=[  218],
     | 99.00th=[  241], 99.50th=[  245], 99.90th=[  255], 99.95th=[  255],
     | 99.99th=[  255]
   bw (  KiB/s): min=  207, max=  352, per=25.19%, avg=277.11, stdev=37.61, samples=37
   iops        : min=   51, max=   88, avg=69.22, stdev= 9.43, samples=37
  lat (msec)   : 2=0.04%, 4=0.04%, 20=0.51%, 50=0.59%, 100=48.83%
  lat (msec)   : 250=49.77%, 500=0.23%
  cpu          : usr=0.29%, sys=1.40%, ctx=3192, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=1187: Fri May 10 07:03:05 2024
   read: IOPS=65, BW=263KiB/s (269kB/s)(4984KiB/18976msec)
    slat (usec): min=13, max=3194, avg=1303.97, stdev=313.86
    clat (usec): min=1551, max=141612, avg=79647.41, stdev=14679.93
     lat (msec): min=2, max=142, avg=80.95, stdev=14.70
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   58], 10.00th=[   64], 20.00th=[   70],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   81], 60.00th=[   84],
     | 70.00th=[   87], 80.00th=[   91], 90.00th=[   97], 95.00th=[  101],
     | 99.00th=[  113], 99.50th=[  116], 99.90th=[  126], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  152, max=  400, per=24.97%, avg=263.92, stdev=55.63, samples=37
   iops        : min=   38, max=  100, avg=65.92, stdev=13.94, samples=37
  write: IOPS=69, BW=277KiB/s (284kB/s)(5256KiB/18976msec)
    slat (usec): min=12, max=260, avg=64.59, stdev=35.40
    clat (msec): min=32, max=252, avg=154.16, stdev=37.06
     lat (msec): min=32, max=252, avg=154.23, stdev=37.06
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   97], 10.00th=[  108], 20.00th=[  122],
     | 30.00th=[  133], 40.00th=[  144], 50.00th=[  153], 60.00th=[  163],
     | 70.00th=[  174], 80.00th=[  188], 90.00th=[  205], 95.00th=[  218],
     | 99.00th=[  230], 99.50th=[  236], 99.90th=[  253], 99.95th=[  253],
     | 99.99th=[  253]
   bw (  KiB/s): min=  192, max=  336, per=25.00%, avg=274.95, stdev=38.21, samples=37
   iops        : min=   48, max=   84, avg=68.68, stdev= 9.53, samples=37
  lat (msec)   : 2=0.04%, 20=0.43%, 50=0.47%, 100=48.32%, 250=50.66%
  lat (msec)   : 500=0.08%
  cpu          : usr=0.20%, sys=1.47%, ctx=3397, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=1188: Fri May 10 07:03:05 2024
   read: IOPS=66, BW=267KiB/s (273kB/s)(5060KiB/18977msec)
    slat (usec): min=32, max=2871, avg=1299.22, stdev=306.27
    clat (msec): min=2, max=126, avg=79.69, stdev=13.89
     lat (msec): min=3, max=127, avg=80.99, stdev=13.90
    clat percentiles (msec):
     |  1.00th=[   50],  5.00th=[   59], 10.00th=[   64], 20.00th=[   68],
     | 30.00th=[   72], 40.00th=[   78], 50.00th=[   81], 60.00th=[   84],
     | 70.00th=[   87], 80.00th=[   91], 90.00th=[   97], 95.00th=[  102],
     | 99.00th=[  112], 99.50th=[  113], 99.90th=[  127], 99.95th=[  127],
     | 99.99th=[  127]
   bw (  KiB/s): min=  176, max=  408, per=25.32%, avg=267.65, stdev=54.72, samples=37
   iops        : min=   44, max=  102, avg=66.89, stdev=13.67, samples=37
  write: IOPS=68, BW=273KiB/s (280kB/s)(5180KiB/18977msec)
    slat (usec): min=14, max=670, avg=67.66, stdev=40.07
    clat (msec): min=32, max=256, avg=155.18, stdev=37.25
     lat (msec): min=32, max=256, avg=155.25, stdev=37.25
    clat percentiles (msec):
     |  1.00th=[   82],  5.00th=[   99], 10.00th=[  109], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  144], 50.00th=[  153], 60.00th=[  163],
     | 70.00th=[  176], 80.00th=[  188], 90.00th=[  209], 95.00th=[  222],
     | 99.00th=[  241], 99.50th=[  245], 99.90th=[  253], 99.95th=[  257],
     | 99.99th=[  257]
   bw (  KiB/s): min=  208, max=  352, per=24.69%, avg=271.54, stdev=35.65, samples=37
   iops        : min=   52, max=   88, avg=67.86, stdev= 8.89, samples=37
  lat (msec)   : 4=0.08%, 10=0.04%, 20=0.12%, 50=0.39%, 100=48.87%
  lat (msec)   : 250=50.39%, 500=0.12%
  cpu          : usr=0.22%, sys=1.47%, ctx=3361, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1057KiB/s (1083kB/s), 263KiB/s-267KiB/s (269kB/s-273kB/s), io=19.6MiB (20.5MB), run=18956-18977msec
  WRITE: bw=1101KiB/s (1127kB/s), 273KiB/s-278KiB/s (280kB/s-284kB/s), io=20.4MiB (21.4MB), run=18956-18977msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=1207: Fri May 10 07:03:06 2024
   read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=9275.9k, max=9275.9k, avg=9275865.00, stdev= 0.00
    clat (nsec): min=92930k, max=92930k, avg=92930008.00, stdev= 0.00
     lat (nsec): min=102221k, max=102221k, avg=102221060.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[92799],  5.00th=[92799], 10.00th=[92799], 20.00th=[92799],
     | 30.00th=[92799], 40.00th=[92799], 50.00th=[92799], 60.00th=[92799],
     | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799],
     | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799],
     | 99.99th=[92799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.82%, ctx=15, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout