== sanity test 398c: run fio to test AIO ================= 00:55:11 (1735106111)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.149489 s, 281 MB/s
osc.lustre-OST0000-osc-ffff8800a8db8000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=5134: Wed Dec 25 00:55:38 2024
  write: IOPS=100, BW=402KiB/s (412kB/s)(10.0MiB/25459msec)
    slat (usec): min=12, max=614, avg=59.23, stdev=37.61
    clat (msec): min=60, max=249, avg=158.87, stdev=26.13
     lat (msec): min=60, max=249, avg=158.93, stdev=26.13
    clat percentiles (msec):
     |  1.00th=[  101],  5.00th=[  114], 10.00th=[  125], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  159], 60.00th=[  165],
     | 70.00th=[  174], 80.00th=[  182], 90.00th=[  194], 95.00th=[  203],
     | 99.00th=[  218], 99.50th=[  220], 99.90th=[  251], 99.95th=[  251],
     | 99.99th=[  251]
   bw (  KiB/s): min=  312, max=  600, per=24.94%, avg=400.78, stdev=47.71, samples=50
   iops        : min=   78, max=  150, avg=100.18, stdev=11.93, samples=50
  lat (msec)   : 100=1.05%, 250=98.95%
  cpu          : usr=0.13%, sys=0.68%, ctx=2184, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5135: Wed Dec 25 00:55:38 2024
  write: IOPS=100, BW=402KiB/s (412kB/s)(10.0MiB/25459msec)
    slat (usec): min=13, max=477, avg=58.88, stdev=37.59
    clat (msec): min=16, max=241, avg=158.95, stdev=26.28
     lat (msec): min=16, max=241, avg=159.01, stdev=26.27
    clat percentiles (msec):
     |  1.00th=[  100],  5.00th=[  114], 10.00th=[  123], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  192], 95.00th=[  203],
     | 99.00th=[  218], 99.50th=[  224], 99.90th=[  234], 99.95th=[  236],
     | 99.99th=[  241]
   bw (  KiB/s): min=  320, max=  632, per=24.90%, avg=400.22, stdev=49.54, samples=50
   iops        : min=   80, max=  158, avg=99.98, stdev=12.42, samples=50
  lat (msec)   : 20=0.04%, 100=1.33%, 250=98.63%
  cpu          : usr=0.15%, sys=0.64%, ctx=2129, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5136: Wed Dec 25 00:55:38 2024
  write: IOPS=100, BW=402KiB/s (412kB/s)(10.0MiB/25442msec)
    slat (usec): min=12, max=571, avg=60.02, stdev=36.40
    clat (msec): min=20, max=278, avg=158.87, stdev=27.37
     lat (msec): min=20, max=278, avg=158.93, stdev=27.37
    clat percentiles (msec):
     |  1.00th=[   99],  5.00th=[  112], 10.00th=[  123], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  174], 80.00th=[  182], 90.00th=[  192], 95.00th=[  203],
     | 99.00th=[  218], 99.50th=[  228], 99.90th=[  253], 99.95th=[  262],
     | 99.99th=[  279]
   bw (  KiB/s): min=  320, max=  528, per=24.98%, avg=401.50, stdev=43.45, samples=50
   iops        : min=   80, max=  132, avg=100.30, stdev=10.88, samples=50
  lat (msec)   : 50=0.23%, 100=1.05%, 250=98.59%, 500=0.12%
  cpu          : usr=0.18%, sys=0.66%, ctx=2248, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5137: Wed Dec 25 00:55:38 2024
  write: IOPS=100, BW=402KiB/s (412kB/s)(10.0MiB/25474msec)
    slat (usec): min=12, max=482, avg=60.60, stdev=39.23
    clat (msec): min=59, max=234, avg=159.02, stdev=25.94
     lat (msec): min=59, max=234, avg=159.08, stdev=25.94
    clat percentiles (msec):
     |  1.00th=[  102],  5.00th=[  114], 10.00th=[  124], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  159], 60.00th=[  165],
     | 70.00th=[  174], 80.00th=[  180], 90.00th=[  194], 95.00th=[  203],
     | 99.00th=[  218], 99.50th=[  222], 99.90th=[  232], 99.95th=[  232],
     | 99.99th=[  234]
   bw (  KiB/s): min=  328, max=  568, per=24.86%, avg=399.44, stdev=43.78, samples=50
   iops        : min=   82, max=  142, avg=99.80, stdev=10.95, samples=50
  lat (msec)   : 100=0.82%, 250=99.18%
  cpu          : usr=0.13%, sys=0.66%, ctx=2396, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1608KiB/s (1647kB/s), 402KiB/s-402KiB/s (412kB/s-412kB/s), io=40.0MiB (41.9MB), run=25442-25474msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=5168: Wed Dec 25 00:55:54 2024
   read: IOPS=83, BW=333KiB/s (341kB/s)(5048KiB/15176msec)
    slat (usec): min=24, max=4031, avg=1139.24, stdev=312.37
    clat (usec): min=1577, max=174556, avg=63674.73, stdev=13301.62
     lat (msec): min=2, max=175, avg=64.82, stdev=13.36
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   45], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   62], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   69], 80.00th=[   72], 90.00th=[   78], 95.00th=[   81],
     | 99.00th=[   92], 99.50th=[  106], 99.90th=[  174], 99.95th=[  176],
     | 99.99th=[  176]
   bw (  KiB/s): min=  216, max=  440, per=25.30%, avg=333.50, stdev=62.72, samples=30
   iops        : min=   54, max=  110, avg=83.30, stdev=15.74, samples=30
  write: IOPS=85, BW=342KiB/s (350kB/s)(5192KiB/15176msec)
    slat (usec): min=12, max=1208, avg=52.18, stdev=43.90
    clat (msec): min=27, max=267, avg=123.81, stdev=30.16
     lat (msec): min=27, max=267, avg=123.86, stdev=30.16
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   79], 10.00th=[   88], 20.00th=[   97],
     | 30.00th=[  105], 40.00th=[  114], 50.00th=[  124], 60.00th=[  132],
     | 70.00th=[  140], 80.00th=[  150], 90.00th=[  161], 95.00th=[  174],
     | 99.00th=[  211], 99.50th=[  213], 99.90th=[  268], 99.95th=[  268],
     | 99.99th=[  268]
   bw (  KiB/s): min=  280, max=  416, per=24.95%, avg=342.30, stdev=34.54, samples=30
   iops        : min=   70, max=  104, avg=85.50, stdev= 8.63, samples=30
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.20%, 20=0.08%, 50=4.73%
  lat (msec)   : 100=55.82%, 250=38.95%, 500=0.08%
  cpu          : usr=0.22%, sys=1.48%, ctx=3152, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5169: Wed Dec 25 00:55:54 2024
   read: IOPS=81, BW=327KiB/s (335kB/s)(4976KiB/15224msec)
    slat (usec): min=647, max=3041, avg=1142.43, stdev=271.63
    clat (msec): min=9, max=126, avg=63.62, stdev=11.74
     lat (msec): min=10, max=128, avg=64.76, stdev=11.78
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   47], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   62], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   69], 80.00th=[   72], 90.00th=[   78], 95.00th=[   82],
     | 99.00th=[   91], 99.50th=[  122], 99.90th=[  125], 99.95th=[  127],
     | 99.99th=[  127]
   bw (  KiB/s): min=  232, max=  416, per=24.70%, avg=325.60, stdev=51.94, samples=30
   iops        : min=   58, max=  104, avg=81.40, stdev=12.98, samples=30
  write: IOPS=86, BW=346KiB/s (354kB/s)(5264KiB/15224msec)
    slat (usec): min=12, max=548, avg=52.05, stdev=28.54
    clat (msec): min=53, max=223, avg=123.46, stdev=28.58
     lat (msec): min=53, max=223, avg=123.51, stdev=28.58
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   81], 10.00th=[   87], 20.00th=[   99],
     | 30.00th=[  106], 40.00th=[  115], 50.00th=[  123], 60.00th=[  130],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  161], 95.00th=[  171],
     | 99.00th=[  197], 99.50th=[  209], 99.90th=[  222], 99.95th=[  224],
     | 99.99th=[  224]
   bw (  KiB/s): min=  272, max=  496, per=25.25%, avg=346.40, stdev=44.52, samples=30
   iops        : min=   68, max=  124, avg=86.60, stdev=11.13, samples=30
  lat (msec)   : 10=0.04%, 20=0.20%, 50=4.57%, 100=55.12%, 250=40.08%
  cpu          : usr=0.26%, sys=1.44%, ctx=3227, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5170: Wed Dec 25 00:55:54 2024
   read: IOPS=81, BW=327KiB/s (335kB/s)(4984KiB/15224msec)
    slat (usec): min=639, max=3604, avg=1142.59, stdev=278.57
    clat (usec): min=1600, max=123348, avg=63327.93, stdev=11472.83
     lat (msec): min=2, max=124, avg=64.47, stdev=11.52
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   46], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   62], 50.00th=[   64], 60.00th=[   67],
     | 70.00th=[   69], 80.00th=[   72], 90.00th=[   77], 95.00th=[   81],
     | 99.00th=[   88], 99.50th=[  116], 99.90th=[  123], 99.95th=[  124],
     | 99.99th=[  124]
   bw (  KiB/s): min=  216, max=  448, per=24.94%, avg=328.70, stdev=57.95, samples=30
   iops        : min=   54, max=  112, avg=82.10, stdev=14.53, samples=30
  write: IOPS=86, BW=345KiB/s (354kB/s)(5256KiB/15224msec)
    slat (usec): min=12, max=876, avg=52.16, stdev=32.57
    clat (msec): min=57, max=221, avg=123.74, stdev=28.87
     lat (msec): min=57, max=221, avg=123.79, stdev=28.87
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   80], 10.00th=[   87], 20.00th=[  100],
     | 30.00th=[  105], 40.00th=[  115], 50.00th=[  124], 60.00th=[  131],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  163], 95.00th=[  176],
     | 99.00th=[  197], 99.50th=[  211], 99.90th=[  220], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  248, max=  432, per=25.05%, avg=343.63, stdev=44.53, samples=30
   iops        : min=   62, max=  108, avg=85.83, stdev=11.13, samples=30
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.04%, 20=0.20%, 50=4.34%
  lat (msec)   : 100=54.80%, 250=40.55%
  cpu          : usr=0.24%, sys=1.49%, ctx=3011, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5171: Wed Dec 25 00:55:54 2024
   read: IOPS=83, BW=332KiB/s (340kB/s)(5060KiB/15224msec)
    slat (usec): min=652, max=3058, avg=1140.62, stdev=271.83
    clat (msec): min=3, max=168, avg=64.32, stdev=12.69
     lat (msec): min=5, max=169, avg=65.46, stdev=12.73
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   48], 10.00th=[   52], 20.00th=[   56],
     | 30.00th=[   59], 40.00th=[   62], 50.00th=[   64], 60.00th=[   67],
     | 70.00th=[   69], 80.00th=[   73], 90.00th=[   79], 95.00th=[   83],
     | 99.00th=[   95], 99.50th=[  121], 99.90th=[  167], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  224, max=  456, per=25.32%, avg=333.77, stdev=62.39, samples=30
   iops        : min=   56, max=  114, avg=83.37, stdev=15.62, samples=30
  write: IOPS=85, BW=340KiB/s (348kB/s)(5180KiB/15224msec)
    slat (usec): min=13, max=1078, avg=52.92, stdev=37.56
    clat (msec): min=53, max=222, avg=123.54, stdev=29.01
     lat (msec): min=53, max=222, avg=123.60, stdev=29.01
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[   80], 10.00th=[   87], 20.00th=[   99],
     | 30.00th=[  106], 40.00th=[  114], 50.00th=[  123], 60.00th=[  131],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  163], 95.00th=[  174],
     | 99.00th=[  199], 99.50th=[  211], 99.90th=[  222], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  240, max=  464, per=24.74%, avg=339.37, stdev=43.18, samples=30
   iops        : min=   60, max=  116, avg=84.77, stdev=10.79, samples=30
  lat (msec)   : 4=0.04%, 20=0.27%, 50=3.40%, 100=56.64%, 250=39.65%
  cpu          : usr=0.18%, sys=1.48%, ctx=3392, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1318KiB/s (1350kB/s), 327KiB/s-333KiB/s (335kB/s-341kB/s), io=19.6MiB (20.5MB), run=15176-15224msec
  WRITE: bw=1372KiB/s (1405kB/s), 340KiB/s-346KiB/s (348kB/s-354kB/s), io=20.4MiB (21.4MB), run=15176-15224msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=5187: Wed Dec 25 00:55:55 2024
   read: IOPS=11, BW=465MiB/s (488MB/s)(40.0MiB/86msec)
    slat (nsec): min=5613.6k, max=5613.6k, avg=5613580.00, stdev= 0.00
    clat (nsec): min=75771k, max=75771k, avg=75770923.00, stdev= 0.00
     lat (nsec): min=81394k, max=81394k, avg=81393810.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[76022],  5.00th=[76022], 10.00th=[76022], 20.00th=[76022],
     | 30.00th=[76022], 40.00th=[76022], 50.00th=[76022], 60.00th=[76022],
     | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022],
     | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.88%, ctx=22, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=465MiB/s (488MB/s), 465MiB/s-465MiB/s (488MB/s-488MB/s), io=40.0MiB (41.9MB), run=86-86msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout