== sanity test 398c: run fio to test AIO ================= 00:53:28 (1747803208)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.151308 s, 277 MB/s
osc.lustre-OST0000-osc-ffff8800aa49f800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16219: Wed May 21 00:54:03 2025
  write: IOPS=76, BW=308KiB/s (315kB/s)(10.0MiB/33250msec)
    slat (usec): min=13, max=318, avg=50.77, stdev=29.68
    clat (msec): min=99, max=345, avg=207.40, stdev=36.00
     lat (msec): min=99, max=345, avg=207.45, stdev=36.00
    clat percentiles (msec):
     |  1.00th=[  132],  5.00th=[  150], 10.00th=[  163], 20.00th=[  178],
     | 30.00th=[  188], 40.00th=[  199], 50.00th=[  207], 60.00th=[  215],
     | 70.00th=[  226], 80.00th=[  236], 90.00th=[  253], 95.00th=[  268],
     | 99.00th=[  292], 99.50th=[  317], 99.90th=[  342], 99.95th=[  342],
     | 99.99th=[  347]
   bw (  KiB/s): min=  232, max=  448, per=24.96%, avg=307.32, stdev=40.12, samples=66
   iops        : min=   58, max=  112, avg=76.77, stdev=10.02, samples=66
  lat (msec)   : 100=0.04%, 250=88.59%, 500=11.37%
  cpu          : usr=0.12%, sys=0.46%, ctx=1910, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16220: Wed May 21 00:54:03 2025
  write: IOPS=77, BW=308KiB/s (316kB/s)(10.0MiB/33194msec)
    slat (usec): min=13, max=319, avg=51.90, stdev=29.75
    clat (msec): min=30, max=371, avg=207.22, stdev=37.20
     lat (msec): min=31, max=371, avg=207.27, stdev=37.20
    clat percentiles (msec):
     |  1.00th=[  129],  5.00th=[  146], 10.00th=[  161], 20.00th=[  178],
     | 30.00th=[  188], 40.00th=[  199], 50.00th=[  207], 60.00th=[  215],
     | 70.00th=[  226], 80.00th=[  239], 90.00th=[  255], 95.00th=[  268],
     | 99.00th=[  296], 99.50th=[  317], 99.90th=[  347], 99.95th=[  359],
     | 99.99th=[  372]
   bw (  KiB/s): min=  216, max=  456, per=25.04%, avg=308.29, stdev=43.41, samples=66
   iops        : min=   54, max=  114, avg=77.02, stdev=10.85, samples=66
  lat (msec)   : 50=0.08%, 100=0.31%, 250=87.07%, 500=12.54%
  cpu          : usr=0.08%, sys=0.52%, ctx=1950, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16221: Wed May 21 00:54:03 2025
  write: IOPS=77, BW=308KiB/s (315kB/s)(10.0MiB/33240msec)
    slat (usec): min=13, max=274, avg=53.29, stdev=29.38
    clat (msec): min=106, max=391, avg=207.46, stdev=36.63
     lat (msec): min=106, max=391, avg=207.52, stdev=36.63
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  146], 10.00th=[  161], 20.00th=[  178],
     | 30.00th=[  188], 40.00th=[  199], 50.00th=[  207], 60.00th=[  215],
     | 70.00th=[  226], 80.00th=[  239], 90.00th=[  255], 95.00th=[  268],
     | 99.00th=[  292], 99.50th=[  317], 99.90th=[  380], 99.95th=[  380],
     | 99.99th=[  393]
   bw (  KiB/s): min=  224, max=  424, per=25.01%, avg=307.92, stdev=39.18, samples=66
   iops        : min=   56, max=  106, avg=76.92, stdev= 9.80, samples=66
  lat (msec)   : 250=87.46%, 500=12.54%
  cpu          : usr=0.10%, sys=0.50%, ctx=2074, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16222: Wed May 21 00:54:03 2025
  write: IOPS=77, BW=308KiB/s (315kB/s)(10.0MiB/33240msec)
    slat (usec): min=13, max=315, avg=49.79, stdev=28.05
    clat (msec): min=106, max=366, avg=207.43, stdev=36.28
     lat (msec): min=106, max=366, avg=207.48, stdev=36.28
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  148], 10.00th=[  163], 20.00th=[  178],
     | 30.00th=[  188], 40.00th=[  199], 50.00th=[  207], 60.00th=[  215],
     | 70.00th=[  226], 80.00th=[  236], 90.00th=[  255], 95.00th=[  268],
     | 99.00th=[  292], 99.50th=[  317], 99.90th=[  342], 99.95th=[  342],
     | 99.99th=[  368]
   bw (  KiB/s): min=  200, max=  448, per=25.01%, avg=307.83, stdev=39.00, samples=66
   iops        : min=   50, max=  112, avg=76.92, stdev= 9.74, samples=66
  lat (msec)   : 250=87.73%, 500=12.27%
  cpu          : usr=0.10%, sys=0.47%, ctx=1816, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1232KiB/s (1261kB/s), 308KiB/s-308KiB/s (315kB/s-316kB/s), io=40.0MiB (41.9MB), run=33194-33250msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16258: Wed May 21 00:54:23 2025
   read: IOPS=65, BW=263KiB/s (269kB/s)(5048KiB/19225msec)
    slat (usec): min=754, max=3933, avg=1416.55, stdev=341.07
    clat (msec): min=11, max=178, avg=80.00, stdev=15.87
     lat (msec): min=12, max=180, avg=81.42, stdev=15.92
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   57], 10.00th=[   63], 20.00th=[   68],
     | 30.00th=[   72], 40.00th=[   77], 50.00th=[   80], 60.00th=[   84],
     | 70.00th=[   88], 80.00th=[   92], 90.00th=[  100], 95.00th=[  106],
     | 99.00th=[  121], 99.50th=[  127], 99.90th=[  163], 99.95th=[  180],
     | 99.99th=[  180]
   bw (  KiB/s): min=  176, max=  408, per=25.19%, avg=262.68, stdev=62.43, samples=38
   iops        : min=   44, max=  102, avg=65.63, stdev=15.65, samples=38
  write: IOPS=67, BW=270KiB/s (277kB/s)(5192KiB/19225msec)
    slat (usec): min=14, max=1559, avg=63.29, stdev=63.10
    clat (msec): min=70, max=280, avg=157.26, stdev=38.36
     lat (msec): min=70, max=280, avg=157.33, stdev=38.36
    clat percentiles (msec):
     |  1.00th=[   78],  5.00th=[   95], 10.00th=[  108], 20.00th=[  124],
     | 30.00th=[  136], 40.00th=[  146], 50.00th=[  159], 60.00th=[  167],
     | 70.00th=[  178], 80.00th=[  190], 90.00th=[  207], 95.00th=[  222],
     | 99.00th=[  253], 99.50th=[  257], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  224, max=  368, per=24.75%, avg=268.79, stdev=37.61, samples=38
   iops        : min=   56, max=   92, avg=67.16, stdev= 9.42, samples=38
  lat (msec)   : 20=0.20%, 50=0.90%, 100=47.62%, 250=50.74%, 500=0.55%
  cpu          : usr=0.18%, sys=1.51%, ctx=2939, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16259: Wed May 21 00:54:23 2025
   read: IOPS=64, BW=259KiB/s (265kB/s)(4976KiB/19235msec)
    slat (usec): min=749, max=3018, avg=1388.21, stdev=314.33
    clat (msec): min=2, max=128, avg=78.38, stdev=16.20
     lat (msec): min=3, max=130, avg=79.77, stdev=16.26
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   55], 10.00th=[   60], 20.00th=[   67],
     | 30.00th=[   71], 40.00th=[   74], 50.00th=[   79], 60.00th=[   83],
     | 70.00th=[   87], 80.00th=[   92], 90.00th=[   99], 95.00th=[  105],
     | 99.00th=[  118], 99.50th=[  121], 99.90th=[  127], 99.95th=[  129],
     | 99.99th=[  129]
   bw (  KiB/s): min=  152, max=  408, per=24.62%, avg=256.79, stdev=60.33, samples=38
   iops        : min=   38, max=  102, avg=64.16, stdev=15.11, samples=38
  write: IOPS=68, BW=274KiB/s (280kB/s)(5264KiB/19235msec)
    slat (usec): min=12, max=330, avg=58.76, stdev=31.95
    clat (msec): min=38, max=280, avg=157.99, stdev=38.09
     lat (msec): min=38, max=280, avg=158.05, stdev=38.09
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   96], 10.00th=[  111], 20.00th=[  127],
     | 30.00th=[  138], 40.00th=[  146], 50.00th=[  157], 60.00th=[  167],
     | 70.00th=[  178], 80.00th=[  192], 90.00th=[  207], 95.00th=[  224],
     | 99.00th=[  249], 99.50th=[  259], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  192, max=  376, per=24.96%, avg=271.11, stdev=37.19, samples=38
   iops        : min=   48, max=   94, avg=67.74, stdev= 9.30, samples=38
  lat (msec)   : 4=0.08%, 20=0.39%, 50=1.09%, 100=46.72%, 250=51.21%
  lat (msec)   : 500=0.51%
  cpu          : usr=0.25%, sys=1.43%, ctx=2903, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16260: Wed May 21 00:54:23 2025
   read: IOPS=64, BW=259KiB/s (266kB/s)(4984KiB/19213msec)
    slat (usec): min=24, max=3775, avg=1395.55, stdev=333.43
    clat (usec): min=1571, max=167620, avg=78982.53, stdev=16617.55
     lat (msec): min=2, max=169, avg=80.38, stdev=16.69
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   55], 10.00th=[   60], 20.00th=[   68],
     | 30.00th=[   71], 40.00th=[   75], 50.00th=[   80], 60.00th=[   84],
     | 70.00th=[   88], 80.00th=[   92], 90.00th=[   99], 95.00th=[  105],
     | 99.00th=[  115], 99.50th=[  121], 99.90th=[  150], 99.95th=[  167],
     | 99.99th=[  167]
   bw (  KiB/s): min=  168, max=  416, per=24.96%, avg=260.37, stdev=60.27, samples=38
   iops        : min=   42, max=  104, avg=65.05, stdev=15.10, samples=38
  write: IOPS=68, BW=274KiB/s (280kB/s)(5256KiB/19213msec)
    slat (usec): min=13, max=1596, avg=61.74, stdev=53.90
    clat (msec): min=34, max=279, avg=157.47, stdev=38.24
     lat (msec): min=34, max=279, avg=157.54, stdev=38.24
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   96], 10.00th=[  112], 20.00th=[  127],
     | 30.00th=[  136], 40.00th=[  148], 50.00th=[  157], 60.00th=[  165],
     | 70.00th=[  178], 80.00th=[  190], 90.00th=[  209], 95.00th=[  224],
     | 99.00th=[  253], 99.50th=[  257], 99.90th=[  275], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  216, max=  392, per=25.04%, avg=271.95, stdev=39.55, samples=38
   iops        : min=   54, max=   98, avg=67.95, stdev= 9.93, samples=38
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.08%, 20=0.27%, 50=1.37%
  lat (msec)   : 100=46.05%, 250=51.45%, 500=0.59%
  cpu          : usr=0.18%, sys=1.50%, ctx=2998, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16261: Wed May 21 00:54:23 2025
   read: IOPS=65, BW=263KiB/s (270kB/s)(5060KiB/19213msec)
    slat (usec): min=737, max=3464, avg=1412.07, stdev=333.94
    clat (msec): min=4, max=176, avg=79.30, stdev=16.54
     lat (msec): min=6, max=177, avg=80.71, stdev=16.59
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   55], 10.00th=[   60], 20.00th=[   67],
     | 30.00th=[   71], 40.00th=[   75], 50.00th=[   80], 60.00th=[   84],
     | 70.00th=[   89], 80.00th=[   93], 90.00th=[  100], 95.00th=[  106],
     | 99.00th=[  118], 99.50th=[  125], 99.90th=[  150], 99.95th=[  178],
     | 99.99th=[  178]
   bw (  KiB/s): min=  160, max=  455, per=25.37%, avg=264.58, stdev=59.25, samples=38
   iops        : min=   40, max=  113, avg=66.11, stdev=14.75, samples=38
  write: IOPS=67, BW=270KiB/s (276kB/s)(5180KiB/19213msec)
    slat (usec): min=13, max=1535, avg=61.44, stdev=58.37
    clat (msec): min=70, max=272, avg=157.91, stdev=37.99
     lat (msec): min=70, max=272, avg=157.97, stdev=37.99
    clat percentiles (msec):
     |  1.00th=[   78],  5.00th=[   96], 10.00th=[  110], 20.00th=[  126],
     | 30.00th=[  138], 40.00th=[  146], 50.00th=[  157], 60.00th=[  165],
     | 70.00th=[  178], 80.00th=[  190], 90.00th=[  209], 95.00th=[  224],
     | 99.00th=[  257], 99.50th=[  266], 99.90th=[  268], 99.95th=[  271],
     | 99.99th=[  271]
   bw (  KiB/s): min=  184, max=  376, per=24.65%, avg=267.74, stdev=43.25, samples=38
   iops        : min=   46, max=   94, avg=66.89, stdev=10.78, samples=38
  lat (msec)   : 10=0.04%, 20=0.27%, 50=1.17%, 100=47.03%, 250=50.78%
  lat (msec)   : 500=0.70%
  cpu          : usr=0.16%, sys=1.54%, ctx=2977, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1043KiB/s (1068kB/s), 259KiB/s-263KiB/s (265kB/s-270kB/s), io=19.6MiB (20.5MB), run=19213-19235msec
  WRITE: bw=1086KiB/s (1112kB/s), 270KiB/s-274KiB/s (276kB/s-280kB/s), io=20.4MiB (21.4MB), run=19213-19235msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16280: Wed May 21 00:54:24 2025
   read: IOPS=7, BW=284MiB/s (297MB/s)(40.0MiB/141msec)
    slat (nsec): min=6358.0k, max=6358.0k, avg=6358012.00, stdev= 0.00
    clat (nsec): min=126695k, max=126695k, avg=126695474.00, stdev= 0.00
     lat (nsec): min=133063k, max=133063k, avg=133062998.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  127],  5.00th=[  127], 10.00th=[  127], 20.00th=[  127],
     | 30.00th=[  127], 40.00th=[  127], 50.00th=[  127], 60.00th=[  127],
     | 70.00th=[  127], 80.00th=[  127], 90.00th=[  127], 95.00th=[  127],
     | 99.00th=[  127], 99.50th=[  127], 99.90th=[  127], 99.95th=[  127],
     | 99.99th=[  127]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.57%, ctx=12, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=284MiB/s (297MB/s), 284MiB/s-284MiB/s (297MB/s-297MB/s), io=40.0MiB (41.9MB), run=141-141msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout