== sanity test 398c: run fio to test AIO ================= 14:08:06 (1777054086)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.325922 s, 129 MB/s
osc.lustre-OST0000-osc-ffff99838633a000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=181246: Fri Apr 24 14:08:34 2026
  write: IOPS=131, BW=527KiB/s (539kB/s)(10.0MiB/19439msec); 0 zone resets
    slat (usec): min=55, max=4553, avg=453.61, stdev=368.57
    clat (msec): min=28, max=221, avg=120.53, stdev=28.52
     lat (msec): min=29, max=221, avg=120.99, stdev=28.57
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   82], 10.00th=[   88], 20.00th=[   97],
     | 30.00th=[  104], 40.00th=[  110], 50.00th=[  116], 60.00th=[  124],
     | 70.00th=[  132], 80.00th=[  140], 90.00th=[  165], 95.00th=[  176],
     | 99.00th=[  203], 99.50th=[  218], 99.90th=[  222], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  344, max=  710, per=24.85%, avg=523.50, stdev=90.08, samples=38
   iops        : min=   86, max=  177, avg=130.68, stdev=22.53, samples=38
  lat (msec)   : 50=0.08%, 100=24.65%, 250=75.27%
  cpu          : usr=0.42%, sys=4.08%, ctx=2866, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181247: Fri Apr 24 14:08:34 2026
  write: IOPS=131, BW=528KiB/s (541kB/s)(10.0MiB/19395msec); 0 zone resets
    slat (usec): min=67, max=4271, avg=478.48, stdev=425.09
    clat (msec): min=14, max=221, avg=120.56, stdev=28.72
     lat (msec): min=15, max=221, avg=121.04, stdev=28.78
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   82], 10.00th=[   89], 20.00th=[   97],
     | 30.00th=[  104], 40.00th=[  110], 50.00th=[  116], 60.00th=[  124],
     | 70.00th=[  131], 80.00th=[  142], 90.00th=[  161], 95.00th=[  180],
     | 99.00th=[  203], 99.50th=[  205], 99.90th=[  215], 99.95th=[  215],
     | 99.99th=[  222]
   bw (  KiB/s): min=  288, max=  710, per=24.92%, avg=525.11, stdev=101.06, samples=38
   iops        : min=   72, max=  177, avg=131.08, stdev=25.23, samples=38
  lat (msec)   : 20=0.04%, 50=0.31%, 100=23.20%, 250=76.45%
  cpu          : usr=0.44%, sys=4.13%, ctx=2817, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181248: Fri Apr 24 14:08:34 2026
  write: IOPS=131, BW=527KiB/s (539kB/s)(10.0MiB/19439msec); 0 zone resets
    slat (usec): min=64, max=4945, avg=447.11, stdev=380.07
    clat (msec): min=29, max=219, avg=120.41, stdev=28.67
     lat (msec): min=30, max=220, avg=120.87, stdev=28.73
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   83], 10.00th=[   89], 20.00th=[   97],
     | 30.00th=[  103], 40.00th=[  110], 50.00th=[  116], 60.00th=[  124],
     | 70.00th=[  131], 80.00th=[  142], 90.00th=[  165], 95.00th=[  180],
     | 99.00th=[  199], 99.50th=[  209], 99.90th=[  218], 99.95th=[  220],
     | 99.99th=[  220]
   bw (  KiB/s): min=  344, max=  718, per=24.82%, avg=522.87, stdev=91.72, samples=38
   iops        : min=   86, max=  179, avg=130.53, stdev=22.93, samples=38
  lat (msec)   : 50=0.16%, 100=24.41%, 250=75.43%
  cpu          : usr=0.34%, sys=4.05%, ctx=2724, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181249: Fri Apr 24 14:08:34 2026
  write: IOPS=131, BW=527KiB/s (540kB/s)(10.0MiB/19431msec); 0 zone resets
    slat (usec): min=81, max=5307, avg=447.88, stdev=388.33
    clat (msec): min=21, max=220, avg=120.41, stdev=28.48
     lat (msec): min=22, max=223, avg=120.86, stdev=28.55
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   83], 10.00th=[   89], 20.00th=[   99],
     | 30.00th=[  104], 40.00th=[  109], 50.00th=[  115], 60.00th=[  123],
     | 70.00th=[  132], 80.00th=[  140], 90.00th=[  163], 95.00th=[  178],
     | 99.00th=[  201], 99.50th=[  209], 99.90th=[  218], 99.95th=[  218],
     | 99.99th=[  222]
   bw (  KiB/s): min=  320, max=  734, per=24.84%, avg=523.29, stdev=94.60, samples=38
   iops        : min=   80, max=  183, avg=130.63, stdev=23.62, samples=38
  lat (msec)   : 50=0.16%, 100=23.63%, 250=76.21%
  cpu          : usr=0.39%, sys=4.03%, ctx=2645, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2107KiB/s (2158kB/s), 527KiB/s-528KiB/s (539kB/s-541kB/s), io=40.0MiB (41.9MB), run=19395-19439msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=181262: Fri Apr 24 14:09:01 2026
  read: IOPS=68, BW=272KiB/s (279kB/s)(5048KiB/18534msec)
    slat (usec): min=2351, max=18689, avg=7290.55, stdev=2438.78
    clat (msec): min=44, max=182, avg=109.83, stdev=20.12
     lat (msec): min=48, max=191, avg=117.13, stdev=20.70
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[   80], 10.00th=[   87], 20.00th=[   93],
     | 30.00th=[   99], 40.00th=[  104], 50.00th=[  108], 60.00th=[  114],
     | 70.00th=[  120], 80.00th=[  128], 90.00th=[  138], 95.00th=[  144],
     | 99.00th=[  165], 99.50th=[  167], 99.90th=[  176], 99.95th=[  184],
     | 99.99th=[  184]
   bw (  KiB/s): min=  162, max=  368, per=25.12%, avg=271.81, stdev=47.54, samples=36
   iops        : min=   40, max=   92, avg=67.50, stdev=11.99, samples=36
  write: IOPS=70, BW=280KiB/s (287kB/s)(5192KiB/18534msec); 0 zone resets
    slat (usec): min=103, max=4464, avg=427.13, stdev=342.60
    clat (msec): min=67, max=182, avg=111.91, stdev=17.82
     lat (msec): min=68, max=182, avg=112.34, stdev=17.85
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[   86], 10.00th=[   91], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  106], 50.00th=[  111], 60.00th=[  116],
     | 70.00th=[  121], 80.00th=[  128], 90.00th=[  136], 95.00th=[  144],
     | 99.00th=[  159], 99.50th=[  163], 99.90th=[  167], 99.95th=[  184],
     | 99.99th=[  184]
   bw (  KiB/s): min=  190, max=  375, per=24.89%, avg=280.53, stdev=42.57, samples=36
   iops        : min=   47, max=   93, avg=69.69, stdev=10.66, samples=36
  lat (msec)   : 50=0.08%, 100=30.35%, 250=69.57%
  cpu          : usr=0.36%, sys=5.06%, ctx=4900, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181263: Fri Apr 24 14:09:01 2026
  read: IOPS=67, BW=268KiB/s (275kB/s)(4976KiB/18534msec)
    slat (usec): min=2604, max=21006, avg=7369.84, stdev=2527.46
    clat (msec): min=11, max=199, avg=110.76, stdev=20.83
     lat (msec): min=17, max=209, avg=118.14, stdev=21.52
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   82], 10.00th=[   86], 20.00th=[   94],
     | 30.00th=[   99], 40.00th=[  104], 50.00th=[  108], 60.00th=[  114],
     | 70.00th=[  120], 80.00th=[  129], 90.00th=[  140], 95.00th=[  150],
     | 99.00th=[  165], 99.50th=[  169], 99.90th=[  194], 99.95th=[  201],
     | 99.99th=[  201]
   bw (  KiB/s): min=  156, max=  391, per=24.86%, avg=268.97, stdev=48.97, samples=36
   iops        : min=   39, max=   97, avg=66.81, stdev=12.30, samples=36
  write: IOPS=71, BW=284KiB/s (291kB/s)(5264KiB/18534msec); 0 zone resets
    slat (usec): min=98, max=4186, avg=430.65, stdev=350.13
    clat (msec): min=63, max=186, avg=111.23, stdev=17.93
     lat (msec): min=63, max=186, avg=111.66, stdev=17.96
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   85], 10.00th=[   89], 20.00th=[   97],
     | 30.00th=[  102], 40.00th=[  106], 50.00th=[  111], 60.00th=[  115],
     | 70.00th=[  118], 80.00th=[  125], 90.00th=[  134], 95.00th=[  144],
     | 99.00th=[  161], 99.50th=[  167], 99.90th=[  182], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  196, max=  391, per=25.20%, avg=284.03, stdev=49.39, samples=36
   iops        : min=   49, max=   97, avg=70.58, stdev=12.35, samples=36
  lat (msec)   : 20=0.04%, 100=30.00%, 250=69.96%
  cpu          : usr=0.38%, sys=5.03%, ctx=4824, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181264: Fri Apr 24 14:09:01 2026
  read: IOPS=67, BW=269KiB/s (275kB/s)(4984KiB/18534msec)
    slat (usec): min=2514, max=22479, avg=7367.36, stdev=2490.50
    clat (msec): min=18, max=208, avg=110.81, stdev=20.66
     lat (msec): min=24, max=216, avg=118.19, stdev=21.27
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   81], 10.00th=[   87], 20.00th=[   94],
     | 30.00th=[  101], 40.00th=[  105], 50.00th=[  109], 60.00th=[  114],
     | 70.00th=[  120], 80.00th=[  127], 90.00th=[  138], 95.00th=[  148],
     | 99.00th=[  169], 99.50th=[  178], 99.90th=[  197], 99.95th=[  209],
     | 99.99th=[  209]
   bw (  KiB/s): min=  163, max=  376, per=24.93%, avg=269.75, stdev=45.07, samples=36
   iops        : min=   40, max=   94, avg=67.00, stdev=11.37, samples=36
  write: IOPS=70, BW=284KiB/s (290kB/s)(5256KiB/18534msec); 0 zone resets
    slat (usec): min=96, max=3072, avg=429.70, stdev=325.33
    clat (msec): min=16, max=193, avg=111.52, stdev=19.21
     lat (msec): min=17, max=194, avg=111.96, stdev=19.25
    clat percentiles (msec):
     |  1.00th=[   78],  5.00th=[   85], 10.00th=[   89], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  106], 50.00th=[  110], 60.00th=[  114],
     | 70.00th=[  120], 80.00th=[  128], 90.00th=[  136], 95.00th=[  148],
     | 99.00th=[  169], 99.50th=[  178], 99.90th=[  188], 99.95th=[  194],
     | 99.99th=[  194]
   bw (  KiB/s): min=  183, max=  367, per=25.10%, avg=282.92, stdev=47.80, samples=36
   iops        : min=   45, max=   91, avg=70.28, stdev=12.00, samples=36
  lat (msec)   : 20=0.08%, 100=29.34%, 250=70.59%
  cpu          : usr=0.48%, sys=4.88%, ctx=4926, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181265: Fri Apr 24 14:09:01 2026
  read: IOPS=68, BW=274KiB/s (281kB/s)(5060KiB/18461msec)
    slat (usec): min=144, max=19861, avg=7325.06, stdev=2562.51
    clat (msec): min=11, max=190, avg=110.01, stdev=21.45
     lat (msec): min=21, max=202, avg=117.34, stdev=21.86
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   80], 10.00th=[   86], 20.00th=[   93],
     | 30.00th=[  100], 40.00th=[  104], 50.00th=[  109], 60.00th=[  113],
     | 70.00th=[  120], 80.00th=[  128], 90.00th=[  140], 95.00th=[  148],
     | 99.00th=[  167], 99.50th=[  171], 99.90th=[  180], 99.95th=[  190],
     | 99.99th=[  190]
   bw (  KiB/s): min=  200, max=  391, per=25.39%, avg=274.67, stdev=46.31, samples=36
   iops        : min=   50, max=   97, avg=68.22, stdev=11.69, samples=36
  write: IOPS=70, BW=281KiB/s (287kB/s)(5180KiB/18461msec); 0 zone resets
    slat (usec): min=99, max=3625, avg=423.66, stdev=327.56
    clat (msec): min=39, max=202, avg=112.48, stdev=20.20
     lat (msec): min=40, max=202, avg=112.91, stdev=20.24
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   85], 10.00th=[   90], 20.00th=[   96],
     | 30.00th=[  101], 40.00th=[  106], 50.00th=[  111], 60.00th=[  116],
     | 70.00th=[  122], 80.00th=[  128], 90.00th=[  140], 95.00th=[  153],
     | 99.00th=[  171], 99.50th=[  176], 99.90th=[  203], 99.95th=[  203],
     | 99.99th=[  203]
   bw (  KiB/s): min=  208, max=  415, per=24.85%, avg=280.11, stdev=44.19, samples=36
   iops        : min=   52, max=  103, avg=69.58, stdev=11.02, samples=36
  lat (msec)   : 20=0.08%, 50=0.23%, 100=30.70%, 250=68.98%
  cpu          : usr=0.45%, sys=4.99%, ctx=4915, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1083KiB/s (1109kB/s), 268KiB/s-274KiB/s (275kB/s-281kB/s), io=19.6MiB (20.5MB), run=18461-18534msec
  WRITE: bw=1127KiB/s (1154kB/s), 280KiB/s-284KiB/s (287kB/s-291kB/s), io=20.4MiB (21.4MB), run=18461-18534msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=181272: Fri Apr 24 14:09:10 2026
  read: IOPS=2, BW=116MiB/s (122MB/s)(40.0MiB/345msec)
    slat (nsec): min=29876k, max=29876k, avg=29875549.00, stdev= 0.00
    clat (nsec): min=293067k, max=293067k, avg=293066559.00, stdev= 0.00
     lat (nsec): min=322949k, max=322949k, avg=322949123.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  292],  5.00th=[  292], 10.00th=[  292], 20.00th=[  292],
     | 30.00th=[  292], 40.00th=[  292], 50.00th=[  292], 60.00th=[  292],
     | 70.00th=[  292], 80.00th=[  292], 90.00th=[  292], 95.00th=[  292],
     | 99.00th=[  292], 99.50th=[  292], 99.90th=[  292], 99.95th=[  292],
     | 99.99th=[  292]
  lat (msec)   : 500=100.00%
  cpu          : usr=0.00%, sys=7.56%, ctx=7, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=116MiB/s (122MB/s), 116MiB/s-116MiB/s (122MB/s-122MB/s), io=40.0MiB (41.9MB), run=345-345msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout