== sanity test 398c: run fio to test AIO ================= 10:00:20 (1747404020)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0881661 s, 476 MB/s
osc.lustre-OST0000-osc-ffff8800b3b02800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23865: Fri May 16 10:00:45 2025
  write: IOPS=110, BW=444KiB/s (454kB/s)(10.0MiB/23072msec)
    slat (usec): min=12, max=253, avg=57.70, stdev=31.04
    clat (msec): min=44, max=233, avg=144.08, stdev=26.35
     lat (msec): min=44, max=233, avg=144.14, stdev=26.35
    clat percentiles (msec):
     |  1.00th=[   87],  5.00th=[  100], 10.00th=[  109], 20.00th=[  122],
     | 30.00th=[  131], 40.00th=[  140], 50.00th=[  146], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  165], 90.00th=[  178], 95.00th=[  188],
     | 99.00th=[  209], 99.50th=[  215], 99.90th=[  232], 99.95th=[  232],
     | 99.99th=[  234]
   bw (  KiB/s): min=  352, max=  640, per=24.94%, avg=442.43, stdev=61.92, samples=46
   iops        : min=   88, max=  160, avg=110.61, stdev=15.48, samples=46
  lat (msec)   : 50=0.04%, 100=6.05%, 250=93.91%
  cpu          : usr=0.14%, sys=0.75%, ctx=2108, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23866: Fri May 16 10:00:45 2025
  write: IOPS=110, BW=444KiB/s (454kB/s)(10.0MiB/23079msec)
    slat (usec): min=11, max=311, avg=58.78, stdev=32.32
    clat (msec): min=43, max=233, avg=144.11, stdev=26.67
     lat (msec): min=43, max=233, avg=144.17, stdev=26.68
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  100], 10.00th=[  109], 20.00th=[  122],
     | 30.00th=[  131], 40.00th=[  138], 50.00th=[  146], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  167], 90.00th=[  178], 95.00th=[  188],
     | 99.00th=[  209], 99.50th=[  213], 99.90th=[  220], 99.95th=[  222],
     | 99.99th=[  234]
   bw (  KiB/s): min=  344, max=  640, per=24.95%, avg=442.59, stdev=58.61, samples=46
   iops        : min=   86, max=  160, avg=110.63, stdev=14.66, samples=46
  lat (msec)   : 50=0.08%, 100=6.09%, 250=93.83%
  cpu          : usr=0.15%, sys=0.74%, ctx=2229, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23867: Fri May 16 10:00:45 2025
  write: IOPS=110, BW=444KiB/s (455kB/s)(10.0MiB/23064msec)
    slat (usec): min=11, max=484, avg=58.26, stdev=33.48
    clat (msec): min=54, max=290, avg=144.01, stdev=27.83
     lat (msec): min=54, max=290, avg=144.07, stdev=27.84
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[   99], 10.00th=[  108], 20.00th=[  122],
     | 30.00th=[  131], 40.00th=[  140], 50.00th=[  144], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  167], 90.00th=[  180], 95.00th=[  188],
     | 99.00th=[  211], 99.50th=[  222], 99.90th=[  236], 99.95th=[  247],
     | 99.99th=[  292]
   bw (  KiB/s): min=  359, max=  648, per=24.94%, avg=442.52, stdev=62.85, samples=46
   iops        : min=   89, max=  162, avg=110.57, stdev=15.74, samples=46
  lat (msec)   : 100=6.48%, 250=93.48%, 500=0.04%
  cpu          : usr=0.16%, sys=0.75%, ctx=2150, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23868: Fri May 16 10:00:45 2025
  write: IOPS=110, BW=444KiB/s (454kB/s)(10.0MiB/23079msec)
    slat (usec): min=11, max=473, avg=58.48, stdev=33.34
    clat (msec): min=53, max=230, avg=144.12, stdev=26.28
     lat (msec): min=53, max=230, avg=144.18, stdev=26.28
    clat percentiles (msec):
     |  1.00th=[   87],  5.00th=[  100], 10.00th=[  109], 20.00th=[  122],
     | 30.00th=[  131], 40.00th=[  140], 50.00th=[  144], 60.00th=[  153],
     | 70.00th=[  159], 80.00th=[  167], 90.00th=[  180], 95.00th=[  186],
     | 99.00th=[  203], 99.50th=[  211], 99.90th=[  222], 99.95th=[  226],
     | 99.99th=[  230]
   bw (  KiB/s): min=  376, max=  640, per=24.94%, avg=442.35, stdev=55.76, samples=46
   iops        : min=   94, max=  160, avg=110.52, stdev=13.96, samples=46
  lat (msec)   : 100=6.41%, 250=93.59%
  cpu          : usr=0.12%, sys=0.77%, ctx=2270, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1775KiB/s (1817kB/s), 444KiB/s-444KiB/s (454kB/s-455kB/s), io=40.0MiB (41.9MB), run=23064-23079msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23897: Fri May 16 10:00:59 2025
   read: IOPS=90, BW=362KiB/s (371kB/s)(5048KiB/13943msec)
    slat (usec): min=482, max=2740, avg=802.74, stdev=192.58
    clat (msec): min=9, max=154, avg=58.45, stdev=11.77
     lat (msec): min=10, max=155, avg=59.26, stdev=11.77
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   43], 10.00th=[   47], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   56], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   68], 90.00th=[   72], 95.00th=[   77],
     | 99.00th=[   87], 99.50th=[   96], 99.90th=[  144], 99.95th=[  155],
     | 99.99th=[  155]
   bw (  KiB/s): min=  240, max=  496, per=25.31%, avg=363.19, stdev=60.76, samples=27
   iops        : min=   60, max=  124, avg=90.74, stdev=15.15, samples=27
  write: IOPS=93, BW=372KiB/s (381kB/s)(5192KiB/13943msec)
    slat (usec): min=12, max=373, avg=51.95, stdev=29.47
    clat (msec): min=44, max=208, avg=113.96, stdev=25.29
     lat (msec): min=44, max=208, avg=114.01, stdev=25.29
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   75], 10.00th=[   83], 20.00th=[   91],
     | 30.00th=[   99], 40.00th=[  106], 50.00th=[  113], 60.00th=[  120],
     | 70.00th=[  128], 80.00th=[  138], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  178], 99.50th=[  184], 99.90th=[  197], 99.95th=[  209],
     | 99.99th=[  209]
   bw (  KiB/s): min=  312, max=  440, per=24.94%, avg=372.67, stdev=30.47, samples=27
   iops        : min=   78, max=  110, avg=93.11, stdev= 7.64, samples=27
  lat (msec)   : 10=0.04%, 20=0.12%, 50=10.66%, 100=54.77%, 250=34.41%
  cpu          : usr=0.19%, sys=1.76%, ctx=2967, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23898: Fri May 16 10:00:59 2025
   read: IOPS=89, BW=356KiB/s (365kB/s)(4976KiB/13976msec)
    slat (usec): min=117, max=2415, avg=801.25, stdev=178.47
    clat (usec): min=2491, max=96430, avg=58189.48, stdev=10993.07
     lat (usec): min=3432, max=97007, avg=58992.51, stdev=11001.29
    clat percentiles (usec):
     |  1.00th=[30802],  5.00th=[42206], 10.00th=[45876], 20.00th=[49546],
     | 30.00th=[52691], 40.00th=[55313], 50.00th=[58459], 60.00th=[61080],
     | 70.00th=[63177], 80.00th=[67634], 90.00th=[71828], 95.00th=[77071],
     | 99.00th=[83362], 99.50th=[85459], 99.90th=[90702], 99.95th=[95945],
     | 99.99th=[95945]
   bw (  KiB/s): min=  240, max=  480, per=24.69%, avg=354.30, stdev=60.80, samples=27
   iops        : min=   60, max=  120, avg=88.52, stdev=15.17, samples=27
  write: IOPS=94, BW=377KiB/s (386kB/s)(5264KiB/13976msec)
    slat (usec): min=13, max=287, avg=52.30, stdev=27.06
    clat (msec): min=35, max=192, avg=113.97, stdev=25.45
     lat (msec): min=35, max=192, avg=114.03, stdev=25.45
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   74], 10.00th=[   83], 20.00th=[   92],
     | 30.00th=[  100], 40.00th=[  107], 50.00th=[  113], 60.00th=[  121],
     | 70.00th=[  127], 80.00th=[  138], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  174], 99.50th=[  182], 99.90th=[  192], 99.95th=[  192],
     | 99.99th=[  192]
   bw (  KiB/s): min=  304, max=  424, per=25.16%, avg=375.93, stdev=36.91, samples=27
   iops        : min=   76, max=  106, avg=93.93, stdev= 9.27, samples=27
  lat (msec)   : 4=0.08%, 10=0.16%, 20=0.08%, 50=10.43%, 100=54.10%
  lat (msec)   : 250=35.16%
  cpu          : usr=0.27%, sys=1.65%, ctx=3174, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23899: Fri May 16 10:00:59 2025
   read: IOPS=89, BW=357KiB/s (365kB/s)(4984KiB/13968msec)
    slat (usec): min=475, max=2107, avg=799.00, stdev=169.38
    clat (usec): min=8308, max=94737, avg=58210.80, stdev=10647.90
     lat (usec): min=9049, max=95487, avg=59011.59, stdev=10648.52
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[42730], 10.00th=[46400], 20.00th=[49546],
     | 30.00th=[53216], 40.00th=[55837], 50.00th=[57934], 60.00th=[60556],
     | 70.00th=[62653], 80.00th=[67634], 90.00th=[71828], 95.00th=[74974],
     | 99.00th=[82314], 99.50th=[85459], 99.90th=[90702], 99.95th=[94897],
     | 99.99th=[94897]
   bw (  KiB/s): min=  272, max=  480, per=25.02%, avg=359.04, stdev=58.53, samples=27
   iops        : min=   68, max=  120, avg=89.70, stdev=14.65, samples=27
  write: IOPS=94, BW=376KiB/s (385kB/s)(5256KiB/13968msec)
    slat (usec): min=13, max=336, avg=51.94, stdev=27.11
    clat (msec): min=45, max=186, avg=113.89, stdev=24.99
     lat (msec): min=45, max=186, avg=113.95, stdev=24.99
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   74], 10.00th=[   84], 20.00th=[   92],
     | 30.00th=[   99], 40.00th=[  106], 50.00th=[  113], 60.00th=[  121],
     | 70.00th=[  128], 80.00th=[  138], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  174], 99.50th=[  178], 99.90th=[  184], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  312, max=  464, per=25.06%, avg=374.44, stdev=38.95, samples=27
   iops        : min=   78, max=  116, avg=93.56, stdev= 9.74, samples=27
  lat (msec)   : 10=0.20%, 20=0.16%, 50=9.84%, 100=54.80%, 250=35.00%
  cpu          : usr=0.21%, sys=1.75%, ctx=2911, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23900: Fri May 16 10:00:59 2025
   read: IOPS=90, BW=363KiB/s (372kB/s)(5060KiB/13932msec)
    slat (usec): min=95, max=3117, avg=800.74, stdev=196.63
    clat (usec): min=856, max=176947, avg=59414.98, stdev=12994.88
     lat (usec): min=1786, max=177701, avg=60217.55, stdev=13007.88
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   44], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   57], 50.00th=[   59], 60.00th=[   62],
     | 70.00th=[   65], 80.00th=[   69], 90.00th=[   73], 95.00th=[   78],
     | 99.00th=[   85], 99.50th=[  123], 99.90th=[  178], 99.95th=[  178],
     | 99.99th=[  178]
   bw (  KiB/s): min=  232, max=  456, per=25.49%, avg=365.85, stdev=50.98, samples=27
   iops        : min=   58, max=  114, avg=91.41, stdev=12.76, samples=27
  write: IOPS=92, BW=372KiB/s (381kB/s)(5180KiB/13932msec)
    slat (usec): min=13, max=932, avg=53.78, stdev=36.99
    clat (msec): min=14, max=235, avg=113.08, stdev=26.53
     lat (msec): min=15, max=235, avg=113.13, stdev=26.53
    clat percentiles (msec):
     |  1.00th=[   61],  5.00th=[   73], 10.00th=[   83], 20.00th=[   92],
     | 30.00th=[   99], 40.00th=[  105], 50.00th=[  112], 60.00th=[  120],
     | 70.00th=[  126], 80.00th=[  138], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  178], 99.50th=[  192], 99.90th=[  236], 99.95th=[  236],
     | 99.99th=[  236]
   bw (  KiB/s): min=  280, max=  432, per=24.90%, avg=372.07, stdev=38.18, samples=27
   iops        : min=   70, max=  108, avg=92.96, stdev= 9.54, samples=27
  lat (usec)   : 1000=0.04%
  lat (msec)   : 4=0.08%, 10=0.23%, 20=0.20%, 50=8.83%, 100=56.60%
  lat (msec)   : 250=34.02%
  cpu          : usr=0.24%, sys=1.68%, ctx=3220, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1436KiB/s (1470kB/s), 356KiB/s-363KiB/s (365kB/s-372kB/s), io=19.6MiB (20.5MB), run=13932-13976msec
  WRITE: bw=1495KiB/s (1531kB/s), 372KiB/s-377KiB/s (381kB/s-386kB/s), io=20.4MiB (21.4MB), run=13932-13976msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23914: Fri May 16 10:01:01 2025
   read: IOPS=12, BW=519MiB/s (545MB/s)(40.0MiB/77msec)
    slat (nsec): min=5980.4k, max=5980.4k, avg=5980423.00, stdev= 0.00
    clat (nsec): min=68067k, max=68067k, avg=68066800.00, stdev= 0.00
     lat (nsec): min=74056k, max=74056k, avg=74055844.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[67634],  5.00th=[67634], 10.00th=[67634], 20.00th=[67634],
     | 30.00th=[67634], 40.00th=[67634], 50.00th=[67634], 60.00th=[67634],
     | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634],
     | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[67634]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.89%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=40.0MiB (41.9MB), run=77-77msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck