== sanity test 398c: run fio to test AIO ================= 23:03:10 (1775876590)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.151286 s, 277 MB/s
osc.lustre-OST0000-osc-ffff8e02e076d000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183231: Fri Apr 10 23:03:22 2026
  write: IOPS=313, BW=1256KiB/s (1286kB/s)(10.0MiB/8153msec); 0 zone resets
    slat (usec): min=38, max=4337, avg=195.27, stdev=160.49
    clat (msec): min=5, max=101, avg=50.28, stdev=11.55
     lat (msec): min=5, max=102, avg=50.48, stdev=11.55
    clat percentiles (msec):
     |  1.00th=[   34],  5.00th=[   36], 10.00th=[   39], 20.00th=[   42],
     | 30.00th=[   44], 40.00th=[   46], 50.00th=[   48], 60.00th=[   51],
     | 70.00th=[   54], 80.00th=[   58], 90.00th=[   65], 95.00th=[   72],
     | 99.00th=[   92], 99.50th=[   97], 99.90th=[  102], 99.95th=[  102],
     | 99.99th=[  103]
   bw (  KiB/s): min=  960, max= 1528, per=24.96%, avg=1253.81, stdev=192.96, samples=16
   iops        : min=  240, max=  382, avg=313.44, stdev=48.23, samples=16
  lat (msec)   : 10=0.08%, 20=0.20%, 50=55.82%, 100=43.63%, 250=0.27%
  cpu          : usr=0.53%, sys=4.43%, ctx=2695, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183232: Fri Apr 10 23:03:22 2026
  write: IOPS=314, BW=1257KiB/s (1287kB/s)(10.0MiB/8147msec); 0 zone resets
    slat (usec): min=34, max=3059, avg=189.96, stdev=155.43
    clat (usec): min=29623, max=98569, avg=50279.59, stdev=11656.80
     lat (usec): min=29879, max=99682, avg=50470.81, stdev=11669.27
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[36439], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[43779], 40.00th=[45876], 50.00th=[48497], 60.00th=[50070],
     | 70.00th=[53216], 80.00th=[57410], 90.00th=[64226], 95.00th=[72877],
     | 99.00th=[92799], 99.50th=[96994], 99.90th=[98042], 99.95th=[98042],
     | 99.99th=[99091]
   bw (  KiB/s): min=  896, max= 1536, per=25.02%, avg=1256.81, stdev=197.30, samples=16
   iops        : min=  224, max=  384, avg=314.19, stdev=49.31, samples=16
  lat (msec)   : 50=58.28%, 100=41.72%
  cpu          : usr=0.49%, sys=4.76%, ctx=2263, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183233: Fri Apr 10 23:03:22 2026
  write: IOPS=315, BW=1260KiB/s (1291kB/s)(10.0MiB/8124msec); 0 zone resets
    slat (usec): min=44, max=31556, avg=223.07, stdev=641.74
    clat (msec): min=9, max=102, avg=50.17, stdev=11.99
     lat (msec): min=9, max=102, avg=50.40, stdev=11.98
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   37], 10.00th=[   39], 20.00th=[   42],
     | 30.00th=[   44], 40.00th=[   46], 50.00th=[   48], 60.00th=[   51],
     | 70.00th=[   54], 80.00th=[   58], 90.00th=[   66], 95.00th=[   74],
     | 99.00th=[   92], 99.50th=[   97], 99.90th=[  100], 99.95th=[  100],
     | 99.99th=[  103]
   bw (  KiB/s): min=  896, max= 1528, per=25.07%, avg=1259.20, stdev=198.79, samples=15
   iops        : min=  224, max=  382, avg=314.73, stdev=49.73, samples=15
  lat (msec)   : 10=0.04%, 20=0.27%, 50=57.42%, 100=42.23%, 250=0.04%
  cpu          : usr=0.44%, sys=5.53%, ctx=2544, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183234: Fri Apr 10 23:03:22 2026
  write: IOPS=314, BW=1259KiB/s (1289kB/s)(10.0MiB/8132msec); 0 zone resets
    slat (usec): min=34, max=2116, avg=195.68, stdev=137.54
    clat (msec): min=29, max=105, avg=50.55, stdev=11.79
     lat (msec): min=29, max=105, avg=50.75, stdev=11.80
    clat percentiles (msec):
     |  1.00th=[   34],  5.00th=[   37], 10.00th=[   39], 20.00th=[   42],
     | 30.00th=[   44], 40.00th=[   46], 50.00th=[   49], 60.00th=[   52],
     | 70.00th=[   54], 80.00th=[   58], 90.00th=[   66], 95.00th=[   73],
     | 99.00th=[   95], 99.50th=[   99], 99.90th=[  105], 99.95th=[  105],
     | 99.99th=[  106]
   bw (  KiB/s): min=  944, max= 1536, per=25.00%, avg=1255.81, stdev=190.15, samples=16
   iops        : min=  236, max=  384, avg=313.94, stdev=47.53, samples=16
  lat (msec)   : 50=56.56%, 100=43.16%, 250=0.27%
  cpu          : usr=0.32%, sys=4.80%, ctx=2551, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5024KiB/s (5144kB/s), 1256KiB/s-1260KiB/s (1286kB/s-1291kB/s), io=40.0MiB (41.9MB), run=8124-8153msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183244: Fri Apr 10 23:03:32 2026
  read: IOPS=191, BW=766KiB/s (785kB/s)(5048KiB/6586msec)
    slat (usec): min=1257, max=7703, avg=2756.85, stdev=844.99
    clat (usec): min=6678, max=61110, avg=39010.96, stdev=6505.46
     lat (usec): min=8903, max=64507, avg=41770.58, stdev=6620.29
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[29492], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[38536], 60.00th=[40109],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[50070],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[60031], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  696, max=  848, per=24.82%, avg=756.42, stdev=43.47, samples=12
   iops        : min=  174, max=  212, avg=189.08, stdev=10.87, samples=12
  write: IOPS=197, BW=788KiB/s (807kB/s)(5192KiB/6586msec); 0 zone resets
    slat (usec): min=56, max=3358, avg=184.37, stdev=154.05
    clat (usec): min=8932, max=61554, avg=39753.80, stdev=5542.75
     lat (usec): min=9108, max=61827, avg=39939.43, stdev=5545.12
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[32375], 10.00th=[33817], 20.00th=[35390],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42206], 80.00th=[43779], 90.00th=[46924], 95.00th=[50070],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  664, max=  848, per=24.80%, avg=786.75, stdev=52.35, samples=12
   iops        : min=  166, max=  212, avg=196.67, stdev=13.11, samples=12
  lat (msec)   : 10=0.20%, 20=0.20%, 50=94.77%, 100=4.84%
  cpu          : usr=0.53%, sys=6.41%, ctx=4380, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183245: Fri Apr 10 23:03:32 2026
  read: IOPS=191, BW=765KiB/s (783kB/s)(4976KiB/6507msec)
    slat (usec): min=35, max=10484, avg=2736.02, stdev=845.99
    clat (usec): min=5805, max=71492, avg=39090.55, stdev=6763.87
     lat (usec): min=5854, max=75624, avg=41828.99, stdev=6921.50
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[29754], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[40109],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[50594],
     | 99.00th=[57934], 99.50th=[62653], 99.90th=[69731], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  704, max=  816, per=24.96%, avg=760.42, stdev=34.21, samples=12
   iops        : min=  176, max=  204, avg=190.08, stdev= 8.55, samples=12
  write: IOPS=202, BW=809KiB/s (828kB/s)(5264KiB/6507msec); 0 zone resets
    slat (usec): min=34, max=1113, avg=179.90, stdev=97.20
    clat (usec): min=15635, max=61570, avg=39258.34, stdev=5398.36
     lat (usec): min=15746, max=61682, avg=39439.44, stdev=5396.55
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[31851], 10.00th=[33162], 20.00th=[35390],
     | 30.00th=[36439], 40.00th=[37487], 50.00th=[38536], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[43254], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[60556], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  720, max=  932, per=25.69%, avg=815.00, stdev=62.72, samples=12
   iops        : min=  180, max=  233, avg=203.75, stdev=15.68, samples=12
  lat (msec)   : 10=0.23%, 20=0.16%, 50=94.88%, 100=4.73%
  cpu          : usr=0.63%, sys=6.13%, ctx=4402, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183246: Fri Apr 10 23:03:32 2026
  read: IOPS=189, BW=758KiB/s (776kB/s)(4984KiB/6573msec)
    slat (usec): min=1307, max=7118, avg=2776.83, stdev=816.42
    clat (usec): min=6442, max=70853, avg=39384.16, stdev=6372.66
     lat (usec): min=8651, max=75813, avg=42164.51, stdev=6480.08
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29492], 10.00th=[31589], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42206], 80.00th=[43779], 90.00th=[47449], 95.00th=[50070],
     | 99.00th=[56886], 99.50th=[59507], 99.90th=[63177], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  632, max=  904, per=24.94%, avg=759.83, stdev=70.76, samples=12
   iops        : min=  158, max=  226, avg=189.92, stdev=17.70, samples=12
  write: IOPS=199, BW=800KiB/s (819kB/s)(5256KiB/6573msec); 0 zone resets
    slat (usec): min=42, max=1531, avg=177.14, stdev=100.27
    clat (usec): min=14138, max=65209, avg=39376.47, stdev=5687.25
     lat (usec): min=14307, max=65332, avg=39554.74, stdev=5686.79
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[31589], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[38536], 60.00th=[40109],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[49546],
     | 99.00th=[55313], 99.50th=[59507], 99.90th=[62129], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  656, max=  880, per=25.04%, avg=794.17, stdev=61.05, samples=12
   iops        : min=  164, max=  220, avg=198.50, stdev=15.26, samples=12
  lat (msec)   : 10=0.04%, 20=0.08%, 50=94.88%, 100=5.00%
  cpu          : usr=0.56%, sys=6.16%, ctx=4390, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183247: Fri Apr 10 23:03:32 2026
  read: IOPS=192, BW=769KiB/s (787kB/s)(5060KiB/6583msec)
    slat (usec): min=1289, max=7711, avg=2771.88, stdev=832.77
    clat (usec): min=8940, max=60683, avg=39188.59, stdev=6277.03
     lat (usec): min=11607, max=65940, avg=41963.12, stdev=6380.18
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[29754], 10.00th=[31589], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[40109],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[47449], 95.00th=[50070],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[60556], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  696, max=  880, per=25.21%, avg=768.25, stdev=50.62, samples=12
   iops        : min=  174, max=  220, avg=192.00, stdev=12.71, samples=12
  write: IOPS=196, BW=787KiB/s (806kB/s)(5180KiB/6583msec); 0 zone resets
    slat (usec): min=48, max=940, avg=178.52, stdev=93.25
    clat (usec): min=26428, max=69784, avg=39515.68, stdev=5368.30
     lat (usec): min=26590, max=69948, avg=39695.35, stdev=5368.52
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[31589], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42206], 80.00th=[43779], 90.00th=[46400], 95.00th=[48497],
     | 99.00th=[54264], 99.50th=[55837], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  696, max=  920, per=24.78%, avg=785.92, stdev=61.42, samples=12
   iops        : min=  174, max=  230, avg=196.42, stdev=15.36, samples=12
  lat (msec)   : 10=0.04%, 50=95.62%, 100=4.34%
  cpu          : usr=0.76%, sys=5.99%, ctx=4438, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3047KiB/s (3120kB/s), 758KiB/s-769KiB/s (776kB/s-787kB/s), io=19.6MiB (20.5MB), run=6507-6586msec
  WRITE: bw=3172KiB/s (3248kB/s), 787KiB/s-809KiB/s (806kB/s-828kB/s), io=20.4MiB (21.4MB), run=6507-6586msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183252: Fri Apr 10 23:03:36 2026
  read: IOPS=6, BW=256MiB/s (269MB/s)(40.0MiB/156msec)
    slat (nsec): min=11741k, max=11741k, avg=11740890.00, stdev= 0.00
    clat (nsec): min=136038k, max=136038k, avg=136038051.00, stdev= 0.00
     lat (nsec): min=147783k, max=147783k, avg=147783196.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  136],  5.00th=[  136], 10.00th=[  136], 20.00th=[  136],
     | 30.00th=[  136], 40.00th=[  136], 50.00th=[  136], 60.00th=[  136],
     | 70.00th=[  136], 80.00th=[  136], 90.00th=[  136], 95.00th=[  136],
     | 99.00th=[  136], 99.50th=[  136], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=6.45%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=40.0MiB (41.9MB), run=156-156msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout