== sanity test 398c: run fio to test AIO ================= 09:07:21 (1786194441)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.15469 s, 271 MB/s
osc.lustre-OST0000-osc-ffff8f84f9215000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=181079: Sat Aug  8 09:07:31 2026
  write: IOPS=434, BW=1737KiB/s (1778kB/s)(10.0MiB/5896msec); 0 zone resets
    slat (usec): min=37, max=27292, avg=186.99, stdev=544.91
    clat (usec): min=11148, max=62713, avg=36362.26, stdev=7383.71
     lat (usec): min=11246, max=62773, avg=36550.44, stdev=7382.39
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[25822], 10.00th=[28181], 20.00th=[31065],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[49021],
     | 99.00th=[59507], 99.50th=[61080], 99.90th=[62129], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1328, max= 2166, per=24.74%, avg=1718.36, stdev=242.17, samples=11
   iops        : min=  332, max=  541, avg=429.55, stdev=60.45, samples=11
  lat (msec)   : 20=0.90%, 50=94.57%, 100=4.53%
  cpu          : usr=0.46%, sys=6.34%, ctx=2571, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181080: Sat Aug  8 09:07:31 2026
  write: IOPS=434, BW=1737KiB/s (1778kB/s)(10.0MiB/5896msec); 0 zone resets
    slat (usec): min=36, max=972, avg=183.74, stdev=103.16
    clat (usec): min=9628, max=61850, avg=36307.53, stdev=7148.61
     lat (usec): min=9858, max=62062, avg=36492.42, stdev=7161.66
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[25560], 10.00th=[27919], 20.00th=[31065],
     | 30.00th=[33162], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[49546],
     | 99.00th=[57934], 99.50th=[58983], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1368, max= 2073, per=24.68%, avg=1714.27, stdev=224.31, samples=11
   iops        : min=  342, max=  518, avg=428.55, stdev=56.04, samples=11
  lat (msec)   : 10=0.04%, 20=1.33%, 50=94.57%, 100=4.06%
  cpu          : usr=0.53%, sys=6.07%, ctx=2634, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181081: Sat Aug  8 09:07:31 2026
  write: IOPS=438, BW=1752KiB/s (1794kB/s)(10.0MiB/5844msec); 0 zone resets
    slat (usec): min=34, max=953, avg=173.21, stdev=97.03
    clat (usec): min=3937, max=61692, avg=36301.25, stdev=7378.48
     lat (usec): min=4083, max=61788, avg=36475.48, stdev=7379.51
    clat percentiles (usec):
     |  1.00th=[17695],  5.00th=[25560], 10.00th=[27919], 20.00th=[30802],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[58459], 99.50th=[60556], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1368, max= 2288, per=24.92%, avg=1730.91, stdev=262.65, samples=11
   iops        : min=  342, max=  572, avg=432.73, stdev=65.66, samples=11
  lat (msec)   : 4=0.04%, 10=0.04%, 20=1.21%, 50=94.73%, 100=3.98%
  cpu          : usr=0.55%, sys=5.77%, ctx=2632, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181082: Sat Aug  8 09:07:31 2026
  write: IOPS=434, BW=1738KiB/s (1779kB/s)(10.0MiB/5893msec); 0 zone resets
    slat (usec): min=36, max=1252, avg=189.80, stdev=111.31
    clat (usec): min=5940, max=61845, avg=36303.64, stdev=7198.54
     lat (usec): min=6025, max=62123, avg=36494.61, stdev=7207.12
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[25297], 10.00th=[27919], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[49546],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1400, max= 2108, per=24.72%, avg=1717.45, stdev=228.37, samples=11
   iops        : min=  350, max=  527, avg=429.36, stdev=57.09, samples=11
  lat (msec)   : 10=0.08%, 20=0.98%, 50=94.69%, 100=4.26%
  cpu          : usr=0.59%, sys=6.62%, ctx=2697, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6947KiB/s (7114kB/s), 1737KiB/s-1752KiB/s (1778kB/s-1794kB/s), io=40.0MiB (41.9MB), run=5844-5896msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=181093: Sat Aug  8 09:07:41 2026
  read: IOPS=177, BW=711KiB/s (728kB/s)(5048KiB/7100msec)
    slat (usec): min=1127, max=8449, avg=2958.72, stdev=988.20
    clat (usec): min=8491, max=69160, avg=42159.63, stdev=7697.55
     lat (usec): min=11648, max=74522, avg=45120.62, stdev=7944.13
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[30802], 10.00th=[33162], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[48497], 90.00th=[52167], 95.00th=[54264],
     | 99.00th=[62653], 99.50th=[65799], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  560, max=  808, per=24.68%, avg=697.54, stdev=72.80, samples=13
   iops        : min=  140, max=  202, avg=174.31, stdev=18.21, samples=13
  write: IOPS=182, BW=731KiB/s (749kB/s)(5192KiB/7100msec); 0 zone resets
    slat (usec): min=53, max=902, avg=173.58, stdev=101.26
    clat (usec): min=4429, max=66384, avg=42550.50, stdev=6763.73
     lat (usec): min=4633, max=66510, avg=42725.42, stdev=6771.88
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[32900], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[42206], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[47973], 90.00th=[51643], 95.00th=[54264],
     | 99.00th=[60556], 99.50th=[61080], 99.90th=[65274], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  584, max=  896, per=24.60%, avg=723.85, stdev=86.03, samples=13
   iops        : min=  146, max=  224, avg=180.92, stdev=21.54, samples=13
  lat (msec)   : 10=0.12%, 20=0.23%, 50=84.96%, 100=14.69%
  cpu          : usr=0.38%, sys=5.72%, ctx=4817, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181094: Sat Aug  8 09:07:41 2026
  read: IOPS=175, BW=703KiB/s (719kB/s)(4976KiB/7082msec)
    slat (usec): min=35, max=8646, avg=2947.74, stdev=1003.05
    clat (usec): min=21799, max=65888, avg=42259.28, stdev=7280.09
     lat (usec): min=26066, max=71518, avg=45209.47, stdev=7524.65
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[31851], 10.00th=[33817], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[47973], 90.00th=[51643], 95.00th=[55313],
     | 99.00th=[62653], 99.50th=[63701], 99.90th=[64750], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  624, max=  808, per=24.35%, avg=688.08, stdev=61.33, samples=13
   iops        : min=  156, max=  202, avg=171.92, stdev=15.40, samples=13
  write: IOPS=185, BW=743KiB/s (761kB/s)(5264KiB/7082msec); 0 zone resets
    slat (usec): min=34, max=1600, avg=175.87, stdev=118.55
    clat (usec): min=13490, max=66262, avg=43028.58, stdev=6758.34
     lat (usec): min=13596, max=66526, avg=43205.58, stdev=6763.16
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[33162], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[41157], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[46400], 80.00th=[48497], 90.00th=[52167], 95.00th=[54789],
     | 99.00th=[60031], 99.50th=[61604], 99.90th=[64226], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  576, max=  920, per=25.22%, avg=742.08, stdev=99.23, samples=13
   iops        : min=  144, max=  230, avg=185.46, stdev=24.82, samples=13
  lat (msec)   : 20=0.20%, 50=84.49%, 100=15.31%
  cpu          : usr=0.52%, sys=5.62%, ctx=4869, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181095: Sat Aug  8 09:07:41 2026
  read: IOPS=175, BW=704KiB/s (721kB/s)(4984KiB/7082msec)
    slat (usec): min=61, max=9466, avg=2966.24, stdev=1018.69
    clat (usec): min=3482, max=67400, avg=42214.57, stdev=7511.29
     lat (usec): min=5733, max=71748, avg=45183.43, stdev=7721.48
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30802], 10.00th=[32900], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[42206], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[47973], 90.00th=[52167], 95.00th=[55313],
     | 99.00th=[60556], 99.50th=[63701], 99.90th=[66323], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  608, max=  904, per=24.73%, avg=698.85, stdev=95.04, samples=13
   iops        : min=  152, max=  226, avg=174.62, stdev=23.79, samples=13
  write: IOPS=185, BW=742KiB/s (760kB/s)(5256KiB/7082msec); 0 zone resets
    slat (usec): min=39, max=1322, avg=177.63, stdev=115.73
    clat (usec): min=10004, max=68384, avg=42849.61, stdev=6923.79
     lat (usec): min=10140, max=68510, avg=43029.13, stdev=6937.52
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[32637], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[39060], 40.00th=[40633], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45876], 80.00th=[48497], 90.00th=[51643], 95.00th=[54789],
     | 99.00th=[60556], 99.50th=[62653], 99.90th=[65799], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  608, max=  856, per=24.79%, avg=729.38, stdev=81.32, samples=13
   iops        : min=  152, max=  214, avg=182.31, stdev=20.38, samples=13
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.12%, 50=85.39%, 100=14.41%
  cpu          : usr=0.42%, sys=5.76%, ctx=4701, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181096: Sat Aug  8 09:07:41 2026
  read: IOPS=178, BW=713KiB/s (730kB/s)(5060KiB/7097msec)
    slat (usec): min=1162, max=9575, avg=2920.38, stdev=965.12
    clat (usec): min=6262, max=70851, avg=42313.22, stdev=7425.19
     lat (usec): min=9775, max=75672, avg=45236.11, stdev=7670.54
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[31589], 10.00th=[33424], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[48497], 90.00th=[52167], 95.00th=[55313],
     | 99.00th=[61604], 99.50th=[64750], 99.90th=[68682], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  617, max=  848, per=24.89%, avg=703.31, stdev=73.04, samples=13
   iops        : min=  154, max=  212, avg=175.77, stdev=18.27, samples=13
  write: IOPS=182, BW=730KiB/s (747kB/s)(5180KiB/7097msec); 0 zone resets
    slat (usec): min=57, max=2510, avg=177.73, stdev=124.02
    clat (usec): min=18295, max=67677, avg=42658.90, stdev=6545.17
     lat (usec): min=18379, max=67832, avg=42837.86, stdev=6546.00
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[32375], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[39060], 40.00th=[40633], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45876], 80.00th=[47973], 90.00th=[51119], 95.00th=[53740],
     | 99.00th=[59507], 99.50th=[62653], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  560, max=  920, per=24.47%, avg=719.92, stdev=92.77, samples=13
   iops        : min=  140, max=  230, avg=179.92, stdev=23.23, samples=13
  lat (msec)   : 10=0.04%, 20=0.20%, 50=85.47%, 100=14.30%
  cpu          : usr=0.56%, sys=5.51%, ctx=4853, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2826KiB/s (2894kB/s), 703KiB/s-713KiB/s (719kB/s-730kB/s), io=19.6MiB (20.5MB), run=7082-7100msec
  WRITE: bw=2943KiB/s (3013kB/s), 730KiB/s-743KiB/s (747kB/s-761kB/s), io=20.4MiB (21.4MB), run=7082-7100msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=181100: Sat Aug  8 09:07:45 2026
  read: IOPS=8, BW=348MiB/s (365MB/s)(40.0MiB/115msec)
    slat (nsec): min=9873.8k, max=9873.8k, avg=9873794.00, stdev= 0.00
    clat (nsec): min=99369k, max=99369k, avg=99369429.00, stdev= 0.00
     lat (nsec): min=109248k, max=109248k, avg=109247568.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[99091],  5.00th=[99091], 10.00th=[99091], 20.00th=[99091],
     | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091],
     | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091],
     | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091],
     | 99.99th=[99091]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.89%, ctx=9, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=40.0MiB (41.9MB), run=115-115msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout