== sanity test 398c: run fio to test AIO ================= 14:08:30 (1721412510)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.138333 s, 303 MB/s
osc.lustre-OST0000-osc-ffff88012c897800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=22333: Fri Jul 19 14:08:37 2024
  write: IOPS=477, BW=1911KiB/s (1957kB/s)(10.0MiB/5359msec)
    slat (usec): min=12, max=596, avg=81.72, stdev=48.52
    clat (usec): min=4292, max=60670, avg=33368.10, stdev=4915.64
     lat (usec): min=4404, max=60708, avg=33451.79, stdev=4920.98
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[22676], 10.00th=[27395], 20.00th=[30540],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[40633], 99.50th=[42730], 99.90th=[56886], 99.95th=[57410],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1720, max= 2360, per=25.06%, avg=1914.40, stdev=191.56, samples=10
   iops        : min=  430, max=  590, avg=478.60, stdev=47.89, samples=10
  lat (msec)   : 10=0.16%, 20=1.13%, 50=98.48%, 100=0.23%
  cpu          : usr=0.71%, sys=4.39%, ctx=1900, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22334: Fri Jul 19 14:08:37 2024
  write: IOPS=477, BW=1912KiB/s (1958kB/s)(10.0MiB/5356msec)
    slat (usec): min=13, max=474, avg=74.18, stdev=45.29
    clat (usec): min=10284, max=57405, avg=33349.71, stdev=4912.83
     lat (usec): min=10320, max=57522, avg=33425.53, stdev=4918.32
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[22938], 10.00th=[27132], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[40633], 99.50th=[41157], 99.90th=[57410], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min= 1720, max= 2392, per=25.07%, avg=1915.20, stdev=200.34, samples=10
   iops        : min=  430, max=  598, avg=478.80, stdev=50.09, samples=10
  lat (msec)   : 20=1.33%, 50=98.44%, 100=0.23%
  cpu          : usr=0.63%, sys=4.16%, ctx=1529, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22335: Fri Jul 19 14:08:37 2024
  write: IOPS=480, BW=1921KiB/s (1967kB/s)(10.0MiB/5331msec)
    slat (usec): min=13, max=556, avg=75.38, stdev=45.45
    clat (usec): min=3793, max=65861, avg=33207.07, stdev=5287.64
     lat (usec): min=3921, max=65897, avg=33284.29, stdev=5287.62
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[22414], 10.00th=[26870], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[40633], 99.50th=[43254], 99.90th=[62129], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1728, max= 2456, per=25.17%, avg=1922.80, stdev=215.14, samples=10
   iops        : min=  432, max=  614, avg=480.70, stdev=53.79, samples=10
  lat (msec)   : 4=0.08%, 10=0.55%, 20=1.25%, 50=97.85%, 100=0.27%
  cpu          : usr=0.83%, sys=4.13%, ctx=1478, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22336: Fri Jul 19 14:08:37 2024
  write: IOPS=477, BW=1910KiB/s (1956kB/s)(10.0MiB/5362msec)
    slat (usec): min=18, max=542, avg=76.91, stdev=47.40
    clat (usec): min=10242, max=63126, avg=33381.77, stdev=4905.34
     lat (usec): min=10271, max=63194, avg=33460.45, stdev=4911.59
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[22676], 10.00th=[27395], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[40633], 99.50th=[41681], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1728, max= 2368, per=25.05%, avg=1913.60, stdev=190.99, samples=10
   iops        : min=  432, max=  592, avg=478.40, stdev=47.75, samples=10
  lat (msec)   : 20=1.25%, 50=98.48%, 100=0.27%
  cpu          : usr=0.62%, sys=4.25%, ctx=1738, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7639KiB/s (7822kB/s), 1910KiB/s-1921KiB/s (1956kB/s-1967kB/s), io=40.0MiB (41.9MB), run=5331-5362msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=22350: Fri Jul 19 14:08:40 2024
   read: IOPS=557, BW=2230KiB/s (2283kB/s)(5048KiB/2264msec)
    slat (usec): min=511, max=2215, avg=918.06, stdev=251.39
    clat (usec): min=8333, max=29224, avg=13345.38, stdev=3170.18
     lat (usec): min=9123, max=31084, avg=14265.66, stdev=3311.52
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10028], 10.00th=[10552], 20.00th=[11076],
     | 30.00th=[11731], 40.00th=[12256], 50.00th=[12780], 60.00th=[13304],
     | 70.00th=[13698], 80.00th=[14484], 90.00th=[15926], 95.00th=[20317],
     | 99.00th=[25822], 99.50th=[27395], 99.90th=[28967], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1608, max= 2424, per=24.69%, avg=2188.00, stdev=388.06, samples=4
   iops        : min=  402, max=  606, avg=547.00, stdev=97.02, samples=4
  write: IOPS=573, BW=2293KiB/s (2348kB/s)(5192KiB/2264msec)
    slat (usec): min=13, max=849, avg=51.53, stdev=42.36
    clat (usec): min=3435, max=30260, avg=13724.74, stdev=2988.73
     lat (usec): min=3465, max=30292, avg=13777.58, stdev=2994.27
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10945], 10.00th=[11469], 20.00th=[11994],
     | 30.00th=[12387], 40.00th=[12649], 50.00th=[13042], 60.00th=[13435],
     | 70.00th=[13829], 80.00th=[14353], 90.00th=[15795], 95.00th=[21627],
     | 99.00th=[26084], 99.50th=[27395], 99.90th=[29754], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1736, max= 2552, per=24.82%, avg=2290.00, stdev=373.37, samples=4
   iops        : min=  434, max=  638, avg=572.50, stdev=93.34, samples=4
  lat (msec)   : 4=0.04%, 10=2.70%, 20=91.21%, 50=6.05%
  cpu          : usr=1.15%, sys=10.34%, ctx=2675, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22351: Fri Jul 19 14:08:40 2024
   read: IOPS=549, BW=2198KiB/s (2251kB/s)(4976KiB/2264msec)
    slat (usec): min=531, max=2577, avg=923.32, stdev=258.61
    clat (usec): min=3044, max=31268, avg=13360.87, stdev=3258.65
     lat (usec): min=3977, max=32334, avg=14286.76, stdev=3402.46
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10028], 10.00th=[10552], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[12256], 50.00th=[12780], 60.00th=[13304],
     | 70.00th=[13829], 80.00th=[14615], 90.00th=[16057], 95.00th=[20579],
     | 99.00th=[26346], 99.50th=[28967], 99.90th=[30016], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1600, max= 2360, per=24.35%, avg=2158.00, stdev=372.40, samples=4
   iops        : min=  400, max=  590, avg=539.50, stdev=93.10, samples=4
  write: IOPS=581, BW=2325KiB/s (2381kB/s)(5264KiB/2264msec)
    slat (usec): min=12, max=296, avg=52.78, stdev=35.45
    clat (usec): min=8860, max=31726, avg=13796.22, stdev=3051.46
     lat (usec): min=8883, max=31784, avg=13850.12, stdev=3055.30
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[10945], 10.00th=[11469], 20.00th=[11994],
     | 30.00th=[12518], 40.00th=[12780], 50.00th=[13173], 60.00th=[13435],
     | 70.00th=[13960], 80.00th=[14484], 90.00th=[15795], 95.00th=[21627],
     | 99.00th=[26084], 99.50th=[29754], 99.90th=[31589], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 1808, max= 2528, per=25.21%, avg=2326.00, stdev=346.71, samples=4
   iops        : min=  452, max=  632, avg=581.50, stdev=86.68, samples=4
  lat (msec)   : 4=0.08%, 10=2.70%, 20=91.29%, 50=5.94%
  cpu          : usr=1.24%, sys=10.30%, ctx=2693, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22352: Fri Jul 19 14:08:40 2024
   read: IOPS=553, BW=2212KiB/s (2265kB/s)(4984KiB/2253msec)
    slat (usec): min=22, max=2318, avg=917.26, stdev=262.52
    clat (usec): min=950, max=30494, avg=13314.45, stdev=3192.22
     lat (usec): min=1921, max=32132, avg=14233.86, stdev=3335.37
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10028], 10.00th=[10552], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[12256], 50.00th=[12780], 60.00th=[13304],
     | 70.00th=[13829], 80.00th=[14484], 90.00th=[15795], 95.00th=[20055],
     | 99.00th=[25560], 99.50th=[27919], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1744, max= 2528, per=24.81%, avg=2198.75, stdev=329.05, samples=4
   iops        : min=  436, max=  632, avg=549.50, stdev=82.19, samples=4
  write: IOPS=583, BW=2333KiB/s (2389kB/s)(5256KiB/2253msec)
    slat (usec): min=13, max=1004, avg=52.85, stdev=47.23
    clat (usec): min=7129, max=28230, avg=13796.32, stdev=3109.80
     lat (usec): min=7165, max=28285, avg=13850.29, stdev=3112.35
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[10945], 10.00th=[11338], 20.00th=[11994],
     | 30.00th=[12387], 40.00th=[12780], 50.00th=[13042], 60.00th=[13304],
     | 70.00th=[13829], 80.00th=[14484], 90.00th=[16188], 95.00th=[22152],
     | 99.00th=[26608], 99.50th=[27395], 99.90th=[27657], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1776, max= 2600, per=25.11%, avg=2316.50, stdev=375.43, samples=4
   iops        : min=  444, max=  650, avg=579.00, stdev=93.76, samples=4
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.08%, 4=0.04%, 10=2.66%, 20=91.37%, 50=5.82%
  cpu          : usr=1.11%, sys=10.12%, ctx=2583, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22353: Fri Jul 19 14:08:40 2024
   read: IOPS=558, BW=2235KiB/s (2289kB/s)(5060KiB/2264msec)
    slat (usec): min=509, max=2640, avg=918.53, stdev=252.47
    clat (usec): min=2528, max=29713, avg=13321.05, stdev=3103.67
     lat (usec): min=3643, max=30910, avg=14241.71, stdev=3248.69
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10028], 10.00th=[10552], 20.00th=[11207],
     | 30.00th=[11731], 40.00th=[12256], 50.00th=[12780], 60.00th=[13173],
     | 70.00th=[13829], 80.00th=[14484], 90.00th=[15926], 95.00th=[19792],
     | 99.00th=[26608], 99.50th=[28705], 99.90th=[29230], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1672, max= 2416, per=24.96%, avg=2212.00, stdev=360.77, samples=4
   iops        : min=  418, max=  604, avg=553.00, stdev=90.19, samples=4
  write: IOPS=571, BW=2288KiB/s (2343kB/s)(5180KiB/2264msec)
    slat (usec): min=12, max=570, avg=50.60, stdev=37.49
    clat (usec): min=4824, max=28899, avg=13774.08, stdev=3154.12
     lat (usec): min=4926, max=28965, avg=13826.02, stdev=3156.98
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[10945], 10.00th=[11338], 20.00th=[11863],
     | 30.00th=[12387], 40.00th=[12649], 50.00th=[13042], 60.00th=[13304],
     | 70.00th=[13829], 80.00th=[14484], 90.00th=[16188], 95.00th=[21890],
     | 99.00th=[26870], 99.50th=[27395], 99.90th=[28181], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1720, max= 2592, per=24.58%, avg=2268.00, stdev=378.88, samples=4
   iops        : min=  430, max=  648, avg=567.00, stdev=94.72, samples=4
  lat (msec)   : 4=0.04%, 10=2.85%, 20=91.37%, 50=5.74%
  cpu          : usr=1.63%, sys=9.68%, ctx=2676, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=8864KiB/s (9077kB/s), 2198KiB/s-2235KiB/s (2251kB/s-2289kB/s), io=19.6MiB (20.5MB), run=2253-2264msec
  WRITE: bw=9228KiB/s (9449kB/s), 2288KiB/s-2333KiB/s (2343kB/s-2389kB/s), io=20.4MiB (21.4MB), run=2253-2264msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=22359: Fri Jul 19 14:08:42 2024
   read: IOPS=8, BW=336MiB/s (352MB/s)(40.0MiB/119msec)
    slat (nsec): min=6711.2k, max=6711.2k, avg=6711202.00, stdev= 0.00
    clat (nsec): min=107398k, max=107398k, avg=107398077.00, stdev= 0.00
     lat (nsec): min=114119k, max=114119k, avg=114119011.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  108], 10.00th=[  108], 20.00th=[  108],
     | 30.00th=[  108], 40.00th=[  108], 50.00th=[  108], 60.00th=[  108],
     | 70.00th=[  108], 80.00th=[  108], 90.00th=[  108], 95.00th=[  108],
     | 99.00th=[  108], 99.50th=[  108], 99.90th=[  108], 99.95th=[  108],
     | 99.99th=[  108]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.93%, ctx=7, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=336MiB/s (352MB/s), 336MiB/s-336MiB/s (352MB/s-352MB/s), io=40.0MiB (41.9MB), run=119-119msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck