== sanity test 398c: run fio to test AIO ================= 00:48:12 (1722487692)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.146884 s, 286 MB/s
osc.lustre-OST0000-osc-ffff8800a9062000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18472: Thu Aug  1 00:48:19 2024
  write: IOPS=463, BW=1855KiB/s (1900kB/s)(10.0MiB/5519msec)
    slat (usec): min=25, max=705, avg=72.47, stdev=42.64
    clat (usec): min=4482, max=58923, avg=34371.54, stdev=4220.41
     lat (usec): min=4590, max=58997, avg=34445.78, stdev=4223.48
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27657], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[42730], 99.50th=[54264], 99.90th=[54789], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1656, max= 1956, per=24.93%, avg=1849.82, stdev=103.51, samples=11
   iops        : min=  414, max=  489, avg=462.45, stdev=25.88, samples=11
  lat (msec)   : 10=0.04%, 50=99.10%, 100=0.86%
  cpu          : usr=0.65%, sys=4.11%, ctx=1413, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18473: Thu Aug  1 00:48:19 2024
  write: IOPS=463, BW=1855KiB/s (1900kB/s)(10.0MiB/5519msec)
    slat (usec): min=24, max=434, avg=76.14, stdev=43.49
    clat (usec): min=3803, max=59065, avg=34351.75, stdev=4104.82
     lat (usec): min=4035, max=59152, avg=34429.95, stdev=4109.24
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27919], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[52167], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1656, max= 1972, per=24.93%, avg=1849.82, stdev=101.42, samples=11
   iops        : min=  414, max=  493, avg=462.45, stdev=25.35, samples=11
  lat (msec)   : 4=0.04%, 50=99.18%, 100=0.78%
  cpu          : usr=0.63%, sys=4.10%, ctx=1601, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18474: Thu Aug  1 00:48:19 2024
  write: IOPS=466, BW=1865KiB/s (1909kB/s)(10.0MiB/5492msec)
    slat (usec): min=19, max=358, avg=80.54, stdev=42.32
    clat (usec): min=3438, max=66875, avg=34202.35, stdev=4451.29
     lat (usec): min=3561, max=66908, avg=34284.84, stdev=4449.95
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26870], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[51643], 99.90th=[66847], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min= 1676, max= 2027, per=25.35%, avg=1881.10, stdev=100.66, samples=10
   iops        : min=  419, max=  506, avg=470.20, stdev=25.05, samples=10
  lat (msec)   : 4=0.04%, 10=0.23%, 20=0.08%, 50=99.14%, 100=0.51%
  cpu          : usr=0.87%, sys=4.21%, ctx=1710, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18475: Thu Aug  1 00:48:19 2024
  write: IOPS=464, BW=1857KiB/s (1901kB/s)(10.0MiB/5515msec)
    slat (usec): min=25, max=34510, avg=95.07, stdev=682.21
    clat (usec): min=4244, max=54872, avg=34239.92, stdev=4168.72
     lat (usec): min=4426, max=54938, avg=34336.99, stdev=4174.48
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27395], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[39060], 95.00th=[39584],
     | 99.00th=[41681], 99.50th=[51643], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1632, max= 1972, per=24.93%, avg=1850.18, stdev=108.06, samples=11
   iops        : min=  408, max=  493, avg=462.55, stdev=27.02, samples=11
  lat (msec)   : 10=0.12%, 20=0.16%, 50=99.06%, 100=0.66%
  cpu          : usr=0.49%, sys=4.75%, ctx=1710, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7422KiB/s (7600kB/s), 1855KiB/s-1865KiB/s (1900kB/s-1909kB/s), io=40.0MiB (41.9MB), run=5492-5519msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18489: Thu Aug  1 00:48:21 2024
   read: IOPS=682, BW=2732KiB/s (2797kB/s)(5048KiB/1848msec)
    slat (usec): min=403, max=2245, avg=694.63, stdev=242.14
    clat (usec): min=2253, max=30078, avg=10840.57, stdev=3153.82
     lat (usec): min=2918, max=31955, avg=11536.82, stdev=3331.82
    clat percentiles (usec):
     |  1.00th=[ 6849],  5.00th=[ 8029], 10.00th=[ 8455], 20.00th=[ 8979],
     | 30.00th=[ 9372], 40.00th=[ 9896], 50.00th=[10290], 60.00th=[10683],
     | 70.00th=[11076], 80.00th=[11600], 90.00th=[12911], 95.00th=[16909],
     | 99.00th=[25822], 99.50th=[27919], 99.90th=[29492], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1936, max= 2992, per=24.23%, avg=2626.67, stdev=598.47, samples=3
   iops        : min=  484, max=  748, avg=656.67, stdev=149.62, samples=3
  write: IOPS=702, BW=2810KiB/s (2877kB/s)(5192KiB/1848msec)
    slat (usec): min=13, max=293, avg=39.23, stdev=23.38
    clat (usec): min=6819, max=29462, avg=11273.36, stdev=3145.69
     lat (usec): min=6894, max=29646, avg=11313.56, stdev=3154.87
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9634],
     | 30.00th=[10028], 40.00th=[10421], 50.00th=[10683], 60.00th=[10945],
     | 70.00th=[11207], 80.00th=[11600], 90.00th=[12649], 95.00th=[17957],
     | 99.00th=[26870], 99.50th=[28181], 99.90th=[28443], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 2184, max= 3080, per=24.53%, avg=2768.00, stdev=506.15, samples=3
   iops        : min=  546, max=  770, avg=692.00, stdev=126.54, samples=3
  lat (msec)   : 4=0.16%, 10=35.59%, 20=60.43%, 50=3.83%
  cpu          : usr=0.76%, sys=9.85%, ctx=2605, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18490: Thu Aug  1 00:48:21 2024
   read: IOPS=674, BW=2700KiB/s (2765kB/s)(4976KiB/1843msec)
    slat (usec): min=33, max=2041, avg=686.98, stdev=242.79
    clat (usec): min=1796, max=29589, avg=10942.56, stdev=3309.52
     lat (usec): min=1830, max=31095, avg=11631.24, stdev=3496.92
    clat percentiles (usec):
     |  1.00th=[ 7111],  5.00th=[ 8094], 10.00th=[ 8586], 20.00th=[ 9110],
     | 30.00th=[ 9503], 40.00th=[ 9896], 50.00th=[10290], 60.00th=[10683],
     | 70.00th=[11076], 80.00th=[11469], 90.00th=[12649], 95.00th=[17433],
     | 99.00th=[26084], 99.50th=[27657], 99.90th=[29230], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 2000, max= 2960, per=24.06%, avg=2608.00, stdev=528.73, samples=3
   iops        : min=  500, max=  740, avg=652.00, stdev=132.18, samples=3
  write: IOPS=714, BW=2856KiB/s (2925kB/s)(5264KiB/1843msec)
    slat (usec): min=13, max=535, avg=40.52, stdev=28.93
    clat (usec): min=7202, max=29538, avg=11283.98, stdev=3060.03
     lat (usec): min=7229, max=29669, avg=11325.56, stdev=3069.14
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 8717], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10028], 40.00th=[10421], 50.00th=[10683], 60.00th=[10945],
     | 70.00th=[11207], 80.00th=[11600], 90.00th=[12649], 95.00th=[18744],
     | 99.00th=[25822], 99.50th=[27919], 99.90th=[29492], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 2232, max= 3128, per=24.90%, avg=2810.67, stdev=501.92, samples=3
   iops        : min=  558, max=  782, avg=702.67, stdev=125.48, samples=3
  lat (msec)   : 2=0.04%, 4=0.08%, 10=33.75%, 20=62.15%, 50=3.98%
  cpu          : usr=1.30%, sys=9.39%, ctx=2652, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18491: Thu Aug  1 00:48:21 2024
   read: IOPS=673, BW=2693KiB/s (2757kB/s)(4984KiB/1851msec)
    slat (usec): min=404, max=2312, avg=696.78, stdev=245.43
    clat (usec): min=1447, max=31452, avg=10906.00, stdev=3170.24
     lat (usec): min=1974, max=33336, avg=11604.45, stdev=3353.81
    clat percentiles (usec):
     |  1.00th=[ 7242],  5.00th=[ 8160], 10.00th=[ 8586], 20.00th=[ 8979],
     | 30.00th=[ 9503], 40.00th=[ 9896], 50.00th=[10290], 60.00th=[10814],
     | 70.00th=[11207], 80.00th=[11600], 90.00th=[12649], 95.00th=[16319],
     | 99.00th=[25822], 99.50th=[28181], 99.90th=[30802], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 2080, max= 2936, per=24.28%, avg=2632.00, stdev=478.87, samples=3
   iops        : min=  520, max=  734, avg=658.00, stdev=119.72, samples=3
  write: IOPS=709, BW=2840KiB/s (2908kB/s)(5256KiB/1851msec)
    slat (usec): min=14, max=222, avg=38.46, stdev=21.14
    clat (usec): min=7199, max=31560, avg=11287.99, stdev=3212.36
     lat (usec): min=7248, max=31660, avg=11327.40, stdev=3219.67
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9634],
     | 30.00th=[10028], 40.00th=[10290], 50.00th=[10552], 60.00th=[10945],
     | 70.00th=[11207], 80.00th=[11600], 90.00th=[12649], 95.00th=[18744],
     | 99.00th=[27657], 99.50th=[28181], 99.90th=[29230], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 2016, max= 3152, per=24.24%, avg=2736.00, stdev=626.05, samples=3
   iops        : min=  504, max=  788, avg=684.00, stdev=156.51, samples=3
  lat (msec)   : 2=0.04%, 10=34.34%, 20=61.95%, 50=3.67%
  cpu          : usr=0.86%, sys=9.68%, ctx=2616, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18492: Thu Aug  1 00:48:21 2024
   read: IOPS=684, BW=2737KiB/s (2802kB/s)(5060KiB/1849msec)
    slat (usec): min=389, max=2319, avg=694.63, stdev=246.30
    clat (usec): min=2942, max=31896, avg=10898.88, stdev=3290.36
     lat (usec): min=3809, max=33675, avg=11595.14, stdev=3477.93
    clat percentiles (usec):
     |  1.00th=[ 6980],  5.00th=[ 8160], 10.00th=[ 8586], 20.00th=[ 9110],
     | 30.00th=[ 9503], 40.00th=[ 9896], 50.00th=[10290], 60.00th=[10683],
     | 70.00th=[10945], 80.00th=[11600], 90.00th=[12518], 95.00th=[17171],
     | 99.00th=[26608], 99.50th=[28181], 99.90th=[31589], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 2024, max= 2992, per=24.35%, avg=2640.00, stdev=535.28, samples=3
   iops        : min=  506, max=  748, avg=660.00, stdev=133.82, samples=3
  write: IOPS=700, BW=2802KiB/s (2869kB/s)(5180KiB/1849msec)
    slat (usec): min=13, max=274, avg=38.99, stdev=23.71
    clat (usec): min=2164, max=29922, avg=11277.59, stdev=3161.02
     lat (usec): min=2202, max=30023, avg=11317.61, stdev=3168.74
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10028], 40.00th=[10290], 50.00th=[10683], 60.00th=[10945],
     | 70.00th=[11207], 80.00th=[11600], 90.00th=[12780], 95.00th=[18220],
     | 99.00th=[26870], 99.50th=[28181], 99.90th=[28705], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 2088, max= 3072, per=24.17%, avg=2728.00, stdev=554.78, samples=3
   iops        : min=  522, max=  768, avg=682.00, stdev=138.69, samples=3
  lat (msec)   : 4=0.12%, 10=35.20%, 20=60.66%, 50=4.02%
  cpu          : usr=1.19%, sys=9.31%, ctx=2627, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.6MiB/s (11.1MB/s), 2693KiB/s-2737KiB/s (2757kB/s-2802kB/s), io=19.6MiB (20.5MB), run=1843-1851msec
  WRITE: bw=11.0MiB/s (11.6MB/s), 2802KiB/s-2856KiB/s (2869kB/s-2925kB/s), io=20.4MiB (21.4MB), run=1843-1851msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18498: Thu Aug  1 00:48:23 2024
   read: IOPS=10, BW=440MiB/s (461MB/s)(40.0MiB/91msec)
    slat (nsec): min=11898k, max=11898k, avg=11898198.00, stdev= 0.00
    clat (nsec): min=74162k, max=74162k, avg=74162110.00, stdev= 0.00
     lat (nsec): min=86069k, max=86069k, avg=86068870.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[73925],  5.00th=[73925], 10.00th=[73925], 20.00th=[73925],
     | 30.00th=[73925], 40.00th=[73925], 50.00th=[73925], 60.00th=[73925],
     | 70.00th=[73925], 80.00th=[73925], 90.00th=[73925], 95.00th=[73925],
     | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=14.44%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=440MiB/s (461MB/s), 440MiB/s-440MiB/s (461MB/s-461MB/s), io=40.0MiB (41.9MB), run=91-91msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck