== sanity test 398c: run fio to test AIO ================= 01:08:54 (1747804134)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.112628 s, 372 MB/s
osc.lustre-OST0000-osc-ffff880084b6b000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=3314: Wed May 21 01:09:03 2025
  write: IOPS=366, BW=1466KiB/s (1502kB/s)(10.0MiB/6983msec)
    slat (usec): min=18, max=662, avg=95.11, stdev=48.30
    clat (usec): min=4641, max=97026, avg=43438.13, stdev=6272.21
     lat (usec): min=4890, max=97086, avg=43535.48, stdev=6274.67
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[31589], 10.00th=[36439], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44827], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[48497], 95.00th=[49021],
     | 99.00th=[51119], 99.50th=[58459], 99.90th=[87557], 99.95th=[91751],
     | 99.99th=[96994]
   bw (  KiB/s): min= 1376, max= 1820, per=25.43%, avg=1476.69, stdev=129.03, samples=13
   iops        : min=  344, max=  455, avg=369.15, stdev=32.26, samples=13
  lat (msec)   : 10=0.27%, 20=0.23%, 50=97.42%, 100=2.07%
  cpu          : usr=0.82%, sys=3.97%, ctx=1930, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=3315: Wed May 21 01:09:03 2025
  write: IOPS=363, BW=1453KiB/s (1488kB/s)(10.0MiB/7049msec)
    slat (usec): min=17, max=397, avg=86.27, stdev=46.77
    clat (usec): min=10033, max=94042, avg=43769.68, stdev=5742.68
     lat (usec): min=10157, max=94080, avg=43858.19, stdev=5748.31
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[33817], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44827], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47973], 90.00th=[48497], 95.00th=[49021],
     | 99.00th=[50594], 99.50th=[54789], 99.90th=[93848], 99.95th=[93848],
     | 99.99th=[93848]
   bw (  KiB/s): min= 1312, max= 1676, per=25.00%, avg=1451.79, stdev=108.46, samples=14
   iops        : min=  328, max=  419, avg=362.93, stdev=27.12, samples=14
  lat (msec)   : 20=0.04%, 50=98.09%, 100=1.88%
  cpu          : usr=0.54%, sys=3.65%, ctx=1537, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=3316: Wed May 21 01:09:03 2025
  write: IOPS=363, BW=1452KiB/s (1487kB/s)(10.0MiB/7052msec)
    slat (usec): min=13, max=567, avg=91.94, stdev=49.31
    clat (usec): min=24755, max=83539, avg=43859.72, stdev=5350.27
     lat (usec): min=24799, max=83644, avg=43953.96, stdev=5352.12
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[33817], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44827], 60.00th=[46400],
     | 70.00th=[46924], 80.00th=[47973], 90.00th=[48497], 95.00th=[49546],
     | 99.00th=[51119], 99.50th=[53216], 99.90th=[79168], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1304, max= 1668, per=24.96%, avg=1449.50, stdev=112.96, samples=14
   iops        : min=  326, max=  417, avg=362.36, stdev=28.25, samples=14
  lat (msec)   : 50=97.42%, 100=2.58%
  cpu          : usr=0.74%, sys=3.70%, ctx=1801, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=3317: Wed May 21 01:09:03 2025
  write: IOPS=363, BW=1455KiB/s (1490kB/s)(10.0MiB/7038msec)
    slat (usec): min=18, max=444, avg=93.92, stdev=48.52
    clat (usec): min=6696, max=98468, avg=43731.85, stdev=5708.29
     lat (usec): min=6845, max=98511, avg=43827.95, stdev=5712.55
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[32637], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44827], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47973], 90.00th=[48497], 95.00th=[49546],
     | 99.00th=[51119], 99.50th=[55313], 99.90th=[88605], 99.95th=[93848],
     | 99.99th=[98042]
   bw (  KiB/s): min= 1312, max= 1696, per=25.03%, avg=1453.64, stdev=113.36, samples=14
   iops        : min=  328, max=  424, avg=363.36, stdev=28.36, samples=14
  lat (msec)   : 10=0.04%, 50=97.70%, 100=2.27%
  cpu          : usr=0.55%, sys=3.98%, ctx=1924, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5808KiB/s (5948kB/s), 1452KiB/s-1466KiB/s (1487kB/s-1502kB/s), io=40.0MiB (41.9MB), run=6983-7052msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=3333: Wed May 21 01:09:07 2025
   read: IOPS=445, BW=1784KiB/s (1827kB/s)(5048KiB/2830msec)
    slat (usec): min=710, max=3481, avg=1255.31, stdev=368.89
    clat (usec): min=2125, max=38043, avg=16547.62, stdev=4191.28
     lat (usec): min=3228, max=41529, avg=17805.11, stdev=4427.86
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12387], 10.00th=[12911], 20.00th=[13829],
     | 30.00th=[14484], 40.00th=[15139], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[20055], 95.00th=[25035],
     | 99.00th=[33817], 99.50th=[35390], 99.90th=[36963], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1152, max= 1984, per=24.69%, avg=1748.80, stdev=339.63, samples=5
   iops        : min=  288, max=  496, avg=437.20, stdev=84.91, samples=5
  write: IOPS=458, BW=1835KiB/s (1879kB/s)(5192KiB/2830msec)
    slat (usec): min=15, max=1520, avg=60.38, stdev=61.30
    clat (usec): min=11411, max=38857, avg=17140.43, stdev=4129.77
     lat (usec): min=11458, max=38905, avg=17202.28, stdev=4136.50
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13566], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[20055], 95.00th=[27657],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[37487], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1096, max= 2136, per=24.80%, avg=1828.80, stdev=416.06, samples=5
   iops        : min=  274, max=  534, avg=457.20, stdev=104.02, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=89.88%, 50=9.92%
  cpu          : usr=1.03%, sys=9.40%, ctx=2898, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=3334: Wed May 21 01:09:07 2025
   read: IOPS=442, BW=1770KiB/s (1813kB/s)(4976KiB/2811msec)
    slat (usec): min=71, max=3322, avg=1251.49, stdev=358.15
    clat (usec): min=1679, max=39806, avg=16547.19, stdev=4433.69
     lat (usec): min=2705, max=41936, avg=17801.03, stdev=4636.93
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12256], 10.00th=[12911], 20.00th=[13829],
     | 30.00th=[14484], 40.00th=[15008], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19792], 95.00th=[26608],
     | 99.00th=[35390], 99.50th=[37487], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1104, max= 1984, per=24.62%, avg=1744.00, stdev=361.11, samples=5
   iops        : min=  276, max=  496, avg=436.00, stdev=90.28, samples=5
  write: IOPS=468, BW=1873KiB/s (1918kB/s)(5264KiB/2811msec)
    slat (usec): min=15, max=401, avg=58.16, stdev=36.97
    clat (usec): min=3762, max=39965, avg=17164.42, stdev=4023.38
     lat (usec): min=3867, max=40003, avg=17224.04, stdev=4029.68
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13435], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[20317], 95.00th=[27919],
     | 99.00th=[33817], 99.50th=[35390], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1312, max= 2088, per=25.39%, avg=1872.00, stdev=319.65, samples=5
   iops        : min=  328, max=  522, avg=468.00, stdev=79.91, samples=5
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.23%, 20=89.57%, 50=10.08%
  cpu          : usr=1.14%, sys=9.15%, ctx=2801, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=3335: Wed May 21 01:09:07 2025
   read: IOPS=439, BW=1760KiB/s (1802kB/s)(4984KiB/2832msec)
    slat (usec): min=698, max=3564, avg=1249.84, stdev=354.42
    clat (usec): min=9684, max=43437, avg=16480.43, stdev=4176.68
     lat (usec): min=10923, max=45412, avg=17732.48, stdev=4388.21
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12256], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14484], 40.00th=[15008], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17957], 90.00th=[20055], 95.00th=[24511],
     | 99.00th=[33817], 99.50th=[36963], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1128, max= 2000, per=24.48%, avg=1733.60, stdev=350.56, samples=5
   iops        : min=  282, max=  500, avg=433.40, stdev=87.64, samples=5
  write: IOPS=463, BW=1856KiB/s (1900kB/s)(5256KiB/2832msec)
    slat (usec): min=14, max=1770, avg=61.74, stdev=64.66
    clat (usec): min=6881, max=39648, avg=17199.97, stdev=4144.18
     lat (usec): min=6933, max=39926, avg=17263.02, stdev=4158.07
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13566], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[20317], 95.00th=[27657],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[39060], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1080, max= 2216, per=24.96%, avg=1840.60, stdev=439.84, samples=5
   iops        : min=  270, max=  554, avg=460.00, stdev=109.89, samples=5
  lat (msec)   : 10=0.08%, 20=89.30%, 50=10.62%
  cpu          : usr=0.88%, sys=9.47%, ctx=2842, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=3336: Wed May 21 01:09:07 2025
   read: IOPS=446, BW=1786KiB/s (1829kB/s)(5060KiB/2833msec)
    slat (usec): min=717, max=3255, avg=1254.76, stdev=350.69
    clat (usec): min=992, max=38011, avg=16592.48, stdev=4357.24
     lat (usec): min=2866, max=40431, avg=17849.48, stdev=4559.26
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[12125], 10.00th=[12911], 20.00th=[13829],
     | 30.00th=[14615], 40.00th=[15270], 50.00th=[15926], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[18220], 90.00th=[20055], 95.00th=[24773],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1104, max= 1976, per=24.71%, avg=1750.40, stdev=365.71, samples=5
   iops        : min=  276, max=  494, avg=437.60, stdev=91.43, samples=5
  write: IOPS=457, BW=1828KiB/s (1872kB/s)(5180KiB/2833msec)
    slat (usec): min=13, max=1146, avg=57.61, stdev=51.35
    clat (usec): min=2869, max=37120, avg=17036.88, stdev=4092.85
     lat (usec): min=2886, max=37232, avg=17095.77, stdev=4101.96
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13435], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16188], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[19792], 95.00th=[25297],
     | 99.00th=[34341], 99.50th=[34866], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1072, max= 2008, per=24.39%, avg=1798.40, stdev=407.66, samples=5
   iops        : min=  268, max=  502, avg=449.60, stdev=101.92, samples=5
  lat (usec)   : 1000=0.04%
  lat (msec)   : 4=0.23%, 10=0.31%, 20=89.41%, 50=10.00%
  cpu          : usr=1.48%, sys=8.72%, ctx=2859, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7084KiB/s (7254kB/s), 1760KiB/s-1786KiB/s (1802kB/s-1829kB/s), io=19.6MiB (20.5MB), run=2811-2833msec
  WRITE: bw=7375KiB/s (7552kB/s), 1828KiB/s-1873KiB/s (1872kB/s-1918kB/s), io=20.4MiB (21.4MB), run=2811-2833msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=3342: Wed May 21 01:09:08 2025
   read: IOPS=5, BW=229MiB/s (240MB/s)(40.0MiB/175msec)
    slat (nsec): min=6828.5k, max=6828.5k, avg=6828465.00, stdev= 0.00
    clat (nsec): min=161857k, max=161857k, avg=161856826.00, stdev= 0.00
     lat (nsec): min=168696k, max=168696k, avg=168695884.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  163], 10.00th=[  163], 20.00th=[  163],
     | 30.00th=[  163], 40.00th=[  163], 50.00th=[  163], 60.00th=[  163],
     | 70.00th=[  163], 80.00th=[  163], 90.00th=[  163], 95.00th=[  163],
     | 99.00th=[  163], 99.50th=[  163], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.45%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=40.0MiB (41.9MB), run=175-175msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout