== sanity test 398c: run fio to test AIO ================= 03:36:58 (1746085018)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0779809 s, 538 MB/s
osc.lustre-OST0000-osc-ffff88013039f800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=22227: Thu May  1 03:37:03 2025
  write: IOPS=801, BW=3204KiB/s (3281kB/s)(10.0MiB/3196msec)
    slat (usec): min=11, max=413, avg=48.65, stdev=31.14
    clat (usec): min=1599, max=33343, avg=19834.79, stdev=3387.38
     lat (usec): min=1644, max=33366, avg=19884.60, stdev=3390.92
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[14484], 10.00th=[15795], 20.00th=[17171],
     | 30.00th=[18220], 40.00th=[19268], 50.00th=[19792], 60.00th=[20579],
     | 70.00th=[21365], 80.00th=[22414], 90.00th=[23725], 95.00th=[25035],
     | 99.00th=[29230], 99.50th=[31327], 99.90th=[32375], 99.95th=[33162],
     | 99.99th=[33424]
   bw (  KiB/s): min= 2856, max= 3656, per=25.02%, avg=3192.00, stdev=316.54, samples=6
   iops        : min=  714, max=  914, avg=798.00, stdev=79.14, samples=6
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.47%, 20=50.86%, 50=48.52%
  cpu          : usr=1.06%, sys=3.94%, ctx=1525, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22228: Thu May  1 03:37:03 2025
  write: IOPS=797, BW=3191KiB/s (3268kB/s)(10.0MiB/3209msec)
    slat (usec): min=12, max=463, avg=46.86, stdev=30.13
    clat (usec): min=2760, max=35986, avg=19886.70, stdev=3279.10
     lat (usec): min=2829, max=36026, avg=19934.74, stdev=3281.53
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[14484], 10.00th=[15795], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19792], 60.00th=[20579],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23725], 95.00th=[25035],
     | 99.00th=[29492], 99.50th=[30802], 99.90th=[32113], 99.95th=[33817],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2872, max= 3600, per=24.92%, avg=3178.67, stdev=298.68, samples=6
   iops        : min=  718, max=  900, avg=794.67, stdev=74.67, samples=6
  lat (msec)   : 4=0.12%, 10=0.08%, 20=51.17%, 50=48.63%
  cpu          : usr=0.84%, sys=3.96%, ctx=1374, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22229: Thu May  1 03:37:03 2025
  write: IOPS=798, BW=3193KiB/s (3270kB/s)(10.0MiB/3207msec)
    slat (usec): min=12, max=426, avg=48.52, stdev=30.32
    clat (usec): min=3117, max=32151, avg=19894.23, stdev=3153.81
     lat (usec): min=3202, max=32247, avg=19944.14, stdev=3155.39
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14746], 10.00th=[15795], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19792], 60.00th=[20579],
     | 70.00th=[21365], 80.00th=[22414], 90.00th=[23725], 95.00th=[24773],
     | 99.00th=[28181], 99.50th=[31065], 99.90th=[31851], 99.95th=[31851],
     | 99.99th=[32113]
   bw (  KiB/s): min= 2824, max= 3568, per=24.91%, avg=3177.33, stdev=301.95, samples=6
   iops        : min=  706, max=  892, avg=794.33, stdev=75.49, samples=6
  lat (msec)   : 4=0.08%, 10=0.04%, 20=51.21%, 50=48.67%
  cpu          : usr=1.25%, sys=3.81%, ctx=1640, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22230: Thu May  1 03:37:03 2025
  write: IOPS=797, BW=3189KiB/s (3266kB/s)(10.0MiB/3211msec)
    slat (usec): min=11, max=459, avg=47.74, stdev=31.18
    clat (usec): min=11216, max=32242, avg=19918.56, stdev=3038.63
     lat (usec): min=11235, max=32286, avg=19967.42, stdev=3041.10
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14877], 10.00th=[15926], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19268], 50.00th=[19792], 60.00th=[20579],
     | 70.00th=[21365], 80.00th=[22414], 90.00th=[23462], 95.00th=[25035],
     | 99.00th=[27395], 99.50th=[31065], 99.90th=[32113], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 2848, max= 3528, per=24.87%, avg=3172.00, stdev=280.83, samples=6
   iops        : min=  712, max=  882, avg=793.00, stdev=70.21, samples=6
  lat (msec)   : 20=52.07%, 50=47.93%
  cpu          : usr=0.87%, sys=4.14%, ctx=1524, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.5MiB/s (13.1MB/s), 3189KiB/s-3204KiB/s (3266kB/s-3281kB/s), io=40.0MiB (41.9MB), run=3196-3211msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=22243: Thu May  1 03:37:06 2025
   read: IOPS=431, BW=1724KiB/s (1765kB/s)(5048KiB/2928msec)
    slat (usec): min=674, max=2765, avg=1188.64, stdev=279.60
    clat (usec): min=2040, max=26362, avg=17189.97, stdev=2829.20
     lat (usec): min=3452, max=28205, avg=18380.56, stdev=2891.20
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12911], 10.00th=[13698], 20.00th=[14746],
     | 30.00th=[15533], 40.00th=[16450], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20841], 95.00th=[21890],
     | 99.00th=[24511], 99.50th=[25035], 99.90th=[26084], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1544, max= 1872, per=24.64%, avg=1688.80, stdev=121.34, samples=5
   iops        : min=  386, max=  468, avg=422.20, stdev=30.33, samples=5
  write: IOPS=443, BW=1773KiB/s (1816kB/s)(5192KiB/2928msec)
    slat (usec): min=12, max=328, avg=45.41, stdev=23.67
    clat (usec): min=2005, max=26468, avg=17833.13, stdev=2366.42
     lat (usec): min=2034, max=26500, avg=17879.72, stdev=2367.45
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14353], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[19530], 90.00th=[21103], 95.00th=[21890],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[25822], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1600, max= 1952, per=24.97%, avg=1781.60, stdev=136.52, samples=5
   iops        : min=  400, max=  488, avg=445.40, stdev=34.13, samples=5
  lat (msec)   : 4=0.16%, 10=0.16%, 20=83.75%, 50=15.94%
  cpu          : usr=1.02%, sys=7.11%, ctx=2393, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22244: Thu May  1 03:37:06 2025
   read: IOPS=427, BW=1711KiB/s (1752kB/s)(4976KiB/2909msec)
    slat (usec): min=15, max=2853, avg=1189.44, stdev=295.86
    clat (usec): min=2281, max=35438, avg=17242.88, stdev=2923.54
     lat (usec): min=2538, max=36748, avg=18434.07, stdev=2999.29
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13042], 10.00th=[13829], 20.00th=[14877],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[20841], 95.00th=[21890],
     | 99.00th=[24773], 99.50th=[26346], 99.90th=[33424], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1568, max= 1864, per=24.71%, avg=1693.60, stdev=106.64, samples=5
   iops        : min=  392, max=  466, avg=423.40, stdev=26.66, samples=5
  write: IOPS=452, BW=1810KiB/s (1853kB/s)(5264KiB/2909msec)
    slat (usec): min=11, max=652, avg=46.57, stdev=31.03
    clat (usec): min=6574, max=34427, avg=17793.74, stdev=2406.12
     lat (usec): min=6587, max=34488, avg=17841.51, stdev=2408.35
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14353], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20579], 95.00th=[22152],
     | 99.00th=[24773], 99.50th=[25297], 99.90th=[27919], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1752, max= 1952, per=25.64%, avg=1829.60, stdev=87.62, samples=5
   iops        : min=  438, max=  488, avg=457.40, stdev=21.90, samples=5
  lat (msec)   : 4=0.12%, 10=0.12%, 20=84.26%, 50=15.51%
  cpu          : usr=0.55%, sys=7.50%, ctx=2390, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22245: Thu May  1 03:37:06 2025
   read: IOPS=426, BW=1705KiB/s (1745kB/s)(4984KiB/2924msec)
    slat (usec): min=671, max=2960, avg=1188.87, stdev=288.65
    clat (usec): min=7252, max=28112, avg=17230.84, stdev=2716.43
     lat (usec): min=8168, max=29731, avg=18421.56, stdev=2773.60
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13173], 10.00th=[13960], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16450], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[20841], 95.00th=[21627],
     | 99.00th=[24511], 99.50th=[25297], 99.90th=[26346], 99.95th=[28181],
     | 99.99th=[28181]
   bw (  KiB/s): min= 1600, max= 1848, per=24.67%, avg=1690.40, stdev=96.98, samples=5
   iops        : min=  400, max=  462, avg=422.60, stdev=24.24, samples=5
  write: IOPS=449, BW=1798KiB/s (1841kB/s)(5256KiB/2924msec)
    slat (usec): min=12, max=264, avg=45.17, stdev=21.90
    clat (usec): min=12338, max=27938, avg=17834.11, stdev=2279.70
     lat (usec): min=12378, max=27987, avg=17880.42, stdev=2281.04
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14484], 10.00th=[15008], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20841], 95.00th=[21890],
     | 99.00th=[24249], 99.50th=[25035], 99.90th=[27132], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1656, max= 2176, per=25.28%, avg=1804.00, stdev=217.51, samples=5
   iops        : min=  414, max=  544, avg=451.00, stdev=54.38, samples=5
  lat (msec)   : 10=0.20%, 20=84.18%, 50=15.62%
  cpu          : usr=1.13%, sys=6.88%, ctx=2369, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22246: Thu May  1 03:37:06 2025
   read: IOPS=432, BW=1731KiB/s (1772kB/s)(5060KiB/2924msec)
    slat (usec): min=644, max=2805, avg=1180.71, stdev=284.24
    clat (usec): min=2392, max=26465, avg=17103.30, stdev=2725.59
     lat (usec): min=3422, max=28399, avg=18285.83, stdev=2802.36
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12780], 10.00th=[13566], 20.00th=[14615],
     | 30.00th=[15533], 40.00th=[16319], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[23987], 99.50th=[24511], 99.90th=[26346], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 1608, max= 1856, per=25.02%, avg=1714.40, stdev=93.71, samples=5
   iops        : min=  402, max=  464, avg=428.60, stdev=23.43, samples=5
  write: IOPS=442, BW=1772KiB/s (1814kB/s)(5180KiB/2924msec)
    slat (usec): min=12, max=1021, avg=48.31, stdev=47.21
    clat (usec): min=3774, max=27064, avg=17856.27, stdev=2398.90
     lat (usec): min=3800, max=27162, avg=17906.10, stdev=2401.69
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14353], 10.00th=[15008], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19792], 90.00th=[21103], 95.00th=[22152],
     | 99.00th=[24249], 99.50th=[24773], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1600, max= 1968, per=24.86%, avg=1773.60, stdev=143.88, samples=5
   iops        : min=  400, max=  492, avg=443.40, stdev=35.97, samples=5
  lat (msec)   : 4=0.08%, 10=0.08%, 20=84.34%, 50=15.51%
  cpu          : usr=0.75%, sys=7.39%, ctx=2455, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6854KiB/s (7018kB/s), 1705KiB/s-1731KiB/s (1745kB/s-1772kB/s), io=19.6MiB (20.5MB), run=2909-2928msec
  WRITE: bw=7135KiB/s (7306kB/s), 1772KiB/s-1810KiB/s (1814kB/s-1853kB/s), io=20.4MiB (21.4MB), run=2909-2928msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=22252: Thu May  1 03:37:08 2025
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=5664.9k, max=5664.9k, avg=5664897.00, stdev= 0.00
    clat (nsec): min=67077k, max=67077k, avg=67077324.00, stdev= 0.00
     lat (nsec): min=72805k, max=72805k, avg=72804661.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[66847],  5.00th=[66847], 10.00th=[66847], 20.00th=[66847],
     | 30.00th=[66847], 40.00th=[66847], 50.00th=[66847], 60.00th=[66847],
     | 70.00th=[66847], 80.00th=[66847], 90.00th=[66847], 95.00th=[66847],
     | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847],
     | 99.99th=[66847]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.89%, ctx=7, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout