== sanity test 398c: run fio to test AIO ================= 07:46:28 (1749123988)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0947787 s, 443 MB/s
osc.lustre-OST0000-osc-ffff8800b63d6800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=5042: Thu Jun  5 07:46:33 2025
  write: IOPS=675, BW=2703KiB/s (2768kB/s)(10.0MiB/3788msec)
    slat (usec): min=13, max=505, avg=46.68, stdev=27.43
    clat (usec): min=2751, max=40179, avg=23525.94, stdev=2843.19
     lat (usec): min=2822, max=40228, avg=23573.76, stdev=2844.41
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[19530], 10.00th=[20579], 20.00th=[21890],
     | 30.00th=[22414], 40.00th=[22938], 50.00th=[23462], 60.00th=[23987],
     | 70.00th=[24773], 80.00th=[25822], 90.00th=[26346], 95.00th=[27657],
     | 99.00th=[30278], 99.50th=[31065], 99.90th=[38011], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2632, max= 2808, per=24.96%, avg=2696.29, stdev=63.88, samples=7
   iops        : min=  658, max=  702, avg=674.00, stdev=15.92, samples=7
  lat (msec)   : 4=0.08%, 20=6.84%, 50=93.09%
  cpu          : usr=0.71%, sys=3.56%, ctx=1316, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5043: Thu Jun  5 07:46:33 2025
  write: IOPS=675, BW=2701KiB/s (2766kB/s)(10.0MiB/3791msec)
    slat (usec): min=12, max=499, avg=49.70, stdev=30.91
    clat (usec): min=3418, max=34286, avg=23557.74, stdev=2692.35
     lat (usec): min=3520, max=34331, avg=23608.65, stdev=2691.96
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[19268], 10.00th=[20579], 20.00th=[21890],
     | 30.00th=[22414], 40.00th=[22938], 50.00th=[23462], 60.00th=[24249],
     | 70.00th=[24773], 80.00th=[25822], 90.00th=[26608], 95.00th=[27919],
     | 99.00th=[30016], 99.50th=[30540], 99.90th=[31327], 99.95th=[31589],
     | 99.99th=[34341]
   bw (  KiB/s): min= 2592, max= 2816, per=24.88%, avg=2687.14, stdev=77.49, samples=7
   iops        : min=  648, max=  704, avg=671.71, stdev=19.30, samples=7
  lat (msec)   : 4=0.04%, 10=0.04%, 20=6.84%, 50=93.09%
  cpu          : usr=0.66%, sys=3.91%, ctx=1444, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5044: Thu Jun  5 07:46:33 2025
  write: IOPS=675, BW=2700KiB/s (2765kB/s)(10.0MiB/3792msec)
    slat (usec): min=12, max=317, avg=46.12, stdev=26.96
    clat (usec): min=11352, max=31261, avg=23531.67, stdev=2648.39
     lat (usec): min=11424, max=31291, avg=23578.95, stdev=2647.40
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[19792], 10.00th=[20579], 20.00th=[21890],
     | 30.00th=[22414], 40.00th=[22938], 50.00th=[23462], 60.00th=[24249],
     | 70.00th=[24773], 80.00th=[25560], 90.00th=[26608], 95.00th=[27395],
     | 99.00th=[29754], 99.50th=[30540], 99.90th=[30802], 99.95th=[31065],
     | 99.99th=[31327]
   bw (  KiB/s): min= 2600, max= 2816, per=24.87%, avg=2685.71, stdev=82.07, samples=7
   iops        : min=  650, max=  704, avg=671.43, stdev=20.52, samples=7
  lat (msec)   : 20=6.64%, 50=93.36%
  cpu          : usr=0.71%, sys=3.51%, ctx=1351, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=5045: Thu Jun  5 07:46:33 2025
  write: IOPS=679, BW=2716KiB/s (2781kB/s)(10.0MiB/3770msec)
    slat (usec): min=13, max=503, avg=50.55, stdev=31.17
    clat (usec): min=4437, max=37863, avg=23434.11, stdev=2924.70
     lat (usec): min=4501, max=37950, avg=23485.91, stdev=2923.38
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[19530], 10.00th=[20579], 20.00th=[21627],
     | 30.00th=[22414], 40.00th=[22938], 50.00th=[23462], 60.00th=[24249],
     | 70.00th=[24773], 80.00th=[25822], 90.00th=[26346], 95.00th=[27132],
     | 99.00th=[30278], 99.50th=[31065], 99.90th=[35390], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 2616, max= 2816, per=25.09%, avg=2710.00, stdev=78.60, samples=7
   iops        : min=  654, max=  704, avg=677.43, stdev=19.59, samples=7
  lat (msec)   : 10=0.27%, 20=6.91%, 50=92.81%
  cpu          : usr=0.66%, sys=3.90%, ctx=1586, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.5MiB/s (11.1MB/s), 2700KiB/s-2716KiB/s (2765kB/s-2781kB/s), io=40.0MiB (41.9MB), run=3770-3792msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=5057: Thu Jun  5 07:46:37 2025
   read: IOPS=395, BW=1580KiB/s (1618kB/s)(5048KiB/3194msec)
    slat (usec): min=739, max=2685, avg=1272.68, stdev=247.74
    clat (usec): min=1493, max=31141, avg=18819.00, stdev=2944.36
     lat (usec): min=2864, max=33828, avg=20093.57, stdev=2980.75
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[14615], 10.00th=[15401], 20.00th=[16450],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[22414], 95.00th=[23987],
     | 99.00th=[26870], 99.50th=[28181], 99.90th=[31065], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1416, max= 1688, per=25.09%, avg=1575.33, stdev=103.83, samples=6
   iops        : min=  354, max=  422, avg=393.83, stdev=25.96, samples=6
  write: IOPS=406, BW=1626KiB/s (1665kB/s)(5192KiB/3194msec)
    slat (usec): min=16, max=169, avg=45.73, stdev=19.23
    clat (usec): min=12297, max=30252, avg=19494.65, stdev=2372.87
     lat (usec): min=12336, max=30278, avg=19541.54, stdev=2372.50
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[16057], 10.00th=[16712], 20.00th=[17695],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22414], 95.00th=[23462],
     | 99.00th=[26346], 99.50th=[27919], 99.90th=[30016], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1496, max= 1728, per=24.90%, avg=1627.33, stdev=76.33, samples=6
   iops        : min=  374, max=  432, avg=406.83, stdev=19.08, samples=6
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.08%, 20=65.35%, 50=34.49%
  cpu          : usr=0.78%, sys=6.92%, ctx=2295, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5058: Thu Jun  5 07:46:37 2025
   read: IOPS=390, BW=1560KiB/s (1598kB/s)(4976KiB/3189msec)
    slat (usec): min=749, max=2662, avg=1279.81, stdev=257.76
    clat (usec): min=4249, max=34979, avg=18988.81, stdev=3105.58
     lat (usec): min=5731, max=36337, avg=20270.41, stdev=3140.99
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14353], 10.00th=[15401], 20.00th=[16450],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20317], 80.00th=[21103], 90.00th=[22676], 95.00th=[24249],
     | 99.00th=[28443], 99.50th=[29492], 99.90th=[33817], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1416, max= 1720, per=24.69%, avg=1550.00, stdev=104.79, samples=6
   iops        : min=  354, max=  430, avg=387.50, stdev=26.20, samples=6
  write: IOPS=412, BW=1651KiB/s (1690kB/s)(5264KiB/3189msec)
    slat (usec): min=13, max=984, avg=47.31, stdev=35.84
    clat (usec): min=4424, max=39730, avg=19404.62, stdev=2669.06
     lat (usec): min=4455, max=39824, avg=19453.09, stdev=2670.16
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[15795], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21103], 90.00th=[22414], 95.00th=[23725],
     | 99.00th=[27395], 99.50th=[28443], 99.90th=[36963], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1576, max= 1764, per=25.49%, avg=1666.00, stdev=79.43, samples=6
   iops        : min=  394, max=  441, avg=416.50, stdev=19.86, samples=6
  lat (msec)   : 10=0.31%, 20=64.06%, 50=35.62%
  cpu          : usr=0.97%, sys=6.62%, ctx=2291, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5059: Thu Jun  5 07:46:37 2025
   read: IOPS=393, BW=1573KiB/s (1610kB/s)(4984KiB/3169msec)
    slat (usec): min=12, max=4644, avg=1277.72, stdev=289.82
    clat (usec): min=1843, max=34593, avg=18729.23, stdev=3237.79
     lat (usec): min=1857, max=36529, avg=20008.82, stdev=3305.33
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[14484], 10.00th=[15401], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22676], 95.00th=[23987],
     | 99.00th=[27395], 99.50th=[28967], 99.90th=[30016], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1448, max= 1700, per=25.17%, avg=1580.67, stdev=91.23, samples=6
   iops        : min=  362, max=  425, avg=395.17, stdev=22.81, samples=6
  write: IOPS=414, BW=1659KiB/s (1698kB/s)(5256KiB/3169msec)
    slat (usec): min=12, max=1167, avg=47.14, stdev=38.46
    clat (usec): min=6368, max=40915, avg=19484.27, stdev=2823.51
     lat (usec): min=6493, max=40962, avg=19532.59, stdev=2824.93
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[15926], 10.00th=[16712], 20.00th=[17695],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22414], 95.00th=[23462],
     | 99.00th=[27395], 99.50th=[28967], 99.90th=[40633], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1416, max= 1752, per=25.29%, avg=1652.67, stdev=132.43, samples=6
   iops        : min=  354, max=  438, avg=413.17, stdev=33.11, samples=6
  lat (msec)   : 2=0.08%, 4=0.08%, 10=0.70%, 20=64.18%, 50=34.96%
  cpu          : usr=1.10%, sys=6.50%, ctx=2272, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=5060: Thu Jun  5 07:46:37 2025
   read: IOPS=395, BW=1583KiB/s (1621kB/s)(5060KiB/3196msec)
    slat (usec): min=724, max=2823, avg=1275.34, stdev=259.20
    clat (usec): min=1573, max=32180, avg=18950.46, stdev=3085.07
     lat (usec): min=3313, max=35005, avg=20227.63, stdev=3120.48
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[14615], 10.00th=[15533], 20.00th=[16450],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[19006], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21365], 90.00th=[22414], 95.00th=[23725],
     | 99.00th=[27657], 99.50th=[28705], 99.90th=[31851], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1456, max= 1640, per=25.09%, avg=1575.33, stdev=67.14, samples=6
   iops        : min=  364, max=  410, avg=393.83, stdev=16.79, samples=6
  write: IOPS=405, BW=1621KiB/s (1660kB/s)(5180KiB/3196msec)
    slat (usec): min=14, max=1022, avg=48.17, stdev=44.52
    clat (usec): min=6462, max=30944, avg=19456.99, stdev=2538.28
     lat (usec): min=6517, max=30985, avg=19506.25, stdev=2536.65
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15926], 10.00th=[16712], 20.00th=[17695],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20579], 80.00th=[21103], 90.00th=[22414], 95.00th=[23725],
     | 99.00th=[26870], 99.50th=[27657], 99.90th=[29754], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1520, max= 1760, per=24.67%, avg=1612.67, stdev=93.10, samples=6
   iops        : min=  380, max=  440, avg=403.17, stdev=23.28, samples=6
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.47%, 20=62.38%, 50=37.07%
  cpu          : usr=0.75%, sys=6.76%, ctx=2312, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6279KiB/s (6430kB/s), 1560KiB/s-1583KiB/s (1598kB/s-1621kB/s), io=19.6MiB (20.5MB), run=3169-3196msec
  WRITE: bw=6537KiB/s (6694kB/s), 1621KiB/s-1659KiB/s (1660kB/s-1698kB/s), io=20.4MiB (21.4MB), run=3169-3196msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=5067: Thu Jun  5 07:46:39 2025
   read: IOPS=11, BW=471MiB/s (493MB/s)(40.0MiB/85msec)
    slat (nsec): min=6858.4k, max=6858.4k, avg=6858441.00, stdev= 0.00
    clat (nsec): min=73212k, max=73212k, avg=73212164.00, stdev= 0.00
     lat (nsec): min=80097k, max=80097k, avg=80096912.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[72877],  5.00th=[72877], 10.00th=[72877], 20.00th=[72877],
     | 30.00th=[72877], 40.00th=[72877], 50.00th=[72877], 60.00th=[72877],
     | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877],
     | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.33%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=471MiB/s (493MB/s), 471MiB/s-471MiB/s (493MB/s-493MB/s), io=40.0MiB (41.9MB), run=85-85msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout