== sanity test 398c: run fio to test AIO ================= 07:59:43 (1715342383)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.152492 s, 275 MB/s
osc.lustre-OST0000-osc-ffff88009b116000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18237: Fri May 10 07:59:52 2024
  write: IOPS=362, BW=1449KiB/s (1484kB/s)(10.0MiB/7068msec)
    slat (usec): min=26, max=597, avg=96.58, stdev=52.98
    clat (usec): min=10965, max=79728, avg=43930.43, stdev=5258.43
     lat (usec): min=11140, max=79768, avg=44029.49, stdev=5259.51
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[36439], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[55837], 99.50th=[55837], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1360, max= 1648, per=24.99%, avg=1448.14, stdev=72.96, samples=14
   iops        : min=  340, max=  412, avg=362.00, stdev=18.26, samples=14
  lat (msec)   : 20=0.23%, 50=93.55%, 100=6.21%
  cpu          : usr=0.58%, sys=3.89%, ctx=1880, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18238: Fri May 10 07:59:52 2024
  write: IOPS=363, BW=1455KiB/s (1490kB/s)(10.0MiB/7038msec)
    slat (usec): min=22, max=655, avg=100.70, stdev=56.18
    clat (usec): min=6176, max=72495, avg=43754.53, stdev=5434.25
     lat (usec): min=6289, max=72592, avg=43857.54, stdev=5436.96
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[36439], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[49021], 95.00th=[50594],
     | 99.00th=[55313], 99.50th=[55837], 99.90th=[62653], 99.95th=[67634],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1384, max= 1704, per=25.08%, avg=1453.50, stdev=82.78, samples=14
   iops        : min=  346, max=  426, avg=363.36, stdev=20.70, samples=14
  lat (msec)   : 10=0.12%, 20=0.66%, 50=92.66%, 100=6.56%
  cpu          : usr=0.67%, sys=3.99%, ctx=1921, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18239: Fri May 10 07:59:52 2024
  write: IOPS=362, BW=1450KiB/s (1485kB/s)(10.0MiB/7061msec)
    slat (usec): min=25, max=855, avg=92.12, stdev=53.68
    clat (usec): min=4433, max=72380, avg=43881.37, stdev=5184.02
     lat (usec): min=4549, max=72499, avg=43975.82, stdev=5185.88
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[36439], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[55837], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1352, max= 1632, per=25.01%, avg=1449.50, stdev=71.47, samples=14
   iops        : min=  338, max=  408, avg=362.36, stdev=17.88, samples=14
  lat (msec)   : 10=0.16%, 50=93.83%, 100=6.02%
  cpu          : usr=0.61%, sys=3.74%, ctx=1653, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18240: Fri May 10 07:59:52 2024
  write: IOPS=362, BW=1449KiB/s (1484kB/s)(10.0MiB/7065msec)
    slat (usec): min=26, max=676, avg=93.79, stdev=50.00
    clat (usec): min=5603, max=76864, avg=43909.07, stdev=5116.91
     lat (usec): min=5745, max=77169, avg=44005.13, stdev=5119.57
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[36439], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[46400], 80.00th=[47449], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[55837], 99.90th=[71828], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1360, max= 1624, per=24.98%, avg=1447.79, stdev=67.37, samples=14
   iops        : min=  340, max=  406, avg=361.93, stdev=16.85, samples=14
  lat (msec)   : 10=0.08%, 20=0.04%, 50=93.67%, 100=6.21%
  cpu          : usr=0.61%, sys=3.78%, ctx=1803, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5795KiB/s (5934kB/s), 1449KiB/s-1455KiB/s (1484kB/s-1490kB/s), io=40.0MiB (41.9MB), run=7038-7068msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18256: Fri May 10 07:59:56 2024
   read: IOPS=422, BW=1690KiB/s (1731kB/s)(5048KiB/2987msec)
    slat (usec): min=14, max=3491, avg=1242.53, stdev=404.46
    clat (usec): min=4760, max=40656, avg=17619.81, stdev=4381.81
     lat (usec): min=5663, max=43327, avg=18864.14, stdev=4611.10
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12780], 10.00th=[13566], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[22414], 95.00th=[27132],
     | 99.00th=[36439], 99.50th=[39060], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1232, max= 1928, per=24.84%, avg=1668.80, stdev=269.49, samples=5
   iops        : min=  308, max=  482, avg=417.20, stdev=67.37, samples=5
  write: IOPS=434, BW=1738KiB/s (1780kB/s)(5192KiB/2987msec)
    slat (usec): min=14, max=276, avg=45.29, stdev=25.97
    clat (usec): min=8962, max=46448, avg=18404.77, stdev=4316.59
     lat (usec): min=8999, max=46520, avg=18451.18, stdev=4325.44
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13829], 10.00th=[14615], 20.00th=[15533],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[22676], 95.00th=[27919],
     | 99.00th=[34866], 99.50th=[37487], 99.90th=[43254], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min= 1200, max= 2112, per=25.14%, avg=1758.40, stdev=356.09, samples=5
   iops        : min=  300, max=  528, avg=439.60, stdev=89.02, samples=5
  lat (msec)   : 10=0.20%, 20=81.02%, 50=18.79%
  cpu          : usr=0.90%, sys=6.33%, ctx=2977, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18257: Fri May 10 07:59:56 2024
   read: IOPS=416, BW=1666KiB/s (1706kB/s)(4976KiB/2986msec)
    slat (usec): min=13, max=3704, avg=1253.14, stdev=405.02
    clat (usec): min=2933, max=43257, avg=17610.16, stdev=4501.16
     lat (usec): min=3979, max=45742, avg=18865.16, stdev=4720.35
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12911], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21890], 95.00th=[27132],
     | 99.00th=[36439], 99.50th=[39584], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1112, max= 1896, per=24.63%, avg=1654.40, stdev=318.01, samples=5
   iops        : min=  278, max=  474, avg=413.60, stdev=79.50, samples=5
  write: IOPS=440, BW=1763KiB/s (1805kB/s)(5264KiB/2986msec)
    slat (usec): min=14, max=236, avg=45.50, stdev=25.99
    clat (usec): min=9062, max=41603, avg=18386.94, stdev=4311.83
     lat (usec): min=9091, max=41655, avg=18433.53, stdev=4319.85
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[14091], 10.00th=[14746], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[22152], 95.00th=[28181],
     | 99.00th=[36439], 99.50th=[39060], 99.90th=[40633], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1344, max= 2072, per=25.42%, avg=1777.60, stdev=283.03, samples=5
   iops        : min=  336, max=  518, avg=444.40, stdev=70.76, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=81.02%, 50=18.79%
  cpu          : usr=0.67%, sys=6.50%, ctx=2937, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18258: Fri May 10 07:59:56 2024
   read: IOPS=419, BW=1676KiB/s (1717kB/s)(4984KiB/2973msec)
    slat (usec): min=24, max=3560, avg=1239.89, stdev=404.80
    clat (usec): min=4536, max=45988, avg=17409.71, stdev=4300.22
     lat (usec): min=5582, max=49094, avg=18651.42, stdev=4529.70
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12649], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21627], 95.00th=[25822],
     | 99.00th=[34866], 99.50th=[36963], 99.90th=[43254], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min= 1240, max= 2000, per=24.89%, avg=1672.00, stdev=315.57, samples=5
   iops        : min=  310, max=  500, avg=418.00, stdev=78.89, samples=5
  write: IOPS=441, BW=1768KiB/s (1810kB/s)(5256KiB/2973msec)
    slat (usec): min=14, max=320, avg=44.84, stdev=24.68
    clat (usec): min=9039, max=40502, avg=18434.35, stdev=4386.62
     lat (usec): min=9114, max=40567, avg=18480.23, stdev=4396.08
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13829], 10.00th=[14615], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[20055], 90.00th=[22414], 95.00th=[29230],
     | 99.00th=[35914], 99.50th=[38011], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1184, max= 2056, per=25.32%, avg=1771.20, stdev=346.95, samples=5
   iops        : min=  296, max=  514, avg=442.80, stdev=86.74, samples=5
  lat (msec)   : 10=0.12%, 20=81.95%, 50=17.93%
  cpu          : usr=0.71%, sys=6.59%, ctx=2938, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18259: Fri May 10 07:59:56 2024
   read: IOPS=423, BW=1696KiB/s (1736kB/s)(5060KiB/2984msec)
    slat (usec): min=14, max=3685, avg=1249.04, stdev=391.36
    clat (usec): min=9417, max=48779, avg=17531.66, stdev=4302.21
     lat (usec): min=10727, max=51869, avg=18782.47, stdev=4514.33
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12780], 10.00th=[13435], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21890], 95.00th=[26346],
     | 99.00th=[33817], 99.50th=[37487], 99.90th=[42206], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1208, max= 1920, per=25.08%, avg=1684.80, stdev=292.66, samples=5
   iops        : min=  302, max=  480, avg=421.20, stdev=73.17, samples=5
  write: IOPS=433, BW=1736KiB/s (1778kB/s)(5180KiB/2984msec)
    slat (usec): min=13, max=1072, avg=45.01, stdev=39.67
    clat (usec): min=9029, max=43269, avg=18440.67, stdev=4561.92
     lat (usec): min=9108, max=43309, avg=18486.75, stdev=4572.21
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13829], 10.00th=[14615], 20.00th=[15533],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19006], 80.00th=[20055], 90.00th=[22152], 95.00th=[28443],
     | 99.00th=[37487], 99.50th=[40109], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1208, max= 2032, per=24.89%, avg=1740.80, stdev=333.83, samples=5
   iops        : min=  302, max=  508, avg=435.20, stdev=83.46, samples=5
  lat (msec)   : 10=0.27%, 20=81.41%, 50=18.32%
  cpu          : usr=0.94%, sys=6.30%, ctx=2781, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6718KiB/s (6880kB/s), 1666KiB/s-1696KiB/s (1706kB/s-1736kB/s), io=19.6MiB (20.5MB), run=2973-2987msec
  WRITE: bw=6994KiB/s (7162kB/s), 1736KiB/s-1768KiB/s (1778kB/s-1810kB/s), io=20.4MiB (21.4MB), run=2973-2987msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18265: Fri May 10 07:59:57 2024
   read: IOPS=8, BW=348MiB/s (365MB/s)(40.0MiB/115msec)
    slat (nsec): min=5851.1k, max=5851.1k, avg=5851116.00, stdev= 0.00
    clat (nsec): min=108537k, max=108537k, avg=108537470.00, stdev= 0.00
     lat (nsec): min=114408k, max=114408k, avg=114408434.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  109],  5.00th=[  109], 10.00th=[  109], 20.00th=[  109],
     | 30.00th=[  109], 40.00th=[  109], 50.00th=[  109], 60.00th=[  109],
     | 70.00th=[  109], 80.00th=[  109], 90.00th=[  109], 95.00th=[  109],
     | 99.00th=[  109], 99.50th=[  109], 99.90th=[  109], 99.95th=[  109],
     | 99.99th=[  109]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.26%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=40.0MiB (41.9MB), run=115-115msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout