== sanity test 398c: run fio to test AIO ================= 10:56:12 (1724770572)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.145033 s, 289 MB/s
osc.lustre-OST0000-osc-ffff8800a821a000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=9387: Tue Aug 27 10:56:20 2024
  write: IOPS=371, BW=1485KiB/s (1521kB/s)(10.0MiB/6896msec)
    slat (usec): min=16, max=443, avg=89.13, stdev=49.11
    clat (usec): min=4686, max=89502, avg=42826.35, stdev=6808.46
     lat (usec): min=4810, max=89562, avg=42917.61, stdev=6807.21
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[29754], 10.00th=[32900], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[48497], 95.00th=[50594],
     | 99.00th=[55313], 99.50th=[61080], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[89654]
   bw (  KiB/s): min= 1304, max= 1912, per=25.18%, avg=1489.54, stdev=166.32, samples=13
   iops        : min=  326, max=  478, avg=372.38, stdev=41.58, samples=13
  lat (msec)   : 10=0.20%, 20=0.31%, 50=92.66%, 100=6.84%
  cpu          : usr=0.52%, sys=3.79%, ctx=1688, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9388: Tue Aug 27 10:56:20 2024
  write: IOPS=370, BW=1481KiB/s (1517kB/s)(10.0MiB/6914msec)
    slat (usec): min=19, max=410, avg=82.43, stdev=46.13
    clat (usec): min=22570, max=76400, avg=42895.94, stdev=6117.28
     lat (usec): min=22655, max=76514, avg=42980.22, stdev=6121.14
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[30540], 10.00th=[33817], 20.00th=[40633],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[54789], 99.50th=[55313], 99.90th=[61604], 99.95th=[71828],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1312, max= 1792, per=25.06%, avg=1482.46, stdev=154.35, samples=13
   iops        : min=  328, max=  448, avg=370.62, stdev=38.59, samples=13
  lat (msec)   : 50=93.63%, 100=6.37%
  cpu          : usr=0.56%, sys=3.59%, ctx=1419, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9389: Tue Aug 27 10:56:20 2024
  write: IOPS=369, BW=1479KiB/s (1514kB/s)(10.0MiB/6924msec)
    slat (usec): min=19, max=794, avg=85.78, stdev=46.59
    clat (usec): min=7079, max=72275, avg=42910.82, stdev=6112.50
     lat (usec): min=7174, max=72333, avg=42998.57, stdev=6119.06
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[30278], 10.00th=[33817], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[54264], 99.50th=[54789], 99.90th=[67634], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1288, max= 1792, per=25.03%, avg=1480.38, stdev=140.24, samples=13
   iops        : min=  322, max=  448, avg=370.08, stdev=35.07, samples=13
  lat (msec)   : 10=0.04%, 50=93.20%, 100=6.76%
  cpu          : usr=0.59%, sys=3.68%, ctx=1565, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9390: Tue Aug 27 10:56:20 2024
  write: IOPS=369, BW=1479KiB/s (1514kB/s)(10.0MiB/6924msec)
    slat (usec): min=14, max=773, avg=89.41, stdev=50.91
    clat (usec): min=5260, max=76557, avg=42997.08, stdev=6235.92
     lat (usec): min=5379, max=76646, avg=43088.46, stdev=6239.95
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[30802], 10.00th=[34341], 20.00th=[40633],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[54789], 99.50th=[55837], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1312, max= 1792, per=25.08%, avg=1483.69, stdev=142.05, samples=13
   iops        : min=  328, max=  448, avg=370.92, stdev=35.51, samples=13
  lat (msec)   : 10=0.08%, 20=0.04%, 50=93.40%, 100=6.48%
  cpu          : usr=0.52%, sys=4.04%, ctx=1727, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5916KiB/s (6058kB/s), 1479KiB/s-1485KiB/s (1514kB/s-1521kB/s), io=40.0MiB (41.9MB), run=6896-6924msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=9405: Tue Aug 27 10:56:24 2024
   read: IOPS=395, BW=1582KiB/s (1620kB/s)(5048KiB/3191msec)
    slat (usec): min=772, max=3137, avg=1400.00, stdev=382.46
    clat (usec): min=2795, max=43677, avg=18656.29, stdev=4724.45
     lat (usec): min=3847, max=45926, avg=20059.41, stdev=4947.52
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17695], 60.00th=[18482],
     | 70.00th=[19530], 80.00th=[20841], 90.00th=[24249], 95.00th=[28967],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[41157], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  952, max= 1864, per=25.29%, avg=1590.00, stdev=324.68, samples=6
   iops        : min=  238, max=  466, avg=397.50, stdev=81.17, samples=6
  write: IOPS=406, BW=1627KiB/s (1666kB/s)(5192KiB/3191msec)
    slat (usec): min=14, max=1709, avg=72.79, stdev=62.83
    clat (usec): min=12191, max=41291, avg=19269.97, stdev=4779.31
     lat (usec): min=12259, max=41375, avg=19344.36, stdev=4785.16
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14353], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[21365], 90.00th=[24511], 95.00th=[30540],
     | 99.00th=[38536], 99.50th=[39060], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  920, max= 1968, per=24.96%, avg=1634.00, stdev=406.87, samples=6
   iops        : min=  230, max=  492, avg=408.50, stdev=101.72, samples=6
  lat (msec)   : 4=0.08%, 10=0.04%, 20=73.09%, 50=26.80%
  cpu          : usr=1.44%, sys=10.09%, ctx=2374, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9406: Tue Aug 27 10:56:24 2024
   read: IOPS=393, BW=1573KiB/s (1611kB/s)(4976KiB/3163msec)
    slat (usec): min=34, max=3604, avg=1380.16, stdev=393.43
    clat (usec): min=1205, max=44974, avg=18665.68, stdev=4931.16
     lat (usec): min=2422, max=48458, avg=20048.94, stdev=5157.92
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16909], 50.00th=[17695], 60.00th=[18482],
     | 70.00th=[19530], 80.00th=[21103], 90.00th=[24249], 95.00th=[28443],
     | 99.00th=[38536], 99.50th=[41157], 99.90th=[44827], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min=  928, max= 1784, per=25.05%, avg=1575.33, stdev=339.38, samples=6
   iops        : min=  232, max=  446, avg=393.83, stdev=84.84, samples=6
  write: IOPS=416, BW=1664KiB/s (1704kB/s)(5264KiB/3163msec)
    slat (usec): min=14, max=570, avg=68.57, stdev=42.81
    clat (usec): min=11105, max=49849, avg=19316.65, stdev=4991.89
     lat (usec): min=11139, max=49897, avg=19386.68, stdev=4997.17
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14484], 10.00th=[15139], 20.00th=[16057],
     | 30.00th=[16712], 40.00th=[17433], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19792], 80.00th=[21103], 90.00th=[24511], 95.00th=[30278],
     | 99.00th=[39584], 99.50th=[41157], 99.90th=[48497], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min= 1144, max= 1888, per=25.63%, avg=1678.00, stdev=294.72, samples=6
   iops        : min=  286, max=  472, avg=419.50, stdev=73.68, samples=6
  lat (msec)   : 2=0.04%, 10=0.23%, 20=71.95%, 50=27.77%
  cpu          : usr=0.98%, sys=10.47%, ctx=2224, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9407: Tue Aug 27 10:56:24 2024
   read: IOPS=391, BW=1568KiB/s (1605kB/s)(4984KiB/3179msec)
    slat (usec): min=767, max=3337, avg=1392.87, stdev=380.22
    clat (usec): min=3471, max=43171, avg=18568.44, stdev=4649.61
     lat (usec): min=5631, max=45534, avg=19964.13, stdev=4872.81
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13566], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17433], 60.00th=[18220],
     | 70.00th=[19268], 80.00th=[20841], 90.00th=[23462], 95.00th=[28443],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  968, max= 1880, per=25.17%, avg=1582.67, stdev=332.57, samples=6
   iops        : min=  242, max=  470, avg=395.67, stdev=83.14, samples=6
  write: IOPS=413, BW=1653KiB/s (1693kB/s)(5256KiB/3179msec)
    slat (usec): min=14, max=626, avg=72.76, stdev=47.10
    clat (usec): min=12350, max=43866, avg=19373.54, stdev=4835.36
     lat (usec): min=12453, max=43906, avg=19448.06, stdev=4836.79
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14746], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19792], 80.00th=[21365], 90.00th=[24511], 95.00th=[30540],
     | 99.00th=[38011], 99.50th=[39584], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1000, max= 1952, per=25.25%, avg=1653.33, stdev=353.04, samples=6
   iops        : min=  250, max=  488, avg=413.33, stdev=88.26, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=73.55%, 50=26.37%
  cpu          : usr=1.38%, sys=10.16%, ctx=2304, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9408: Tue Aug 27 10:56:24 2024
   read: IOPS=396, BW=1586KiB/s (1624kB/s)(5060KiB/3191msec)
    slat (usec): min=760, max=3785, avg=1388.22, stdev=355.41
    clat (usec): min=9227, max=45246, avg=18651.37, stdev=4658.22
     lat (usec): min=10292, max=48031, avg=20042.90, stdev=4875.30
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13435], 10.00th=[14353], 20.00th=[15401],
     | 30.00th=[16188], 40.00th=[16909], 50.00th=[17695], 60.00th=[18482],
     | 70.00th=[19530], 80.00th=[20841], 90.00th=[23725], 95.00th=[28967],
     | 99.00th=[36963], 99.50th=[40109], 99.90th=[42730], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min=  912, max= 1816, per=25.29%, avg=1590.00, stdev=351.16, samples=6
   iops        : min=  228, max=  454, avg=397.50, stdev=87.79, samples=6
  write: IOPS=405, BW=1623KiB/s (1662kB/s)(5180KiB/3191msec)
    slat (usec): min=12, max=2107, avg=76.69, stdev=86.53
    clat (usec): min=11029, max=40552, avg=19236.16, stdev=4725.94
     lat (usec): min=11102, max=40605, avg=19314.40, stdev=4730.91
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14615], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16712], 40.00th=[17433], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19530], 80.00th=[21103], 90.00th=[23987], 95.00th=[29492],
     | 99.00th=[38011], 99.50th=[38536], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min=  968, max= 1872, per=24.94%, avg=1632.67, stdev=352.46, samples=6
   iops        : min=  242, max=  468, avg=408.17, stdev=88.11, samples=6
  lat (msec)   : 10=0.04%, 20=73.20%, 50=26.76%
  cpu          : usr=1.07%, sys=10.50%, ctx=2325, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6289KiB/s (6440kB/s), 1568KiB/s-1586KiB/s (1605kB/s-1624kB/s), io=19.6MiB (20.5MB), run=3163-3191msec
  WRITE: bw=6547KiB/s (6704kB/s), 1623KiB/s-1664KiB/s (1662kB/s-1704kB/s), io=20.4MiB (21.4MB), run=3163-3191msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=9414: Tue Aug 27 10:56:26 2024
   read: IOPS=7, BW=299MiB/s (313MB/s)(40.0MiB/134msec)
    slat (nsec): min=11072k, max=11072k, avg=11072424.00, stdev= 0.00
    clat (nsec): min=116467k, max=116467k, avg=116467162.00, stdev= 0.00
     lat (nsec): min=127559k, max=127559k, avg=127559276.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  117],  5.00th=[  117], 10.00th=[  117], 20.00th=[  117],
     | 30.00th=[  117], 40.00th=[  117], 50.00th=[  117], 60.00th=[  117],
     | 70.00th=[  117], 80.00th=[  117], 90.00th=[  117], 95.00th=[  117],
     | 99.00th=[  117], 99.50th=[  117], 99.90th=[  117], 99.95th=[  117],
     | 99.99th=[  117]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=9.02%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=299MiB/s (313MB/s), 299MiB/s-299MiB/s (313MB/s-313MB/s), io=40.0MiB (41.9MB), run=134-134msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout