== sanity test 398c: run fio to test AIO ================= 15:03:30 (1731355410)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.255663 s, 164 MB/s
osc.lustre-OST0000-osc-ffff8800b6ffb000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=17292: Mon Nov 11 15:03:38 2024
  write: IOPS=388, BW=1552KiB/s (1590kB/s)(10.0MiB/6596msec)
    slat (usec): min=14, max=743, avg=87.61, stdev=53.48
    clat (usec): min=9412, max=79209, avg=40966.36, stdev=8901.18
     lat (usec): min=9529, max=79279, avg=41056.28, stdev=8915.43
    clat percentiles (usec):
     |  1.00th=[17695],  5.00th=[20055], 10.00th=[23725], 20.00th=[36439],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[50594], 99.50th=[57934], 99.90th=[73925], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1352, max= 2416, per=25.00%, avg=1552.15, stdev=294.56, samples=13
   iops        : min=  338, max=  604, avg=388.00, stdev=73.65, samples=13
  lat (msec)   : 10=0.04%, 20=4.53%, 50=93.91%, 100=1.52%
  cpu          : usr=0.62%, sys=3.88%, ctx=1572, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17293: Mon Nov 11 15:03:38 2024
  write: IOPS=388, BW=1553KiB/s (1591kB/s)(10.0MiB/6592msec)
    slat (usec): min=15, max=726, avg=81.07, stdev=51.25
    clat (usec): min=4249, max=85090, avg=40904.82, stdev=8955.41
     lat (usec): min=4312, max=85148, avg=40987.79, stdev=8970.24
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[20317], 10.00th=[23462], 20.00th=[36963],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[43779], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[50594], 99.50th=[58459], 99.90th=[74974], 99.95th=[80217],
     | 99.99th=[85459]
   bw (  KiB/s): min= 1344, max= 2424, per=24.98%, avg=1551.15, stdev=293.29, samples=13
   iops        : min=  336, max=  606, avg=387.77, stdev=73.33, samples=13
  lat (msec)   : 10=0.12%, 20=4.49%, 50=94.34%, 100=1.05%
  cpu          : usr=0.49%, sys=3.76%, ctx=1251, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17294: Mon Nov 11 15:03:38 2024
  write: IOPS=388, BW=1554KiB/s (1592kB/s)(10.0MiB/6588msec)
    slat (usec): min=14, max=841, avg=83.27, stdev=51.12
    clat (usec): min=15440, max=80785, avg=40954.39, stdev=8711.63
     lat (usec): min=15491, max=80861, avg=41039.69, stdev=8725.15
    clat percentiles (usec):
     |  1.00th=[17695],  5.00th=[20317], 10.00th=[24249], 20.00th=[36439],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[43779], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[51119], 99.50th=[55837], 99.90th=[60556], 99.95th=[70779],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1344, max= 2384, per=25.04%, avg=1554.54, stdev=285.04, samples=13
   iops        : min=  336, max=  596, avg=388.62, stdev=71.27, samples=13
  lat (msec)   : 20=4.18%, 50=94.14%, 100=1.68%
  cpu          : usr=0.49%, sys=3.66%, ctx=1473, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17295: Mon Nov 11 15:03:38 2024
  write: IOPS=391, BW=1565KiB/s (1602kB/s)(10.0MiB/6545msec)
    slat (usec): min=14, max=793, avg=87.14, stdev=55.03
    clat (usec): min=5634, max=71319, avg=40695.33, stdev=9212.30
     lat (usec): min=5978, max=71438, avg=40784.62, stdev=9220.97
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[19792], 10.00th=[23200], 20.00th=[35914],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[43779], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1360, max= 2424, per=25.21%, avg=1565.08, stdev=299.77, samples=13
   iops        : min=  340, max=  606, avg=391.23, stdev=74.96, samples=13
  lat (msec)   : 10=0.55%, 20=4.69%, 50=93.28%, 100=1.48%
  cpu          : usr=0.75%, sys=3.73%, ctx=1479, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6210KiB/s (6359kB/s), 1552KiB/s-1565KiB/s (1590kB/s-1602kB/s), io=40.0MiB (41.9MB), run=6545-6596msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17310: Mon Nov 11 15:03:42 2024
   read: IOPS=418, BW=1674KiB/s (1714kB/s)(5048KiB/3016msec)
    slat (usec): min=627, max=3017, avg=1227.69, stdev=343.87
    clat (usec): min=1652, max=34008, avg=17864.17, stdev=4004.50
     lat (usec): min=2519, max=35573, avg=19093.90, stdev=4188.93
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12780], 10.00th=[13698], 20.00th=[14746],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[19006], 80.00th=[20317], 90.00th=[23462], 95.00th=[26084],
     | 99.00th=[30540], 99.50th=[31851], 99.90th=[32900], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1280, max= 1960, per=25.13%, avg=1670.67, stdev=273.38, samples=6
   iops        : min=  320, max=  490, avg=417.67, stdev=68.35, samples=6
  write: IOPS=430, BW=1721KiB/s (1763kB/s)(5192KiB/3016msec)
    slat (usec): min=13, max=1678, avg=53.60, stdev=59.13
    clat (usec): min=3430, max=33199, avg=18069.96, stdev=3816.12
     lat (usec): min=3499, max=33249, avg=18124.78, stdev=3825.93
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13829], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18744], 80.00th=[20317], 90.00th=[24249], 95.00th=[26608],
     | 99.00th=[30278], 99.50th=[31327], 99.90th=[32637], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1296, max= 2032, per=24.89%, avg=1722.67, stdev=331.31, samples=6
   iops        : min=  324, max=  508, avg=430.67, stdev=82.83, samples=6
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.08%, 20=78.05%, 50=21.80%
  cpu          : usr=0.96%, sys=7.43%, ctx=2848, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17311: Mon Nov 11 15:03:42 2024
   read: IOPS=416, BW=1664KiB/s (1704kB/s)(4976KiB/2990msec)
    slat (usec): min=643, max=2831, avg=1222.92, stdev=335.93
    clat (usec): min=2645, max=42957, avg=17693.48, stdev=4156.21
     lat (usec): min=3915, max=44491, avg=18918.55, stdev=4333.34
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12780], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[16188], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[22676], 95.00th=[26084],
     | 99.00th=[31851], 99.50th=[34341], 99.90th=[41681], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1232, max= 1976, per=25.88%, avg=1720.00, stdev=284.48, samples=5
   iops        : min=  308, max=  494, avg=430.00, stdev=71.12, samples=5
  write: IOPS=440, BW=1761KiB/s (1803kB/s)(5264KiB/2990msec)
    slat (usec): min=16, max=317, avg=52.66, stdev=34.77
    clat (usec): min=8382, max=42917, avg=18088.34, stdev=3830.20
     lat (usec): min=8420, max=42949, avg=18142.15, stdev=3836.60
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13698], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[19006], 80.00th=[20317], 90.00th=[22938], 95.00th=[26608],
     | 99.00th=[30016], 99.50th=[32900], 99.90th=[40109], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1528, max= 2016, per=26.75%, avg=1851.20, stdev=196.33, samples=5
   iops        : min=  382, max=  504, avg=462.80, stdev=49.08, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=79.77%, 50=20.12%
  cpu          : usr=0.77%, sys=7.49%, ctx=2875, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17312: Mon Nov 11 15:03:42 2024
   read: IOPS=419, BW=1676KiB/s (1717kB/s)(4984KiB/2973msec)
    slat (usec): min=26, max=4128, avg=1231.73, stdev=351.28
    clat (usec): min=3806, max=53949, avg=17564.24, stdev=4268.48
     lat (usec): min=3851, max=55536, avg=18797.91, stdev=4444.83
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12649], 10.00th=[13566], 20.00th=[14615],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[22676], 95.00th=[25822],
     | 99.00th=[31589], 99.50th=[33162], 99.90th=[51643], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min= 1416, max= 1920, per=26.29%, avg=1747.20, stdev=207.74, samples=5
   iops        : min=  354, max=  480, avg=436.80, stdev=51.93, samples=5
  write: IOPS=441, BW=1768KiB/s (1810kB/s)(5256KiB/2973msec)
    slat (usec): min=14, max=1115, avg=52.31, stdev=45.91
    clat (usec): min=10556, max=51400, avg=18179.59, stdev=4019.80
     lat (usec): min=10645, max=51441, avg=18233.02, stdev=4026.23
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13960], 10.00th=[14615], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18744], 80.00th=[20317], 90.00th=[23725], 95.00th=[26608],
     | 99.00th=[30802], 99.50th=[32900], 99.90th=[44827], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min= 1432, max= 2136, per=26.66%, avg=1844.80, stdev=263.67, samples=5
   iops        : min=  358, max=  534, avg=461.20, stdev=65.92, samples=5
  lat (msec)   : 4=0.08%, 10=0.20%, 20=79.49%, 50=20.12%, 100=0.12%
  cpu          : usr=0.67%, sys=7.47%, ctx=2934, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17313: Mon Nov 11 15:03:42 2024
   read: IOPS=419, BW=1676KiB/s (1716kB/s)(5060KiB/3019msec)
    slat (usec): min=661, max=3394, avg=1238.00, stdev=333.40
    clat (usec): min=1927, max=34159, avg=17705.83, stdev=3972.64
     lat (usec): min=3301, max=35659, avg=18945.71, stdev=4143.97
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12911], 10.00th=[13698], 20.00th=[14746],
     | 30.00th=[15533], 40.00th=[16188], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18744], 80.00th=[20055], 90.00th=[22938], 95.00th=[25822],
     | 99.00th=[31327], 99.50th=[32113], 99.90th=[33817], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1264, max= 1936, per=25.17%, avg=1673.33, stdev=274.64, samples=6
   iops        : min=  316, max=  484, avg=418.33, stdev=68.66, samples=6
  write: IOPS=428, BW=1716KiB/s (1757kB/s)(5180KiB/3019msec)
    slat (usec): min=13, max=395, avg=51.77, stdev=33.96
    clat (usec): min=9954, max=34551, avg=18218.64, stdev=3830.27
     lat (usec): min=9982, max=34599, avg=18271.53, stdev=3840.23
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13829], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[19006], 80.00th=[20579], 90.00th=[23987], 95.00th=[26608],
     | 99.00th=[30540], 99.50th=[31851], 99.90th=[33817], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1272, max= 2040, per=24.66%, avg=1706.67, stdev=331.58, samples=6
   iops        : min=  318, max=  510, avg=426.67, stdev=82.89, samples=6
  lat (msec)   : 2=0.08%, 10=0.16%, 20=77.97%, 50=21.80%
  cpu          : usr=1.03%, sys=7.26%, ctx=2902, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6647KiB/s (6807kB/s), 1664KiB/s-1676KiB/s (1704kB/s-1717kB/s), io=19.6MiB (20.5MB), run=2973-3019msec
  WRITE: bw=6920KiB/s (7086kB/s), 1716KiB/s-1768KiB/s (1757kB/s-1810kB/s), io=20.4MiB (21.4MB), run=2973-3019msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=17319: Mon Nov 11 15:03:43 2024
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=5534.7k, max=5534.7k, avg=5534737.00, stdev= 0.00
    clat (nsec): min=63987k, max=63987k, avg=63986983.00, stdev= 0.00
     lat (nsec): min=69531k, max=69531k, avg=69531277.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[64226],  5.00th=[64226], 10.00th=[64226], 20.00th=[64226],
     | 30.00th=[64226], 40.00th=[64226], 50.00th=[64226], 60.00th=[64226],
     | 70.00th=[64226], 80.00th=[64226], 90.00th=[64226], 95.00th=[64226],
     | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64226]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.00%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout