== sanity test 398c: run fio to test AIO ================= 02:04:14 (1721887454)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.127871 s, 328 MB/s
osc.lustre-OST0000-osc-ffff8800a83ef000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=7892: Thu Jul 25 02:04:23 2024
  write: IOPS=362, BW=1452KiB/s (1487kB/s)(10.0MiB/7053msec)
    slat (usec): min=16, max=362, avg=90.86, stdev=44.56
    clat (usec): min=19797, max=74589, avg=43774.06, stdev=5065.83
     lat (usec): min=20000, max=74685, avg=43867.07, stdev=5070.10
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[34341], 10.00th=[36963], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44827], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[48497], 95.00th=[49546],
     | 99.00th=[54264], 99.50th=[54789], 99.90th=[69731], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1304, max= 1688, per=24.99%, avg=1449.64, stdev=108.29, samples=14
   iops        : min=  326, max=  422, avg=362.36, stdev=27.07, samples=14
  lat (msec)   : 20=0.12%, 50=95.94%, 100=3.95%
  cpu          : usr=0.55%, sys=4.04%, ctx=1453, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7893: Thu Jul 25 02:04:23 2024
  write: IOPS=362, BW=1451KiB/s (1486kB/s)(10.0MiB/7058msec)
    slat (usec): min=16, max=369, avg=91.48, stdev=43.84
    clat (usec): min=25681, max=74047, avg=43837.37, stdev=4945.36
     lat (usec): min=25766, max=74169, avg=43930.97, stdev=4949.37
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[34341], 10.00th=[36963], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44827], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[48497], 95.00th=[49546],
     | 99.00th=[53740], 99.50th=[54789], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1304, max= 1680, per=25.00%, avg=1449.71, stdev=106.19, samples=14
   iops        : min=  326, max=  420, avg=362.43, stdev=26.55, samples=14
  lat (msec)   : 50=95.94%, 100=4.06%
  cpu          : usr=0.58%, sys=4.15%, ctx=1422, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7894: Thu Jul 25 02:04:23 2024
  write: IOPS=363, BW=1456KiB/s (1491kB/s)(10.0MiB/7035msec)
    slat (usec): min=16, max=496, avg=90.61, stdev=45.91
    clat (usec): min=9809, max=78823, avg=43710.07, stdev=5387.03
     lat (usec): min=9842, max=78881, avg=43802.76, stdev=5385.16
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[33817], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44827], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[48497], 95.00th=[49546],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[73925], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1325, max= 1760, per=25.07%, avg=1454.07, stdev=122.14, samples=14
   iops        : min=  331, max=  440, avg=363.50, stdev=30.56, samples=14
  lat (msec)   : 10=0.08%, 20=0.39%, 50=95.47%, 100=4.06%
  cpu          : usr=0.75%, sys=3.78%, ctx=1337, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7895: Thu Jul 25 02:04:23 2024
  write: IOPS=362, BW=1450KiB/s (1485kB/s)(10.0MiB/7062msec)
    slat (usec): min=13, max=310, avg=77.45, stdev=41.91
    clat (usec): min=6467, max=73423, avg=43808.19, stdev=4713.36
     lat (usec): min=6628, max=73543, avg=43887.68, stdev=4719.71
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[33817], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[43254], 50.00th=[44827], 60.00th=[45876],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[48497], 95.00th=[49546],
     | 99.00th=[53216], 99.50th=[53740], 99.90th=[54789], 99.95th=[69731],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1288, max= 1664, per=24.94%, avg=1446.29, stdev=108.62, samples=14
   iops        : min=  322, max=  416, avg=361.57, stdev=27.15, samples=14
  lat (msec)   : 10=0.04%, 50=96.95%, 100=3.01%
  cpu          : usr=0.55%, sys=3.34%, ctx=742, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5800KiB/s (5939kB/s), 1450KiB/s-1456KiB/s (1485kB/s-1491kB/s), io=40.0MiB (41.9MB), run=7035-7062msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=7911: Thu Jul 25 02:04:28 2024
   read: IOPS=322, BW=1288KiB/s (1319kB/s)(5048KiB/3918msec)
    slat (usec): min=955, max=3250, avg=1624.34, stdev=338.03
    clat (usec): min=5030, max=39897, avg=22988.33, stdev=4262.54
     lat (usec): min=6616, max=41978, avg=24616.11, stdev=4381.55
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[17171], 10.00th=[18220], 20.00th=[19530],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22676], 60.00th=[23725],
     | 70.00th=[24773], 80.00th=[26084], 90.00th=[28443], 95.00th=[31065],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[38536], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min=  880, max= 1448, per=24.42%, avg=1251.00, stdev=183.74, samples=7
   iops        : min=  220, max=  362, avg=312.71, stdev=45.92, samples=7
  write: IOPS=331, BW=1325KiB/s (1357kB/s)(5192KiB/3918msec)
    slat (usec): min=15, max=1719, avg=82.28, stdev=81.62
    clat (usec): min=4950, max=37293, avg=23726.51, stdev=3489.30
     lat (usec): min=5014, max=37352, avg=23810.51, stdev=3489.77
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[19006], 10.00th=[20055], 20.00th=[21103],
     | 30.00th=[21890], 40.00th=[22676], 50.00th=[23200], 60.00th=[23987],
     | 70.00th=[24773], 80.00th=[26084], 90.00th=[28181], 95.00th=[30540],
     | 99.00th=[34866], 99.50th=[35390], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min=  904, max= 1480, per=24.56%, avg=1309.29, stdev=190.19, samples=7
   iops        : min=  226, max=  370, avg=327.29, stdev=47.54, samples=7
  lat (msec)   : 10=0.23%, 20=16.80%, 50=82.97%
  cpu          : usr=1.25%, sys=9.73%, ctx=2236, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7912: Thu Jul 25 02:04:28 2024
   read: IOPS=317, BW=1271KiB/s (1302kB/s)(4976KiB/3914msec)
    slat (usec): min=832, max=3758, avg=1615.77, stdev=337.89
    clat (usec): min=2355, max=45436, avg=23015.12, stdev=4324.11
     lat (usec): min=3811, max=47228, avg=24634.14, stdev=4451.70
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[17171], 10.00th=[18220], 20.00th=[19530],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[24511], 80.00th=[26084], 90.00th=[28443], 95.00th=[30540],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[40633], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min=  872, max= 1400, per=24.19%, avg=1238.86, stdev=180.10, samples=7
   iops        : min=  218, max=  350, avg=309.71, stdev=45.02, samples=7
  write: IOPS=336, BW=1345KiB/s (1377kB/s)(5264KiB/3914msec)
    slat (usec): min=14, max=299, avg=78.20, stdev=38.87
    clat (usec): min=5718, max=42476, avg=23888.22, stdev=3764.98
     lat (usec): min=5787, max=42514, avg=23968.22, stdev=3765.71
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[19268], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[21890], 40.00th=[22676], 50.00th=[23200], 60.00th=[23987],
     | 70.00th=[25035], 80.00th=[26346], 90.00th=[28705], 95.00th=[31589],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[40633], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  976, max= 1456, per=24.99%, avg=1332.57, stdev=171.94, samples=7
   iops        : min=  244, max=  364, avg=333.14, stdev=42.99, samples=7
  lat (msec)   : 4=0.04%, 10=0.08%, 20=17.03%, 50=82.85%
  cpu          : usr=0.92%, sys=9.86%, ctx=2202, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7913: Thu Jul 25 02:04:28 2024
   read: IOPS=318, BW=1275KiB/s (1306kB/s)(4984KiB/3908msec)
    slat (usec): min=727, max=3328, avg=1617.58, stdev=335.74
    clat (usec): min=5471, max=42257, avg=22987.56, stdev=4165.18
     lat (usec): min=7184, max=44729, avg=24608.20, stdev=4289.38
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[17433], 10.00th=[18220], 20.00th=[19530],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22676], 60.00th=[23462],
     | 70.00th=[24511], 80.00th=[25822], 90.00th=[28443], 95.00th=[30802],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[41157], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min=  888, max= 1392, per=24.54%, avg=1257.14, stdev=170.98, samples=7
   iops        : min=  222, max=  348, avg=314.29, stdev=42.75, samples=7
  write: IOPS=336, BW=1345KiB/s (1377kB/s)(5256KiB/3908msec)
    slat (usec): min=17, max=452, avg=80.43, stdev=41.63
    clat (usec): min=8528, max=40691, avg=23742.29, stdev=3695.31
     lat (usec): min=8589, max=40760, avg=23824.51, stdev=3697.35
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[18744], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[21890], 40.00th=[22414], 50.00th=[23200], 60.00th=[23987],
     | 70.00th=[25035], 80.00th=[26084], 90.00th=[28181], 95.00th=[30802],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min=  936, max= 1544, per=24.84%, avg=1324.57, stdev=207.68, samples=7
   iops        : min=  234, max=  386, avg=331.14, stdev=51.92, samples=7
  lat (msec)   : 10=0.12%, 20=17.97%, 50=81.91%
  cpu          : usr=0.97%, sys=10.08%, ctx=2229, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7914: Thu Jul 25 02:04:28 2024
   read: IOPS=325, BW=1300KiB/s (1331kB/s)(5060KiB/3892msec)
    slat (usec): min=24, max=4261, avg=1611.62, stdev=380.54
    clat (usec): min=2844, max=40530, avg=23100.43, stdev=4385.04
     lat (usec): min=2902, max=42789, avg=24715.12, stdev=4538.00
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[17433], 10.00th=[18482], 20.00th=[19530],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22676], 60.00th=[23725],
     | 70.00th=[25035], 80.00th=[26084], 90.00th=[28443], 95.00th=[30802],
     | 99.00th=[36439], 99.50th=[38011], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min=  984, max= 1440, per=25.10%, avg=1285.71, stdev=165.04, samples=7
   iops        : min=  246, max=  360, avg=321.43, stdev=41.26, samples=7
  write: IOPS=332, BW=1331KiB/s (1363kB/s)(5180KiB/3892msec)
    slat (usec): min=17, max=388, avg=78.78, stdev=39.53
    clat (usec): min=11643, max=40131, avg=23730.08, stdev=3549.36
     lat (usec): min=11774, max=40247, avg=23810.78, stdev=3551.54
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[19006], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[21890], 40.00th=[22676], 50.00th=[23200], 60.00th=[23987],
     | 70.00th=[25035], 80.00th=[26084], 90.00th=[28181], 95.00th=[30540],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1000, max= 1408, per=24.52%, avg=1307.43, stdev=141.67, samples=7
   iops        : min=  250, max=  352, avg=326.86, stdev=35.42, samples=7
  lat (msec)   : 4=0.12%, 10=0.08%, 20=17.27%, 50=82.54%
  cpu          : usr=1.82%, sys=9.05%, ctx=2235, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5122KiB/s (5245kB/s), 1271KiB/s-1300KiB/s (1302kB/s-1331kB/s), io=19.6MiB (20.5MB), run=3892-3918msec
  WRITE: bw=5332KiB/s (5460kB/s), 1325KiB/s-1345KiB/s (1357kB/s-1377kB/s), io=20.4MiB (21.4MB), run=3892-3918msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=7920: Thu Jul 25 02:04:29 2024
   read: IOPS=6, BW=250MiB/s (262MB/s)(40.0MiB/160msec)
    slat (nsec): min=5885.1k, max=5885.1k, avg=5885070.00, stdev= 0.00
    clat (nsec): min=146201k, max=146201k, avg=146200807.00, stdev= 0.00
     lat (nsec): min=152096k, max=152096k, avg=152095877.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  146],  5.00th=[  146], 10.00th=[  146], 20.00th=[  146],
     | 30.00th=[  146], 40.00th=[  146], 50.00th=[  146], 60.00th=[  146],
     | 70.00th=[  146], 80.00th=[  146], 90.00th=[  146], 95.00th=[  146],
     | 99.00th=[  146], 99.50th=[  146], 99.90th=[  146], 99.95th=[  146],
     | 99.99th=[  146]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.77%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=250MiB/s (262MB/s), 250MiB/s-250MiB/s (262MB/s-262MB/s), io=40.0MiB (41.9MB), run=160-160msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout