== sanity test 398c: run fio to test AIO ================= 23:32:29 (1723260749)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.147048 s, 285 MB/s
osc.lustre-OST0000-osc-ffff88008a381800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=8140: Fri Aug  9 23:32:38 2024
  write: IOPS=373, BW=1494KiB/s (1530kB/s)(10.0MiB/6855msec)
    slat (usec): min=16, max=674, avg=85.78, stdev=51.37
    clat (usec): min=10818, max=76429, avg=42550.29, stdev=4208.00
     lat (usec): min=10954, max=76469, avg=42638.01, stdev=4208.58
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[35914], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[41157], 40.00th=[42730], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47449],
     | 99.00th=[49546], 99.50th=[49546], 99.90th=[72877], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1408, max= 1592, per=24.93%, avg=1489.85, stdev=51.66, samples=13
   iops        : min=  352, max=  398, avg=372.46, stdev=12.91, samples=13
  lat (msec)   : 20=0.04%, 50=99.69%, 100=0.27%
  cpu          : usr=0.74%, sys=3.56%, ctx=1370, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8141: Fri Aug  9 23:32:38 2024
  write: IOPS=373, BW=1495KiB/s (1531kB/s)(10.0MiB/6851msec)
    slat (usec): min=17, max=473, avg=78.37, stdev=46.39
    clat (usec): min=24644, max=77343, avg=42606.17, stdev=4187.92
     lat (usec): min=24749, max=77403, avg=42686.57, stdev=4189.27
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[35390], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47973],
     | 99.00th=[49546], 99.50th=[52691], 99.90th=[68682], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1413, max= 1648, per=24.98%, avg=1492.69, stdev=69.14, samples=13
   iops        : min=  353, max=  412, avg=373.15, stdev=17.31, samples=13
  lat (msec)   : 50=99.26%, 100=0.74%
  cpu          : usr=0.67%, sys=3.31%, ctx=1140, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8142: Fri Aug  9 23:32:38 2024
  write: IOPS=375, BW=1501KiB/s (1537kB/s)(10.0MiB/6821msec)
    slat (usec): min=19, max=718, avg=93.26, stdev=54.15
    clat (usec): min=3771, max=78870, avg=42436.96, stdev=4596.49
     lat (usec): min=3904, max=78915, avg=42532.45, stdev=4595.90
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[34866], 10.00th=[36439], 20.00th=[40109],
     | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47973],
     | 99.00th=[49546], 99.50th=[54264], 99.90th=[73925], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1405, max= 1600, per=25.12%, avg=1500.69, stdev=56.47, samples=13
   iops        : min=  351, max=  400, avg=375.15, stdev=14.15, samples=13
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.23%, 50=99.06%, 100=0.62%
  cpu          : usr=0.92%, sys=3.77%, ctx=1666, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8143: Fri Aug  9 23:32:38 2024
  write: IOPS=373, BW=1496KiB/s (1532kB/s)(10.0MiB/6846msec)
    slat (usec): min=17, max=696, avg=90.19, stdev=51.88
    clat (usec): min=6471, max=73090, avg=42527.92, stdev=4249.11
     lat (usec): min=6584, max=73140, avg=42620.23, stdev=4250.09
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[35390], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[43779],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47449],
     | 99.00th=[49546], 99.50th=[50070], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1416, max= 1632, per=25.01%, avg=1494.15, stdev=60.85, samples=13
   iops        : min=  354, max=  408, avg=373.54, stdev=15.21, samples=13
  lat (msec)   : 10=0.08%, 20=0.04%, 50=99.34%, 100=0.55%
  cpu          : usr=0.82%, sys=3.59%, ctx=1634, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5975KiB/s (6119kB/s), 1494KiB/s-1501KiB/s (1530kB/s-1537kB/s), io=40.0MiB (41.9MB), run=6821-6855msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=8159: Fri Aug  9 23:32:43 2024
   read: IOPS=303, BW=1215KiB/s (1244kB/s)(5048KiB/4156msec)
    slat (usec): min=866, max=3796, avg=1725.06, stdev=394.50
    clat (usec): min=13795, max=40918, avg=24272.67, stdev=4958.95
     lat (usec): min=15362, max=43900, avg=26000.99, stdev=5119.97
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17171], 10.00th=[18220], 20.00th=[19792],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[26608], 80.00th=[28443], 90.00th=[31065], 95.00th=[33162],
     | 99.00th=[38536], 99.50th=[39584], 99.90th=[40633], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  840, max= 1536, per=24.96%, avg=1202.00, stdev=207.33, samples=8
   iops        : min=  210, max=  384, avg=300.50, stdev=51.83, samples=8
  write: IOPS=312, BW=1249KiB/s (1279kB/s)(5192KiB/4156msec)
    slat (usec): min=13, max=912, avg=79.61, stdev=47.83
    clat (usec): min=4000, max=42034, avg=25461.90, stdev=4786.21
     lat (usec): min=4050, max=42079, avg=25543.12, stdev=4787.47
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[18744], 10.00th=[20055], 20.00th=[21103],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24773], 60.00th=[26084],
     | 70.00th=[27395], 80.00th=[29230], 90.00th=[31851], 95.00th=[34866],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[41157], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min=  880, max= 1528, per=24.74%, avg=1240.00, stdev=201.89, samples=8
   iops        : min=  220, max=  382, avg=310.00, stdev=50.47, samples=8
  lat (msec)   : 10=0.04%, 20=15.70%, 50=84.26%
  cpu          : usr=1.13%, sys=9.07%, ctx=2619, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8160: Fri Aug  9 23:32:43 2024
   read: IOPS=298, BW=1194KiB/s (1223kB/s)(4976KiB/4167msec)
    slat (usec): min=903, max=3592, avg=1719.48, stdev=388.94
    clat (usec): min=5978, max=47621, avg=24406.24, stdev=5103.90
     lat (usec): min=7926, max=49489, avg=26129.48, stdev=5259.60
    clat percentiles (usec):
     |  1.00th=[15008],  5.00th=[17171], 10.00th=[18482], 20.00th=[20055],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[26608], 80.00th=[28443], 90.00th=[31327], 95.00th=[33424],
     | 99.00th=[38011], 99.50th=[39584], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min=  768, max= 1440, per=24.34%, avg=1172.00, stdev=204.18, samples=8
   iops        : min=  192, max=  360, avg=293.00, stdev=51.05, samples=8
  write: IOPS=315, BW=1263KiB/s (1294kB/s)(5264KiB/4167msec)
    slat (usec): min=14, max=2218, avg=83.60, stdev=86.87
    clat (usec): min=4377, max=45929, avg=25406.44, stdev=4907.81
     lat (usec): min=4477, max=46008, avg=25491.79, stdev=4911.56
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[18744], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[22414], 40.00th=[23725], 50.00th=[24773], 60.00th=[26084],
     | 70.00th=[27657], 80.00th=[29230], 90.00th=[32113], 95.00th=[34341],
     | 99.00th=[39584], 99.50th=[41157], 99.90th=[43779], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min=  888, max= 1496, per=25.01%, avg=1254.00, stdev=207.50, samples=8
   iops        : min=  222, max=  374, avg=313.50, stdev=51.88, samples=8
  lat (msec)   : 10=0.20%, 20=14.80%, 50=85.00%
  cpu          : usr=1.18%, sys=8.95%, ctx=2646, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8161: Fri Aug  9 23:32:43 2024
   read: IOPS=300, BW=1204KiB/s (1233kB/s)(4984KiB/4140msec)
    slat (usec): min=26, max=4269, avg=1716.55, stdev=421.59
    clat (usec): min=3406, max=42125, avg=24299.63, stdev=5169.19
     lat (usec): min=3440, max=44426, avg=26019.67, stdev=5346.90
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[17433], 10.00th=[18482], 20.00th=[20055],
     | 30.00th=[21365], 40.00th=[22676], 50.00th=[23725], 60.00th=[24773],
     | 70.00th=[26608], 80.00th=[28443], 90.00th=[31327], 95.00th=[33817],
     | 99.00th=[39060], 99.50th=[40109], 99.90th=[41157], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min=  894, max= 1488, per=24.96%, avg=1201.75, stdev=204.74, samples=8
   iops        : min=  223, max=  372, avg=300.38, stdev=51.29, samples=8
  write: IOPS=317, BW=1270KiB/s (1300kB/s)(5256KiB/4140msec)
    slat (usec): min=14, max=827, avg=81.25, stdev=49.58
    clat (usec): min=10004, max=43785, avg=25513.29, stdev=4919.46
     lat (usec): min=10038, max=43831, avg=25596.25, stdev=4921.97
    clat percentiles (usec):
     |  1.00th=[16581],  5.00th=[18482], 10.00th=[19792], 20.00th=[21365],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[25035], 60.00th=[26084],
     | 70.00th=[27657], 80.00th=[29230], 90.00th=[32113], 95.00th=[34866],
     | 99.00th=[39060], 99.50th=[41157], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  950, max= 1456, per=25.03%, avg=1254.75, stdev=171.95, samples=8
   iops        : min=  237, max=  364, avg=313.62, stdev=43.11, samples=8
  lat (msec)   : 4=0.08%, 10=0.12%, 20=15.82%, 50=83.98%
  cpu          : usr=1.52%, sys=8.63%, ctx=2726, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8162: Fri Aug  9 23:32:43 2024
   read: IOPS=304, BW=1217KiB/s (1246kB/s)(5060KiB/4159msec)
    slat (usec): min=875, max=3764, avg=1731.41, stdev=395.61
    clat (usec): min=7734, max=42733, avg=24281.70, stdev=4985.14
     lat (usec): min=10290, max=44993, avg=26016.37, stdev=5147.80
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17433], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[21365], 40.00th=[22676], 50.00th=[23725], 60.00th=[24773],
     | 70.00th=[26346], 80.00th=[28181], 90.00th=[31327], 95.00th=[33817],
     | 99.00th=[38011], 99.50th=[39060], 99.90th=[42206], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  840, max= 1432, per=24.86%, avg=1197.00, stdev=184.32, samples=8
   iops        : min=  210, max=  358, avg=299.25, stdev=46.08, samples=8
  write: IOPS=311, BW=1245KiB/s (1275kB/s)(5180KiB/4159msec)
    slat (usec): min=17, max=539, avg=80.42, stdev=44.36
    clat (usec): min=6099, max=43748, avg=25478.96, stdev=4933.44
     lat (usec): min=6160, max=43880, avg=25561.37, stdev=4937.76
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[18482], 10.00th=[19792], 20.00th=[21103],
     | 30.00th=[22414], 40.00th=[23725], 50.00th=[24773], 60.00th=[26084],
     | 70.00th=[27657], 80.00th=[29492], 90.00th=[32375], 95.00th=[34341],
     | 99.00th=[39584], 99.50th=[41157], 99.90th=[43254], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  872, max= 1512, per=24.72%, avg=1239.00, stdev=212.93, samples=8
   iops        : min=  218, max=  378, avg=309.75, stdev=53.23, samples=8
  lat (msec)   : 10=0.08%, 20=15.74%, 50=84.18%
  cpu          : usr=1.30%, sys=8.87%, ctx=2664, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4816KiB/s (4932kB/s), 1194KiB/s-1217KiB/s (1223kB/s-1246kB/s), io=19.6MiB (20.5MB), run=4140-4167msec
  WRITE: bw=5014KiB/s (5134kB/s), 1245KiB/s-1270KiB/s (1275kB/s-1300kB/s), io=20.4MiB (21.4MB), run=4140-4167msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=8169: Fri Aug  9 23:32:44 2024
   read: IOPS=6, BW=265MiB/s (278MB/s)(40.0MiB/151msec)
    slat (nsec): min=6088.8k, max=6088.8k, avg=6088841.00, stdev= 0.00
    clat (nsec): min=136917k, max=136917k, avg=136916924.00, stdev= 0.00
     lat (nsec): min=143017k, max=143017k, avg=143017148.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  138],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  138], 40.00th=[  138], 50.00th=[  138], 60.00th=[  138],
     | 70.00th=[  138], 80.00th=[  138], 90.00th=[  138], 95.00th=[  138],
     | 99.00th=[  138], 99.50th=[  138], 99.90th=[  138], 99.95th=[  138],
     | 99.99th=[  138]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.67%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=40.0MiB (41.9MB), run=151-151msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout