== sanity test 398c: run fio to test AIO ================= 02:23:24 (1721888604)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.132317 s, 317 MB/s
osc.lustre-OST0000-osc-ffff88012f88f000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=6383: Thu Jul 25 02:23:33 2024
  write: IOPS=351, BW=1407KiB/s (1441kB/s)(10.0MiB/7276msec)
    slat (usec): min=23, max=659, avg=92.06, stdev=46.83
    clat (usec): min=4927, max=91194, avg=45222.96, stdev=6716.08
     lat (usec): min=4995, max=91324, avg=45317.17, stdev=6717.35
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[34866], 10.00th=[37487], 20.00th=[42206],
     | 30.00th=[43779], 40.00th=[45351], 50.00th=[46400], 60.00th=[47449],
     | 70.00th=[47973], 80.00th=[49021], 90.00th=[50070], 95.00th=[53216],
     | 99.00th=[56886], 99.50th=[59507], 99.90th=[85459], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min= 1248, max= 1880, per=25.24%, avg=1411.21, stdev=155.63, samples=14
   iops        : min=  312, max=  470, avg=352.79, stdev=38.92, samples=14
  lat (msec)   : 10=0.08%, 20=1.13%, 50=88.63%, 100=10.16%
  cpu          : usr=0.58%, sys=3.96%, ctx=1759, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6384: Thu Jul 25 02:23:33 2024
  write: IOPS=349, BW=1398KiB/s (1432kB/s)(10.0MiB/7324msec)
    slat (usec): min=23, max=326, avg=73.64, stdev=38.73
    clat (usec): min=6578, max=84816, avg=45466.86, stdev=5719.15
     lat (usec): min=6729, max=84889, avg=45542.33, stdev=5724.08
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[35390], 10.00th=[38011], 20.00th=[42730],
     | 30.00th=[43779], 40.00th=[45351], 50.00th=[46924], 60.00th=[47449],
     | 70.00th=[48497], 80.00th=[49021], 90.00th=[50070], 95.00th=[53216],
     | 99.00th=[55837], 99.50th=[56886], 99.90th=[58459], 99.95th=[80217],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1280, max= 1664, per=25.04%, avg=1400.14, stdev=113.00, samples=14
   iops        : min=  320, max=  416, avg=350.00, stdev=28.26, samples=14
  lat (msec)   : 10=0.04%, 20=0.62%, 50=88.40%, 100=10.94%
  cpu          : usr=0.44%, sys=3.21%, ctx=984, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6385: Thu Jul 25 02:23:33 2024
  write: IOPS=349, BW=1399KiB/s (1433kB/s)(10.0MiB/7319msec)
    slat (usec): min=23, max=667, avg=86.05, stdev=46.95
    clat (usec): min=7947, max=89656, avg=45484.05, stdev=6113.20
     lat (usec): min=8066, max=89856, avg=45572.28, stdev=6119.35
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[35390], 10.00th=[38536], 20.00th=[42206],
     | 30.00th=[43779], 40.00th=[45876], 50.00th=[46924], 60.00th=[47449],
     | 70.00th=[48497], 80.00th=[49021], 90.00th=[50070], 95.00th=[53216],
     | 99.00th=[56886], 99.50th=[59507], 99.90th=[80217], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min= 1232, max= 1792, per=25.09%, avg=1403.00, stdev=135.98, samples=14
   iops        : min=  308, max=  448, avg=350.71, stdev=33.99, samples=14
  lat (msec)   : 10=0.04%, 50=89.14%, 100=10.82%
  cpu          : usr=0.55%, sys=3.65%, ctx=1556, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6386: Thu Jul 25 02:23:33 2024
  write: IOPS=349, BW=1400KiB/s (1433kB/s)(10.0MiB/7316msec)
    slat (usec): min=23, max=1091, avg=84.18, stdev=46.63
    clat (usec): min=14637, max=86421, avg=45433.73, stdev=6225.20
     lat (usec): min=14791, max=86570, avg=45519.87, stdev=6229.03
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[35390], 10.00th=[38011], 20.00th=[42206],
     | 30.00th=[44303], 40.00th=[45876], 50.00th=[46924], 60.00th=[47449],
     | 70.00th=[47973], 80.00th=[49021], 90.00th=[50070], 95.00th=[53216],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[82314], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1224, max= 1792, per=25.08%, avg=1402.43, stdev=139.67, samples=14
   iops        : min=  306, max=  448, avg=350.57, stdev=34.93, samples=14
  lat (msec)   : 20=0.70%, 50=88.48%, 100=10.82%
  cpu          : usr=0.51%, sys=3.54%, ctx=1420, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5593KiB/s (5727kB/s), 1398KiB/s-1407KiB/s (1432kB/s-1441kB/s), io=40.0MiB (41.9MB), run=7276-7324msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=6403: Thu Jul 25 02:23:39 2024
   read: IOPS=252, BW=1009KiB/s (1033kB/s)(5048KiB/5004msec)
    slat (usec): min=22, max=4272, avg=1964.28, stdev=466.73
    clat (usec): min=2868, max=56587, avg=29731.42, stdev=5672.29
     lat (usec): min=2990, max=58161, avg=31699.11, stdev=5841.23
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[21103], 10.00th=[22938], 20.00th=[25035],
     | 30.00th=[26608], 40.00th=[28181], 50.00th=[29492], 60.00th=[30802],
     | 70.00th=[32375], 80.00th=[34341], 90.00th=[36963], 95.00th=[39060],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[54264], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  864, max= 1232, per=25.12%, avg=1001.90, stdev=106.04, samples=10
   iops        : min=  216, max=  308, avg=250.40, stdev=26.56, samples=10
  write: IOPS=259, BW=1038KiB/s (1062kB/s)(5192KiB/5004msec)
    slat (usec): min=15, max=412, avg=79.67, stdev=39.30
    clat (usec): min=5831, max=50333, avg=30647.37, stdev=4833.46
     lat (usec): min=5945, max=50411, avg=30728.79, stdev=4838.00
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[22938], 10.00th=[24511], 20.00th=[26608],
     | 30.00th=[28443], 40.00th=[29754], 50.00th=[30802], 60.00th=[32113],
     | 70.00th=[33162], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[48497], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  910, max= 1336, per=24.88%, avg=1033.10, stdev=127.05, samples=10
   iops        : min=  227, max=  334, avg=258.20, stdev=31.81, samples=10
  lat (msec)   : 4=0.08%, 10=0.16%, 20=1.64%, 50=98.01%, 100=0.12%
  cpu          : usr=1.18%, sys=7.28%, ctx=2774, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6404: Thu Jul 25 02:23:39 2024
   read: IOPS=247, BW=989KiB/s (1013kB/s)(4976KiB/5031msec)
    slat (usec): min=895, max=3842, avg=1968.42, stdev=421.13
    clat (usec): min=16045, max=48670, avg=29505.02, stdev=5272.53
     lat (usec): min=17824, max=51039, avg=31476.77, stdev=5425.35
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[21103], 10.00th=[23200], 20.00th=[25035],
     | 30.00th=[26346], 40.00th=[27919], 50.00th=[29230], 60.00th=[30540],
     | 70.00th=[32113], 80.00th=[34341], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[45876], 99.90th=[48497], 99.95th=[48497],
     | 99.99th=[48497]
   bw (  KiB/s): min=  752, max= 1152, per=24.73%, avg=986.10, stdev=120.74, samples=10
   iops        : min=  188, max=  288, avg=246.50, stdev=30.16, samples=10
  write: IOPS=261, BW=1046KiB/s (1071kB/s)(5264KiB/5031msec)
    slat (usec): min=13, max=523, avg=78.80, stdev=44.50
    clat (usec): min=18322, max=46704, avg=30924.88, stdev=4530.56
     lat (usec): min=18419, max=46794, avg=31005.53, stdev=4533.48
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[23200], 10.00th=[24773], 20.00th=[26870],
     | 30.00th=[28705], 40.00th=[30016], 50.00th=[31065], 60.00th=[32113],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[40109], 99.50th=[41157], 99.90th=[43779], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  872, max= 1416, per=25.20%, avg=1046.10, stdev=147.54, samples=10
   iops        : min=  218, max=  354, avg=261.50, stdev=36.89, samples=10
  lat (msec)   : 20=1.91%, 50=98.09%
  cpu          : usr=1.07%, sys=7.24%, ctx=2685, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6405: Thu Jul 25 02:23:39 2024
   read: IOPS=247, BW=991KiB/s (1015kB/s)(4984KiB/5029msec)
    slat (usec): min=842, max=4431, avg=1973.24, stdev=414.86
    clat (usec): min=16762, max=46454, avg=29717.96, stdev=5274.21
     lat (usec): min=18352, max=48731, avg=31694.69, stdev=5436.46
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[21103], 10.00th=[22938], 20.00th=[25297],
     | 30.00th=[26870], 40.00th=[27919], 50.00th=[29230], 60.00th=[30802],
     | 70.00th=[32637], 80.00th=[34341], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  782, max= 1232, per=24.90%, avg=993.10, stdev=112.17, samples=10
   iops        : min=  195, max=  308, avg=248.20, stdev=28.14, samples=10
  write: IOPS=261, BW=1045KiB/s (1070kB/s)(5256KiB/5029msec)
    slat (usec): min=15, max=391, avg=80.61, stdev=39.58
    clat (usec): min=18443, max=54331, avg=30714.82, stdev=4613.04
     lat (usec): min=18504, max=54381, avg=30797.11, stdev=4617.00
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[22676], 10.00th=[24773], 20.00th=[26870],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30802], 60.00th=[31851],
     | 70.00th=[33162], 80.00th=[34866], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[41157], 99.50th=[42206], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  880, max= 1320, per=25.09%, avg=1041.90, stdev=139.39, samples=10
   iops        : min=  220, max=  330, avg=260.40, stdev=34.91, samples=10
  lat (msec)   : 20=1.68%, 50=98.20%, 100=0.12%
  cpu          : usr=0.91%, sys=7.52%, ctx=2838, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6406: Thu Jul 25 02:23:39 2024
   read: IOPS=251, BW=1006KiB/s (1030kB/s)(5060KiB/5031msec)
    slat (usec): min=855, max=4075, avg=1952.07, stdev=414.62
    clat (usec): min=16638, max=43848, avg=29452.38, stdev=5204.24
     lat (usec): min=17925, max=46366, avg=31408.03, stdev=5346.80
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[21365], 10.00th=[22938], 20.00th=[25035],
     | 30.00th=[26346], 40.00th=[27657], 50.00th=[28967], 60.00th=[30540],
     | 70.00th=[32113], 80.00th=[34341], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[42730], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  776, max= 1280, per=25.21%, avg=1005.40, stdev=132.67, samples=10
   iops        : min=  194, max=  320, avg=251.30, stdev=33.19, samples=10
  write: IOPS=257, BW=1030KiB/s (1054kB/s)(5180KiB/5031msec)
    slat (usec): min=18, max=941, avg=80.53, stdev=51.47
    clat (usec): min=19286, max=50808, avg=30877.83, stdev=4464.32
     lat (usec): min=19358, max=50957, avg=30960.17, stdev=4469.07
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23462], 10.00th=[24773], 20.00th=[26608],
     | 30.00th=[28705], 40.00th=[30016], 50.00th=[31065], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[40633], 99.50th=[42730], 99.90th=[45876], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  824, max= 1288, per=24.71%, avg=1026.10, stdev=128.05, samples=10
   iops        : min=  206, max=  322, avg=256.50, stdev=32.00, samples=10
  lat (msec)   : 20=1.37%, 50=98.59%, 100=0.04%
  cpu          : usr=1.19%, sys=7.46%, ctx=2606, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3989KiB/s (4085kB/s), 989KiB/s-1009KiB/s (1013kB/s-1033kB/s), io=19.6MiB (20.5MB), run=5004-5031msec
  WRITE: bw=4153KiB/s (4252kB/s), 1030KiB/s-1046KiB/s (1054kB/s-1071kB/s), io=20.4MiB (21.4MB), run=5004-5031msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=6414: Thu Jul 25 02:23:41 2024
   read: IOPS=5, BW=231MiB/s (242MB/s)(40.0MiB/173msec)
    slat (nsec): min=5235.3k, max=5235.3k, avg=5235335.00, stdev= 0.00
    clat (nsec): min=161307k, max=161307k, avg=161307479.00, stdev= 0.00
     lat (nsec): min=166552k, max=166552k, avg=166551779.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  161],  5.00th=[  161], 10.00th=[  161], 20.00th=[  161],
     | 30.00th=[  161], 40.00th=[  161], 50.00th=[  161], 60.00th=[  161],
     | 70.00th=[  161], 80.00th=[  161], 90.00th=[  161], 95.00th=[  161],
     | 99.00th=[  161], 99.50th=[  161], 99.90th=[  161], 99.95th=[  161],
     | 99.99th=[  161]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=2.91%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=40.0MiB (41.9MB), run=173-173msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout