== sanity test 398c: run fio to test AIO ================= 13:24:46 (1722792286)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.114617 s, 366 MB/s
osc.lustre-OST0000-osc-ffff88012a49c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=6104: Sun Aug  4 13:24:54 2024
  write: IOPS=372, BW=1490KiB/s (1526kB/s)(10.0MiB/6872msec)
    slat (usec): min=25, max=437, avg=87.28, stdev=40.32
    clat (usec): min=5945, max=59724, avg=42611.04, stdev=4401.00
     lat (usec): min=6078, max=59847, avg=42700.60, stdev=4400.97
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[35390], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[45876], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1405, max= 1736, per=24.95%, avg=1486.85, stdev=93.11, samples=13
   iops        : min=  351, max=  434, avg=371.69, stdev=23.30, samples=13
  lat (msec)   : 10=0.04%, 20=0.62%, 50=99.14%, 100=0.20%
  cpu          : usr=0.67%, sys=3.92%, ctx=1265, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6105: Sun Aug  4 13:24:54 2024
  write: IOPS=372, BW=1490KiB/s (1526kB/s)(10.0MiB/6871msec)
    slat (usec): min=19, max=404, avg=77.89, stdev=43.51
    clat (usec): min=4252, max=49862, avg=42581.04, stdev=4471.93
     lat (usec): min=4289, max=49893, avg=42661.10, stdev=4475.80
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[35390], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[45876], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[49546], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min= 1405, max= 1680, per=24.81%, avg=1478.85, stdev=91.91, samples=13
   iops        : min=  351, max=  420, avg=369.69, stdev=22.99, samples=13
  lat (msec)   : 10=0.08%, 20=0.35%, 50=99.57%
  cpu          : usr=0.61%, sys=3.41%, ctx=895, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6106: Sun Aug  4 13:24:54 2024
  write: IOPS=373, BW=1493KiB/s (1529kB/s)(10.0MiB/6859msec)
    slat (usec): min=26, max=371, avg=83.64, stdev=43.13
    clat (usec): min=5285, max=76008, avg=42590.11, stdev=4924.64
     lat (usec): min=5380, max=76074, avg=42675.99, stdev=4921.02
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[34866], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[45876], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1405, max= 1728, per=25.01%, avg=1490.54, stdev=110.31, samples=13
   iops        : min=  351, max=  432, avg=372.62, stdev=27.59, samples=13
  lat (msec)   : 10=0.12%, 20=0.70%, 50=98.95%, 100=0.23%
  cpu          : usr=0.63%, sys=3.73%, ctx=1152, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=6107: Sun Aug  4 13:24:54 2024
  write: IOPS=372, BW=1491KiB/s (1527kB/s)(10.0MiB/6868msec)
    slat (usec): min=24, max=568, avg=93.12, stdev=46.62
    clat (usec): min=8587, max=66367, avg=42621.35, stdev=4312.34
     lat (usec): min=8689, max=66443, avg=42716.78, stdev=4314.61
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[35390], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[45876], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[61604], 99.95th=[62129],
     | 99.99th=[66323]
   bw (  KiB/s): min= 1405, max= 1752, per=24.98%, avg=1488.69, stdev=97.70, samples=13
   iops        : min=  351, max=  438, avg=372.15, stdev=24.44, samples=13
  lat (msec)   : 10=0.12%, 20=0.04%, 50=99.65%, 100=0.20%
  cpu          : usr=0.63%, sys=4.43%, ctx=1521, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5960KiB/s (6103kB/s), 1490KiB/s-1493KiB/s (1526kB/s-1529kB/s), io=40.0MiB (41.9MB), run=6859-6872msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=6122: Sun Aug  4 13:24:58 2024
   read: IOPS=458, BW=1834KiB/s (1878kB/s)(5048KiB/2752msec)
    slat (usec): min=614, max=3073, avg=1120.49, stdev=351.49
    clat (usec): min=3880, max=38386, avg=16162.38, stdev=3934.93
     lat (usec): min=4934, max=40614, avg=17284.70, stdev=4159.17
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[11994], 10.00th=[12649], 20.00th=[13435],
     | 30.00th=[14091], 40.00th=[14746], 50.00th=[15401], 60.00th=[16057],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19530], 95.00th=[23725],
     | 99.00th=[31851], 99.50th=[34866], 99.90th=[38011], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1232, max= 1984, per=24.62%, avg=1795.20, stdev=316.71, samples=5
   iops        : min=  308, max=  496, avg=448.80, stdev=79.18, samples=5
  write: IOPS=471, BW=1887KiB/s (1932kB/s)(5192KiB/2752msec)
    slat (usec): min=14, max=267, avg=48.34, stdev=28.53
    clat (usec): min=7580, max=36186, avg=16678.91, stdev=3568.26
     lat (usec): min=7620, max=36331, avg=16728.36, stdev=3577.45
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13435], 10.00th=[13960], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15401], 50.00th=[15926], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[19268], 95.00th=[24773],
     | 99.00th=[32113], 99.50th=[32900], 99.90th=[34341], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 1208, max= 2160, per=24.81%, avg=1883.20, stdev=384.94, samples=5
   iops        : min=  302, max=  540, avg=470.80, stdev=96.24, samples=5
  lat (msec)   : 4=0.04%, 10=0.20%, 20=90.86%, 50=8.91%
  cpu          : usr=0.87%, sys=7.42%, ctx=2837, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6123: Sun Aug  4 13:24:58 2024
   read: IOPS=452, BW=1809KiB/s (1852kB/s)(4976KiB/2751msec)
    slat (usec): min=602, max=3208, avg=1118.95, stdev=342.45
    clat (usec): min=2103, max=41084, avg=16045.81, stdev=4181.38
     lat (usec): min=3013, max=43234, avg=17166.62, stdev=4400.33
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[11731], 10.00th=[12518], 20.00th=[13304],
     | 30.00th=[14222], 40.00th=[14746], 50.00th=[15401], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17695], 90.00th=[19268], 95.00th=[23987],
     | 99.00th=[34866], 99.50th=[36439], 99.90th=[40109], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min= 1088, max= 2048, per=24.42%, avg=1780.80, stdev=397.42, samples=5
   iops        : min=  272, max=  512, avg=445.20, stdev=99.35, samples=5
  write: IOPS=478, BW=1913KiB/s (1959kB/s)(5264KiB/2751msec)
    slat (usec): min=13, max=256, avg=45.20, stdev=23.24
    clat (usec): min=3560, max=39231, avg=16782.56, stdev=3914.51
     lat (usec): min=3617, max=39295, avg=16828.99, stdev=3922.53
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13304], 10.00th=[13960], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15533], 50.00th=[15926], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19268], 95.00th=[25297],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[37487], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1320, max= 2184, per=25.17%, avg=1910.40, stdev=338.30, samples=5
   iops        : min=  330, max=  546, avg=477.60, stdev=84.57, samples=5
  lat (msec)   : 4=0.23%, 10=0.23%, 20=90.82%, 50=8.71%
  cpu          : usr=0.73%, sys=7.56%, ctx=2673, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6124: Sun Aug  4 13:24:58 2024
   read: IOPS=455, BW=1821KiB/s (1865kB/s)(4984KiB/2737msec)
    slat (usec): min=30, max=5518, avg=1118.29, stdev=380.26
    clat (usec): min=1099, max=38801, avg=16164.12, stdev=4026.93
     lat (usec): min=1145, max=40718, avg=17284.38, stdev=4246.27
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[11863], 10.00th=[12649], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15401], 60.00th=[16057],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19792], 95.00th=[23725],
     | 99.00th=[32637], 99.50th=[34866], 99.90th=[36963], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1336, max= 2112, per=24.84%, avg=1811.20, stdev=286.47, samples=5
   iops        : min=  334, max=  528, avg=452.80, stdev=71.62, samples=5
  write: IOPS=480, BW=1920KiB/s (1966kB/s)(5256KiB/2737msec)
    slat (usec): min=13, max=223, avg=45.96, stdev=24.65
    clat (usec): min=3282, max=36536, avg=16798.92, stdev=3726.33
     lat (usec): min=3326, max=36579, avg=16846.04, stdev=3734.39
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13173], 10.00th=[13829], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[15926], 60.00th=[16581],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19530], 95.00th=[25560],
     | 99.00th=[32637], 99.50th=[34341], 99.90th=[34866], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 1336, max= 2152, per=25.19%, avg=1912.00, stdev=331.73, samples=5
   iops        : min=  334, max=  538, avg=478.00, stdev=82.93, samples=5
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.23%, 20=90.16%, 50=9.41%
  cpu          : usr=0.95%, sys=7.16%, ctx=2766, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=6125: Sun Aug  4 13:24:58 2024
   read: IOPS=460, BW=1842KiB/s (1886kB/s)(5060KiB/2747msec)
    slat (usec): min=641, max=3135, avg=1121.08, stdev=336.75
    clat (usec): min=2856, max=39820, avg=16125.22, stdev=4097.07
     lat (usec): min=3881, max=41781, avg=17248.24, stdev=4305.91
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11863], 10.00th=[12649], 20.00th=[13566],
     | 30.00th=[14091], 40.00th=[14877], 50.00th=[15401], 60.00th=[15926],
     | 70.00th=[16712], 80.00th=[17695], 90.00th=[19530], 95.00th=[24773],
     | 99.00th=[33817], 99.50th=[36963], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1240, max= 2048, per=25.17%, avg=1835.20, stdev=337.50, samples=5
   iops        : min=  310, max=  512, avg=458.80, stdev=84.38, samples=5
  write: IOPS=471, BW=1886KiB/s (1931kB/s)(5180KiB/2747msec)
    slat (usec): min=13, max=282, avg=45.24, stdev=24.16
    clat (usec): min=11242, max=36959, avg=16773.20, stdev=3773.52
     lat (usec): min=11286, max=37057, avg=16819.53, stdev=3779.81
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13304], 10.00th=[13960], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15533], 50.00th=[15926], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19268], 95.00th=[25822],
     | 99.00th=[33817], 99.50th=[34866], 99.90th=[35914], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1264, max= 2112, per=24.58%, avg=1865.60, stdev=347.31, samples=5
   iops        : min=  316, max=  528, avg=466.40, stdev=86.83, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=90.98%, 50=8.83%
  cpu          : usr=0.87%, sys=7.32%, ctx=2746, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7292KiB/s (7467kB/s), 1809KiB/s-1842KiB/s (1852kB/s-1886kB/s), io=19.6MiB (20.5MB), run=2737-2752msec
  WRITE: bw=7592KiB/s (7774kB/s), 1886KiB/s-1920KiB/s (1931kB/s-1966kB/s), io=20.4MiB (21.4MB), run=2737-2752msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=6131: Sun Aug  4 13:25:00 2024
   read: IOPS=9, BW=360MiB/s (378MB/s)(40.0MiB/111msec)
    slat (nsec): min=5843.3k, max=5843.3k, avg=5843306.00, stdev= 0.00
    clat (nsec): min=99475k, max=99475k, avg=99475323.00, stdev= 0.00
     lat (nsec): min=105331k, max=105331k, avg=105331452.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[99091],  5.00th=[99091], 10.00th=[99091], 20.00th=[99091],
     | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091],
     | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091],
     | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091],
     | 99.99th=[99091]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.45%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=40.0MiB (41.9MB), run=111-111msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout