== sanity test 398c: run fio to test AIO ================= 08:13:34 (1744892014)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.129759 s, 323 MB/s
osc.lustre-OST0000-osc-ffff8800b35a6800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=29414: Thu Apr 17 08:13:42 2025
  write: IOPS=396, BW=1588KiB/s (1626kB/s)(10.0MiB/6449msec)
    slat (usec): min=14, max=462, avg=82.01, stdev=43.75
    clat (usec): min=6062, max=73255, avg=40052.47, stdev=6512.10
     lat (usec): min=6150, max=73347, avg=40136.46, stdev=6517.79
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[29230], 10.00th=[31327], 20.00th=[34866],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[45351], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[51119], 99.50th=[52167], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1381, max= 1920, per=24.90%, avg=1579.08, stdev=174.13, samples=12
   iops        : min=  345, max=  480, avg=394.75, stdev=43.56, samples=12
  lat (msec)   : 10=0.12%, 20=1.05%, 50=96.84%, 100=1.99%
  cpu          : usr=0.57%, sys=3.99%, ctx=1439, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29415: Thu Apr 17 08:13:42 2025
  write: IOPS=396, BW=1587KiB/s (1625kB/s)(10.0MiB/6454msec)
    slat (usec): min=13, max=1177, avg=77.03, stdev=49.81
    clat (usec): min=9444, max=64896, avg=40123.87, stdev=6141.71
     lat (usec): min=9562, max=64979, avg=40202.80, stdev=6149.35
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[29230], 10.00th=[31327], 20.00th=[34866],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[40633], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[45351], 90.00th=[46924], 95.00th=[47973],
     | 99.00th=[51119], 99.50th=[52167], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[64750]
   bw (  KiB/s): min= 1389, max= 1920, per=24.81%, avg=1573.75, stdev=172.48, samples=12
   iops        : min=  347, max=  480, avg=393.42, stdev=43.15, samples=12
  lat (msec)   : 10=0.04%, 20=0.78%, 50=97.07%, 100=2.11%
  cpu          : usr=0.70%, sys=3.47%, ctx=1313, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29416: Thu Apr 17 08:13:42 2025
  write: IOPS=396, BW=1586KiB/s (1624kB/s)(10.0MiB/6458msec)
    slat (usec): min=15, max=348, avg=73.40, stdev=40.72
    clat (usec): min=17917, max=61929, avg=40076.88, stdev=6039.25
     lat (usec): min=17998, max=62026, avg=40152.17, stdev=6047.25
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[29230], 10.00th=[31065], 20.00th=[34866],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[46400], 95.00th=[47973],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[59507], 99.95th=[60031],
     | 99.99th=[62129]
   bw (  KiB/s): min= 1397, max= 1880, per=24.79%, avg=1572.42, stdev=172.62, samples=12
   iops        : min=  349, max=  470, avg=393.08, stdev=43.18, samples=12
  lat (msec)   : 20=0.20%, 50=97.97%, 100=1.84%
  cpu          : usr=0.50%, sys=3.65%, ctx=1019, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29417: Thu Apr 17 08:13:42 2025
  write: IOPS=397, BW=1590KiB/s (1628kB/s)(10.0MiB/6441msec)
    slat (usec): min=16, max=669, avg=83.99, stdev=47.39
    clat (usec): min=4175, max=69122, avg=40043.43, stdev=6574.42
     lat (usec): min=4205, max=69172, avg=40129.46, stdev=6572.64
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[28967], 10.00th=[31327], 20.00th=[34866],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[51119], 99.50th=[52167], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1389, max= 1960, per=24.91%, avg=1579.75, stdev=185.82, samples=12
   iops        : min=  347, max=  490, avg=394.92, stdev=46.48, samples=12
  lat (msec)   : 10=0.27%, 20=0.70%, 50=96.48%, 100=2.54%
  cpu          : usr=0.56%, sys=4.07%, ctx=1460, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6343KiB/s (6495kB/s), 1586KiB/s-1590KiB/s (1624kB/s-1628kB/s), io=40.0MiB (41.9MB), run=6441-6458msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=29432: Thu Apr 17 08:13:46 2025
   read: IOPS=465, BW=1861KiB/s (1906kB/s)(5048KiB/2712msec)
    slat (usec): min=18, max=4202, avg=1113.29, stdev=337.92
    clat (usec): min=3167, max=37260, avg=16031.03, stdev=3417.17
     lat (usec): min=3287, max=39995, avg=17146.11, stdev=3593.55
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[12387], 10.00th=[13042], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15401], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19006], 95.00th=[20841],
     | 99.00th=[31589], 99.50th=[32113], 99.90th=[34866], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1480, max= 2032, per=25.28%, avg=1851.20, stdev=213.74, samples=5
   iops        : min=  370, max=  508, avg=462.80, stdev=53.43, samples=5
  write: IOPS=478, BW=1914KiB/s (1960kB/s)(5192KiB/2712msec)
    slat (usec): min=13, max=241, avg=40.38, stdev=21.31
    clat (usec): min=5451, max=37920, avg=16622.99, stdev=3464.63
     lat (usec): min=5530, max=38016, avg=16664.34, stdev=3471.21
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13304], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15139], 40.00th=[15533], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19006], 95.00th=[22414],
     | 99.00th=[33162], 99.50th=[35390], 99.90th=[36963], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1544, max= 2088, per=25.16%, avg=1918.40, stdev=224.46, samples=5
   iops        : min=  386, max=  522, avg=479.60, stdev=56.11, samples=5
  lat (msec)   : 4=0.04%, 10=0.35%, 20=92.89%, 50=6.72%
  cpu          : usr=1.11%, sys=7.37%, ctx=2225, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29433: Thu Apr 17 08:13:46 2025
   read: IOPS=455, BW=1823KiB/s (1867kB/s)(4976KiB/2729msec)
    slat (usec): min=601, max=3799, avg=1120.80, stdev=312.27
    clat (usec): min=2398, max=38854, avg=16071.98, stdev=3483.77
     lat (usec): min=3585, max=41398, avg=17194.88, stdev=3657.62
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12256], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[19006], 95.00th=[21365],
     | 99.00th=[30540], 99.50th=[35390], 99.90th=[36963], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1312, max= 2032, per=24.69%, avg=1808.00, stdev=283.52, samples=5
   iops        : min=  328, max=  508, avg=452.00, stdev=70.88, samples=5
  write: IOPS=482, BW=1929KiB/s (1975kB/s)(5264KiB/2729msec)
    slat (usec): min=13, max=206, avg=39.00, stdev=19.16
    clat (usec): min=10606, max=36553, avg=16618.10, stdev=3484.58
     lat (usec): min=10636, max=36595, avg=16658.04, stdev=3488.74
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13304], 10.00th=[13960], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15533], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[21890],
     | 99.00th=[33424], 99.50th=[34866], 99.90th=[35390], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 1576, max= 2088, per=25.39%, avg=1936.00, stdev=213.17, samples=5
   iops        : min=  394, max=  522, avg=484.00, stdev=53.29, samples=5
  lat (msec)   : 4=0.04%, 10=0.20%, 20=93.16%, 50=6.60%
  cpu          : usr=1.36%, sys=6.93%, ctx=2118, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29434: Thu Apr 17 08:13:46 2025
   read: IOPS=455, BW=1820KiB/s (1864kB/s)(4984KiB/2738msec)
    slat (usec): min=613, max=3788, avg=1123.23, stdev=308.69
    clat (usec): min=3238, max=35545, avg=15913.91, stdev=3170.12
     lat (usec): min=4635, max=38406, avg=17038.95, stdev=3332.93
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12125], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17695], 90.00th=[19006], 95.00th=[20317],
     | 99.00th=[28705], 99.50th=[32637], 99.90th=[34866], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1400, max= 2048, per=24.73%, avg=1811.20, stdev=248.36, samples=5
   iops        : min=  350, max=  512, avg=452.80, stdev=62.09, samples=5
  write: IOPS=479, BW=1920KiB/s (1966kB/s)(5256KiB/2738msec)
    slat (usec): min=14, max=187, avg=41.72, stdev=22.01
    clat (usec): min=5636, max=37746, avg=16705.29, stdev=3447.52
     lat (usec): min=5675, max=37777, avg=16747.95, stdev=3454.11
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13173], 10.00th=[13960], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15664], 50.00th=[16188], 60.00th=[16581],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[19006], 95.00th=[22414],
     | 99.00th=[32637], 99.50th=[34341], 99.90th=[36439], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1416, max= 2144, per=24.97%, avg=1904.00, stdev=288.72, samples=5
   iops        : min=  354, max=  536, avg=476.00, stdev=72.18, samples=5
  lat (msec)   : 4=0.04%, 10=0.27%, 20=93.24%, 50=6.45%
  cpu          : usr=0.80%, sys=7.64%, ctx=2218, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29435: Thu Apr 17 08:13:46 2025
   read: IOPS=461, BW=1847KiB/s (1891kB/s)(5060KiB/2740msec)
    slat (usec): min=667, max=3640, avg=1118.20, stdev=297.85
    clat (usec): min=1053, max=42423, avg=16003.86, stdev=3296.55
     lat (usec): min=2134, max=44501, avg=17123.72, stdev=3447.82
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12256], 10.00th=[12911], 20.00th=[13829],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17695], 90.00th=[19006], 95.00th=[20317],
     | 99.00th=[29754], 99.50th=[33162], 99.90th=[39060], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1400, max= 1984, per=25.17%, avg=1843.20, stdev=248.81, samples=5
   iops        : min=  350, max=  496, avg=460.80, stdev=62.20, samples=5
  write: IOPS=472, BW=1891KiB/s (1936kB/s)(5180KiB/2740msec)
    slat (usec): min=13, max=293, avg=41.18, stdev=22.27
    clat (usec): min=5781, max=37951, avg=16563.38, stdev=3436.04
     lat (usec): min=5816, max=38017, avg=16605.48, stdev=3442.15
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13173], 10.00th=[13829], 20.00th=[14484],
     | 30.00th=[15008], 40.00th=[15533], 50.00th=[16057], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[22152],
     | 99.00th=[32375], 99.50th=[34341], 99.90th=[36963], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1376, max= 2088, per=24.58%, avg=1873.60, stdev=291.17, samples=5
   iops        : min=  344, max=  522, avg=468.40, stdev=72.79, samples=5
  lat (msec)   : 2=0.04%, 10=0.20%, 20=93.63%, 50=6.13%
  cpu          : usr=0.73%, sys=7.78%, ctx=2213, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7324KiB/s (7500kB/s), 1820KiB/s-1861KiB/s (1864kB/s-1906kB/s), io=19.6MiB (20.5MB), run=2712-2740msec
  WRITE: bw=7625KiB/s (7808kB/s), 1891KiB/s-1929KiB/s (1936kB/s-1975kB/s), io=20.4MiB (21.4MB), run=2712-2740msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=29441: Thu Apr 17 08:13:47 2025
   read: IOPS=6, BW=265MiB/s (278MB/s)(40.0MiB/151msec)
    slat (nsec): min=5934.7k, max=5934.7k, avg=5934712.00, stdev= 0.00
    clat (nsec): min=138600k, max=138600k, avg=138600268.00, stdev= 0.00
     lat (nsec): min=144544k, max=144544k, avg=144544475.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  140],  5.00th=[  140], 10.00th=[  140], 20.00th=[  140],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.00%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=40.0MiB (41.9MB), run=151-151msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout