== sanity test 398c: run fio to test AIO ================= 01:20:51 (1735107651)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.091364 s, 459 MB/s
osc.lustre-OST0000-osc-ffff8800a9921800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=27587: Wed Dec 25 01:20:55 2024
  write: IOPS=715, BW=2862KiB/s (2931kB/s)(10.0MiB/3578msec)
    slat (usec): min=13, max=269, avg=48.29, stdev=25.64
    clat (usec): min=9829, max=47318, avg=22206.93, stdev=3245.55
     lat (usec): min=9864, max=47352, avg=22256.47, stdev=3246.34
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[16909], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22152], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[25297], 95.00th=[26870],
     | 99.00th=[31327], 99.50th=[32637], 99.90th=[42206], 99.95th=[44827],
     | 99.99th=[47449]
   bw (  KiB/s): min= 2592, max= 3328, per=25.08%, avg=2869.71, stdev=233.00, samples=7
   iops        : min=  648, max=  832, avg=717.43, stdev=58.25, samples=7
  lat (msec)   : 10=0.62%, 20=18.55%, 50=80.82%
  cpu          : usr=0.59%, sys=4.05%, ctx=1569, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=27588: Wed Dec 25 01:20:55 2024
  write: IOPS=720, BW=2880KiB/s (2950kB/s)(10.0MiB/3555msec)
    slat (usec): min=13, max=427, avg=49.65, stdev=27.96
    clat (usec): min=3947, max=44217, avg=22074.96, stdev=3468.11
     lat (usec): min=4034, max=44272, avg=22125.90, stdev=3469.97
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[16712], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21627], 50.00th=[22152], 60.00th=[22676],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[25560], 95.00th=[26608],
     | 99.00th=[31065], 99.50th=[32375], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min= 2592, max= 3424, per=25.19%, avg=2882.29, stdev=264.39, samples=7
   iops        : min=  648, max=  856, avg=720.57, stdev=66.10, samples=7
  lat (msec)   : 4=0.04%, 10=0.78%, 20=20.20%, 50=78.98%
  cpu          : usr=0.59%, sys=4.19%, ctx=1695, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=27589: Wed Dec 25 01:20:55 2024
  write: IOPS=715, BW=2860KiB/s (2929kB/s)(10.0MiB/3580msec)
    slat (usec): min=12, max=248, avg=45.98, stdev=24.44
    clat (usec): min=3227, max=37253, avg=22182.14, stdev=3004.84
     lat (usec): min=3337, max=37285, avg=22229.29, stdev=3007.05
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[17171], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21890], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23725], 80.00th=[24249], 90.00th=[25297], 95.00th=[26608],
     | 99.00th=[31065], 99.50th=[31851], 99.90th=[32375], 99.95th=[32375],
     | 99.99th=[37487]
   bw (  KiB/s): min= 2592, max= 3273, per=24.97%, avg=2857.29, stdev=215.75, samples=7
   iops        : min=  648, max=  818, avg=714.29, stdev=53.86, samples=7
  lat (msec)   : 4=0.04%, 20=20.94%, 50=79.02%
  cpu          : usr=0.61%, sys=3.80%, ctx=1357, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=27590: Wed Dec 25 01:20:55 2024
  write: IOPS=715, BW=2864KiB/s (2932kB/s)(10.0MiB/3576msec)
    slat (usec): min=12, max=363, avg=47.62, stdev=25.91
    clat (usec): min=4747, max=48006, avg=22171.03, stdev=3360.79
     lat (usec): min=4766, max=48099, avg=22219.93, stdev=3362.57
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[17171], 10.00th=[18220], 20.00th=[20055],
     | 30.00th=[21103], 40.00th=[21627], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[25297], 95.00th=[26870],
     | 99.00th=[31327], 99.50th=[32637], 99.90th=[45876], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min= 2624, max= 3329, per=25.09%, avg=2871.00, stdev=221.30, samples=7
   iops        : min=  656, max=  832, avg=717.71, stdev=55.24, samples=7
  lat (msec)   : 10=0.66%, 20=18.91%, 50=80.43%
  cpu          : usr=0.39%, sys=4.00%, ctx=1501, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.2MiB/s (11.7MB/s), 2860KiB/s-2880KiB/s (2929kB/s-2950kB/s), io=40.0MiB (41.9MB), run=3555-3580msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=27603: Wed Dec 25 01:20:59 2024
   read: IOPS=445, BW=1784KiB/s (1827kB/s)(5048KiB/2830msec)
    slat (usec): min=625, max=2485, avg=1129.40, stdev=232.22
    clat (usec): min=2432, max=24996, avg=16658.96, stdev=2224.74
     lat (usec): min=3722, max=26694, avg=17790.12, stdev=2252.30
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13304], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[23987], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1560, max= 1856, per=24.82%, avg=1758.40, stdev=114.74, samples=5
   iops        : min=  390, max=  464, avg=439.60, stdev=28.68, samples=5
  write: IOPS=458, BW=1835KiB/s (1879kB/s)(5192KiB/2830msec)
    slat (usec): min=13, max=786, avg=45.82, stdev=32.54
    clat (usec): min=12265, max=23456, avg=17219.55, stdev=1829.79
     lat (usec): min=12301, max=23543, avg=17266.56, stdev=1829.63
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14353], 10.00th=[15008], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22152], 99.50th=[22414], 99.90th=[23200], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1600, max= 1928, per=24.99%, avg=1843.20, stdev=136.77, samples=5
   iops        : min=  400, max=  482, avg=460.80, stdev=34.19, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=92.38%, 50=7.54%
  cpu          : usr=1.06%, sys=7.00%, ctx=2738, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=27604: Wed Dec 25 01:20:59 2024
   read: IOPS=440, BW=1763KiB/s (1805kB/s)(4976KiB/2823msec)
    slat (usec): min=629, max=2512, avg=1143.04, stdev=237.59
    clat (usec): min=3063, max=32755, avg=16705.19, stdev=2477.09
     lat (usec): min=4272, max=33664, avg=17849.98, stdev=2492.01
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13173], 10.00th=[13829], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[16057], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[22676], 99.50th=[24511], 99.90th=[28181], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1496, max= 1880, per=24.66%, avg=1747.20, stdev=149.73, samples=5
   iops        : min=  374, max=  470, avg=436.80, stdev=37.43, samples=5
  write: IOPS=466, BW=1865KiB/s (1909kB/s)(5264KiB/2823msec)
    slat (usec): min=13, max=290, avg=43.56, stdev=21.75
    clat (usec): min=6046, max=31777, avg=17163.54, stdev=2097.26
     lat (usec): min=6078, max=31808, avg=17208.29, stdev=2098.15
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13960], 10.00th=[14746], 20.00th=[15533],
     | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22414], 99.50th=[22414], 99.90th=[27395], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 1728, max= 1992, per=25.38%, avg=1872.00, stdev=104.77, samples=5
   iops        : min=  432, max=  498, avg=468.00, stdev=26.19, samples=5
  lat (msec)   : 4=0.04%, 10=0.27%, 20=91.45%, 50=8.24%
  cpu          : usr=0.85%, sys=6.98%, ctx=2647, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=27605: Wed Dec 25 01:20:59 2024
   read: IOPS=444, BW=1777KiB/s (1820kB/s)(4984KiB/2804msec)
    slat (usec): min=21, max=2640, avg=1130.89, stdev=255.95
    clat (usec): min=1840, max=25538, avg=16616.63, stdev=2526.38
     lat (usec): min=1869, max=27267, avg=17749.25, stdev=2546.93
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[13042], 10.00th=[13829], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[25560], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 1656, max= 1968, per=25.06%, avg=1776.00, stdev=122.11, samples=5
   iops        : min=  414, max=  492, avg=444.00, stdev=30.53, samples=5
  write: IOPS=468, BW=1874KiB/s (1919kB/s)(5256KiB/2804msec)
    slat (usec): min=13, max=193, avg=45.23, stdev=21.98
    clat (usec): min=4081, max=27839, avg=17165.11, stdev=2119.28
     lat (usec): min=4130, max=27888, avg=17211.43, stdev=2118.51
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14091], 10.00th=[14746], 20.00th=[15664],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[24249], 99.90th=[26084], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1720, max= 2000, per=25.31%, avg=1867.20, stdev=136.77, samples=5
   iops        : min=  430, max=  500, avg=466.80, stdev=34.19, samples=5
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.47%, 20=91.91%, 50=7.46%
  cpu          : usr=0.96%, sys=7.06%, ctx=2767, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=27606: Wed Dec 25 01:20:59 2024
   read: IOPS=446, BW=1787KiB/s (1830kB/s)(5060KiB/2832msec)
    slat (usec): min=640, max=2505, avg=1139.40, stdev=237.05
    clat (usec): min=1325, max=23438, avg=16697.41, stdev=2452.74
     lat (usec): min=3523, max=24760, avg=17838.54, stdev=2471.50
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[13173], 10.00th=[13829], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22152], 99.50th=[22676], 99.90th=[23200], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1560, max= 1896, per=24.91%, avg=1764.80, stdev=124.27, samples=5
   iops        : min=  390, max=  474, avg=441.20, stdev=31.07, samples=5
  write: IOPS=457, BW=1829KiB/s (1873kB/s)(5180KiB/2832msec)
    slat (usec): min=13, max=1096, avg=45.54, stdev=36.69
    clat (usec): min=3526, max=24043, avg=17192.57, stdev=2030.13
     lat (usec): min=3541, max=24085, avg=17239.30, stdev=2029.92
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14222], 10.00th=[14877], 20.00th=[15533],
     | 30.00th=[16188], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22152], 99.50th=[22414], 99.90th=[23462], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min= 1560, max= 1912, per=24.57%, avg=1812.80, stdev=143.62, samples=5
   iops        : min=  390, max=  478, avg=453.20, stdev=35.91, samples=5
  lat (msec)   : 2=0.04%, 4=0.23%, 10=0.23%, 20=91.29%, 50=8.20%
  cpu          : usr=1.02%, sys=6.89%, ctx=2731, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7086KiB/s (7256kB/s), 1763KiB/s-1787KiB/s (1805kB/s-1830kB/s), io=19.6MiB (20.5MB), run=2804-2832msec
  WRITE: bw=7377KiB/s (7554kB/s), 1829KiB/s-1874KiB/s (1873kB/s-1919kB/s), io=20.4MiB (21.4MB), run=2804-2832msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=27612: Wed Dec 25 01:21:01 2024
   read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=5514.4k, max=5514.4k, avg=5514404.00, stdev= 0.00
    clat (nsec): min=99686k, max=99686k, avg=99686347.00, stdev= 0.00
     lat (nsec): min=105212k, max=105212k, avg=105212131.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  101],  5.00th=[  101], 10.00th=[  101], 20.00th=[  101],
     | 30.00th=[  101], 40.00th=[  101], 50.00th=[  101], 60.00th=[  101],
     | 70.00th=[  101], 80.00th=[  101], 90.00th=[  101], 95.00th=[  101],
     | 99.00th=[  101], 99.50th=[  101], 99.90th=[  101], 99.95th=[  101],
     | 99.99th=[  101]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.92%, sys=4.59%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout