== sanity test 398c: run fio to test AIO ================= 19:20:39 (1717024839)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.073673 s, 569 MB/s
osc.lustre-OST0000-osc-ffff880137399800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=25273: Wed May 29 19:20:44 2024
  write: IOPS=786, BW=3148KiB/s (3223kB/s)(10.0MiB/3253msec)
    slat (usec): min=11, max=310, avg=41.90, stdev=22.35
    clat (usec): min=4574, max=37586, avg=20204.98, stdev=2541.47
     lat (usec): min=4614, max=37626, avg=20248.03, stdev=2542.85
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[16712], 10.00th=[18220], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[19792], 50.00th=[20317], 60.00th=[20579],
     | 70.00th=[20841], 80.00th=[21627], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[27132], 99.50th=[31327], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[37487]
   bw (  KiB/s): min= 2984, max= 3384, per=24.99%, avg=3142.67, stdev=141.35, samples=6
   iops        : min=  746, max=  846, avg=785.67, stdev=35.34, samples=6
  lat (msec)   : 10=0.04%, 20=44.88%, 50=55.08%
  cpu          : usr=0.68%, sys=3.84%, ctx=1300, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25274: Wed May 29 19:20:44 2024
  write: IOPS=789, BW=3157KiB/s (3232kB/s)(10.0MiB/3244msec)
    slat (usec): min=10, max=268, avg=43.41, stdev=23.42
    clat (usec): min=4308, max=37582, avg=20151.71, stdev=2759.81
     lat (usec): min=4362, max=37637, avg=20196.24, stdev=2759.94
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[15795], 10.00th=[18220], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[19792], 50.00th=[20317], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21627], 90.00th=[22414], 95.00th=[23200],
     | 99.00th=[26608], 99.50th=[35914], 99.90th=[36963], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 3000, max= 3496, per=25.14%, avg=3161.33, stdev=181.33, samples=6
   iops        : min=  750, max=  874, avg=790.33, stdev=45.33, samples=6
  lat (msec)   : 10=0.39%, 20=44.22%, 50=55.39%
  cpu          : usr=0.43%, sys=4.13%, ctx=1402, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25275: Wed May 29 19:20:44 2024
  write: IOPS=786, BW=3146KiB/s (3221kB/s)(10.0MiB/3255msec)
    slat (usec): min=11, max=632, avg=45.97, stdev=29.17
    clat (usec): min=6444, max=37191, avg=20209.63, stdev=2490.92
     lat (usec): min=6514, max=37226, avg=20256.78, stdev=2491.02
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[16909], 10.00th=[18220], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[19792], 50.00th=[20317], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21627], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[26870], 99.50th=[31065], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 2984, max= 3392, per=24.97%, avg=3140.00, stdev=145.66, samples=6
   iops        : min=  746, max=  848, avg=785.00, stdev=36.41, samples=6
  lat (msec)   : 10=0.08%, 20=44.45%, 50=55.47%
  cpu          : usr=0.77%, sys=3.90%, ctx=1731, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25276: Wed May 29 19:20:44 2024
  write: IOPS=785, BW=3144KiB/s (3219kB/s)(10.0MiB/3257msec)
    slat (usec): min=12, max=369, avg=43.65, stdev=22.26
    clat (usec): min=10310, max=37564, avg=20227.14, stdev=2385.89
     lat (usec): min=10360, max=37609, avg=20271.93, stdev=2386.01
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[16909], 10.00th=[18220], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[19792], 50.00th=[20317], 60.00th=[20579],
     | 70.00th=[21103], 80.00th=[21627], 90.00th=[22414], 95.00th=[23200],
     | 99.00th=[26084], 99.50th=[34341], 99.90th=[35914], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 2976, max= 3352, per=24.94%, avg=3136.00, stdev=136.24, samples=6
   iops        : min=  744, max=  838, avg=784.00, stdev=34.06, samples=6
  lat (msec)   : 20=44.49%, 50=55.51%
  cpu          : usr=0.61%, sys=3.87%, ctx=1523, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.3MiB/s (12.9MB/s), 3144KiB/s-3157KiB/s (3219kB/s-3232kB/s), io=40.0MiB (41.9MB), run=3244-3257msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=25289: Wed May 29 19:20:48 2024
   read: IOPS=406, BW=1627KiB/s (1666kB/s)(5048KiB/3103msec)
    slat (usec): min=13, max=3492, avg=1222.61, stdev=340.05
    clat (usec): min=1902, max=38021, avg=18581.92, stdev=3615.75
     lat (usec): min=3018, max=39899, avg=19806.36, stdev=3743.26
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[13435], 10.00th=[14353], 20.00th=[15533],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[18220], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21365], 90.00th=[23200], 95.00th=[25035],
     | 99.00th=[27395], 99.50th=[28705], 99.90th=[35914], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1376, max= 1800, per=25.01%, avg=1614.00, stdev=163.36, samples=6
   iops        : min=  344, max=  450, avg=403.50, stdev=40.84, samples=6
  write: IOPS=418, BW=1673KiB/s (1713kB/s)(5192KiB/3103msec)
    slat (usec): min=12, max=352, avg=45.42, stdev=27.98
    clat (usec): min=5156, max=35767, avg=18914.25, stdev=3321.75
     lat (usec): min=5258, max=35803, avg=18960.80, stdev=3324.86
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14353], 10.00th=[15139], 20.00th=[16319],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20317], 80.00th=[21365], 90.00th=[22938], 95.00th=[24249],
     | 99.00th=[29754], 99.50th=[33424], 99.90th=[35390], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1536, max= 2008, per=24.68%, avg=1658.00, stdev=176.94, samples=6
   iops        : min=  384, max=  502, avg=414.50, stdev=44.23, samples=6
  lat (msec)   : 2=0.04%, 10=0.59%, 20=66.91%, 50=32.46%
  cpu          : usr=0.52%, sys=7.06%, ctx=2465, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25290: Wed May 29 19:20:48 2024
   read: IOPS=400, BW=1601KiB/s (1639kB/s)(4976KiB/3109msec)
    slat (usec): min=14, max=3007, avg=1237.06, stdev=329.99
    clat (usec): min=9391, max=37478, avg=18576.65, stdev=3450.65
     lat (usec): min=10693, max=39170, avg=19815.54, stdev=3570.96
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13435], 10.00th=[14353], 20.00th=[15533],
     | 30.00th=[16712], 40.00th=[17695], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22938], 95.00th=[24249],
     | 99.00th=[27395], 99.50th=[30540], 99.90th=[34866], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1432, max= 1792, per=24.59%, avg=1587.33, stdev=141.00, samples=6
   iops        : min=  358, max=  448, avg=396.83, stdev=35.25, samples=6
  write: IOPS=423, BW=1693KiB/s (1734kB/s)(5264KiB/3109msec)
    slat (usec): min=12, max=1301, avg=47.73, stdev=51.88
    clat (usec): min=6894, max=36355, avg=18977.25, stdev=3309.75
     lat (usec): min=6925, max=36414, avg=19026.09, stdev=3312.11
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[14353], 10.00th=[15139], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18744], 60.00th=[19530],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22938], 95.00th=[23987],
     | 99.00th=[29492], 99.50th=[32375], 99.90th=[35914], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 1448, max= 2032, per=25.09%, avg=1686.00, stdev=191.60, samples=6
   iops        : min=  362, max=  508, avg=421.50, stdev=47.90, samples=6
  lat (msec)   : 10=0.31%, 20=66.13%, 50=33.55%
  cpu          : usr=1.06%, sys=6.56%, ctx=2332, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25291: Wed May 29 19:20:48 2024
   read: IOPS=400, BW=1603KiB/s (1642kB/s)(4984KiB/3109msec)
    slat (usec): min=11, max=2954, avg=1218.83, stdev=328.57
    clat (usec): min=4434, max=33581, avg=18464.40, stdev=3448.18
     lat (usec): min=5957, max=34773, avg=19685.28, stdev=3556.63
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13173], 10.00th=[14353], 20.00th=[15533],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22676], 95.00th=[24773],
     | 99.00th=[27657], 99.50th=[29492], 99.90th=[32113], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1376, max= 1936, per=24.76%, avg=1598.17, stdev=195.51, samples=6
   iops        : min=  344, max=  484, avg=399.50, stdev=48.92, samples=6
  write: IOPS=422, BW=1691KiB/s (1731kB/s)(5256KiB/3109msec)
    slat (usec): min=11, max=305, avg=45.32, stdev=25.01
    clat (usec): min=6860, max=35973, avg=19105.63, stdev=3422.86
     lat (usec): min=6933, max=36044, avg=19152.10, stdev=3425.01
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[14484], 10.00th=[15270], 20.00th=[16319],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18744], 60.00th=[19530],
     | 70.00th=[20579], 80.00th=[21890], 90.00th=[23462], 95.00th=[25035],
     | 99.00th=[30016], 99.50th=[32113], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1512, max= 1888, per=24.85%, avg=1670.00, stdev=151.95, samples=6
   iops        : min=  378, max=  472, avg=417.50, stdev=37.99, samples=6
  lat (msec)   : 10=0.31%, 20=66.99%, 50=32.70%
  cpu          : usr=0.71%, sys=7.01%, ctx=2466, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25292: Wed May 29 19:20:48 2024
   read: IOPS=406, BW=1628KiB/s (1667kB/s)(5060KiB/3109msec)
    slat (usec): min=12, max=3038, avg=1213.70, stdev=329.56
    clat (usec): min=4380, max=38015, avg=18481.88, stdev=3472.45
     lat (usec): min=5949, max=39297, avg=19697.52, stdev=3582.17
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13304], 10.00th=[14222], 20.00th=[15533],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[18220], 60.00th=[19006],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22938], 95.00th=[23987],
     | 99.00th=[28181], 99.50th=[28967], 99.90th=[35914], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1376, max= 1864, per=24.95%, avg=1610.00, stdev=164.69, samples=6
   iops        : min=  344, max=  466, avg=402.50, stdev=41.17, samples=6
  write: IOPS=416, BW=1666KiB/s (1706kB/s)(5180KiB/3109msec)
    slat (usec): min=13, max=1356, avg=46.55, stdev=45.50
    clat (usec): min=4267, max=35804, avg=19082.69, stdev=3458.56
     lat (usec): min=4281, max=35827, avg=19130.48, stdev=3461.46
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14353], 10.00th=[15270], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18744], 60.00th=[19530],
     | 70.00th=[20317], 80.00th=[21365], 90.00th=[23462], 95.00th=[25297],
     | 99.00th=[30278], 99.50th=[33162], 99.90th=[34341], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1520, max= 1960, per=24.64%, avg=1655.33, stdev=163.22, samples=6
   iops        : min=  380, max=  490, avg=413.83, stdev=40.80, samples=6
  lat (msec)   : 10=0.31%, 20=67.58%, 50=32.11%
  cpu          : usr=0.48%, sys=7.18%, ctx=2456, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6455KiB/s (6610kB/s), 1601KiB/s-1628KiB/s (1639kB/s-1667kB/s), io=19.6MiB (20.5MB), run=3103-3109msec
  WRITE: bw=6720KiB/s (6881kB/s), 1666KiB/s-1693KiB/s (1706kB/s-1734kB/s), io=20.4MiB (21.4MB), run=3103-3109msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=25298: Wed May 29 19:20:49 2024
   read: IOPS=11, BW=460MiB/s (482MB/s)(40.0MiB/87msec)
    slat (nsec): min=6384.9k, max=6384.9k, avg=6384877.00, stdev= 0.00
    clat (nsec): min=79463k, max=79463k, avg=79462938.00, stdev= 0.00
     lat (nsec): min=85873k, max=85873k, avg=85872718.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[79168],  5.00th=[79168], 10.00th=[79168], 20.00th=[79168],
     | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168],
     | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168],
     | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.98%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=460MiB/s (482MB/s), 460MiB/s-460MiB/s (482MB/s-482MB/s), io=40.0MiB (41.9MB), run=87-87msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout