== sanity test 398c: run fio to test AIO ================= 12:56:20 (1729097780)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.144493 s, 290 MB/s
osc.lustre-OST0000-osc-ffff88007aa5c000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=12992: Wed Oct 16 12:56:29 2024
  write: IOPS=370, BW=1481KiB/s (1517kB/s)(10.0MiB/6912msec)
    slat (usec): min=17, max=543, avg=74.69, stdev=46.10
    clat (usec): min=4933, max=74201, avg=42960.11, stdev=5193.84
     lat (usec): min=5015, max=74270, avg=43036.55, stdev=5200.48
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[33162], 10.00th=[36439], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[43254], 60.00th=[44827],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[49021], 95.00th=[50070],
     | 99.00th=[53216], 99.50th=[56361], 99.90th=[62129], 99.95th=[65274],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1317, max= 1664, per=25.01%, avg=1481.15, stdev=112.61, samples=13
   iops        : min=  329, max=  416, avg=370.23, stdev=28.21, samples=13
  lat (msec)   : 10=0.04%, 50=94.06%, 100=5.90%
  cpu          : usr=0.55%, sys=3.01%, ctx=1304, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=12993: Wed Oct 16 12:56:29 2024
  write: IOPS=371, BW=1487KiB/s (1522kB/s)(10.0MiB/6888msec)
    slat (usec): min=17, max=360, avg=82.96, stdev=46.62
    clat (usec): min=4204, max=81550, avg=42859.64, stdev=6137.38
     lat (usec): min=4319, max=81659, avg=42944.36, stdev=6136.10
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[32900], 10.00th=[35914], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[43254], 60.00th=[44827],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[53740], 99.50th=[61604], 99.90th=[76022], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1365, max= 1720, per=25.23%, avg=1494.08, stdev=119.42, samples=13
   iops        : min=  341, max=  430, avg=373.46, stdev=29.91, samples=13
  lat (msec)   : 10=0.27%, 20=0.16%, 50=92.23%, 100=7.34%
  cpu          : usr=0.35%, sys=3.85%, ctx=1671, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=12994: Wed Oct 16 12:56:29 2024
  write: IOPS=370, BW=1483KiB/s (1518kB/s)(10.0MiB/6907msec)
    slat (usec): min=18, max=414, avg=71.73, stdev=44.17
    clat (usec): min=25222, max=74817, avg=42973.80, stdev=5322.26
     lat (usec): min=25326, max=74890, avg=43047.20, stdev=5324.91
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33817], 10.00th=[35914], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[43779], 60.00th=[44827],
     | 70.00th=[45876], 80.00th=[47449], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[53216], 99.50th=[56361], 99.90th=[56361], 99.95th=[65274],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1293, max= 1664, per=25.03%, avg=1482.38, stdev=126.55, samples=13
   iops        : min=  323, max=  416, avg=370.54, stdev=31.70, samples=13
  lat (msec)   : 50=93.24%, 100=6.76%
  cpu          : usr=0.45%, sys=2.97%, ctx=1195, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=12995: Wed Oct 16 12:56:29 2024
  write: IOPS=370, BW=1481KiB/s (1516kB/s)(10.0MiB/6916msec)
    slat (usec): min=20, max=498, avg=71.60, stdev=43.79
    clat (usec): min=5443, max=73854, avg=42955.04, stdev=5304.41
     lat (usec): min=5568, max=73890, avg=43028.21, stdev=5308.50
    clat percentiles (usec):
     |  1.00th=[30540],  5.00th=[33817], 10.00th=[36439], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[43254], 60.00th=[44827],
     | 70.00th=[45876], 80.00th=[46924], 90.00th=[49021], 95.00th=[50070],
     | 99.00th=[53216], 99.50th=[53740], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1336, max= 1664, per=25.04%, avg=1482.77, stdev=101.89, samples=13
   iops        : min=  334, max=  416, avg=370.69, stdev=25.47, samples=13
  lat (msec)   : 10=0.16%, 20=0.08%, 50=94.18%, 100=5.59%
  cpu          : usr=0.39%, sys=3.07%, ctx=1082, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5922KiB/s (6065kB/s), 1481KiB/s-1487KiB/s (1516kB/s-1522kB/s), io=40.0MiB (41.9MB), run=6888-6916msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=13010: Wed Oct 16 12:56:33 2024
   read: IOPS=341, BW=1366KiB/s (1399kB/s)(5048KiB/3696msec)
    slat (usec): min=25, max=3681, avg=1562.30, stdev=419.06
    clat (usec): min=2374, max=43336, avg=21784.14, stdev=5066.27
     lat (usec): min=3261, max=46571, avg=23349.73, stdev=5262.93
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[15926], 10.00th=[16909], 20.00th=[17957],
     | 30.00th=[19006], 40.00th=[20055], 50.00th=[20841], 60.00th=[22152],
     | 70.00th=[23200], 80.00th=[24773], 90.00th=[28443], 95.00th=[31851],
     | 99.00th=[38011], 99.50th=[40633], 99.90th=[41681], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1008, max= 1592, per=25.25%, avg=1360.00, stdev=215.26, samples=7
   iops        : min=  252, max=  398, avg=340.00, stdev=53.81, samples=7
  write: IOPS=351, BW=1405KiB/s (1438kB/s)(5192KiB/3696msec)
    slat (usec): min=16, max=2688, avg=84.59, stdev=86.12
    clat (usec): min=5972, max=43381, avg=22619.16, stdev=4458.57
     lat (usec): min=6219, max=43454, avg=22705.82, stdev=4461.98
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17171], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20317], 40.00th=[21103], 50.00th=[21890], 60.00th=[22938],
     | 70.00th=[23725], 80.00th=[25297], 90.00th=[27919], 95.00th=[31589],
     | 99.00th=[36963], 99.50th=[40633], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  968, max= 1576, per=24.96%, avg=1400.00, stdev=206.61, samples=7
   iops        : min=  242, max=  394, avg=350.00, stdev=51.65, samples=7
  lat (msec)   : 4=0.20%, 10=0.20%, 20=33.83%, 50=65.78%
  cpu          : usr=1.35%, sys=9.53%, ctx=2895, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13011: Wed Oct 16 12:56:33 2024
   read: IOPS=336, BW=1346KiB/s (1378kB/s)(4976KiB/3698msec)
    slat (usec): min=750, max=3451, avg=1582.21, stdev=402.75
    clat (usec): min=2536, max=45700, avg=21628.66, stdev=4629.91
     lat (usec): min=4256, max=47954, avg=23214.45, stdev=4808.79
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[15926], 10.00th=[16909], 20.00th=[18220],
     | 30.00th=[19006], 40.00th=[19792], 50.00th=[20841], 60.00th=[21890],
     | 70.00th=[22938], 80.00th=[24249], 90.00th=[27395], 95.00th=[30540],
     | 99.00th=[38536], 99.50th=[40633], 99.90th=[42730], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min=  872, max= 1584, per=24.67%, avg=1329.14, stdev=240.77, samples=7
   iops        : min=  218, max=  396, avg=332.29, stdev=60.19, samples=7
  write: IOPS=355, BW=1423KiB/s (1458kB/s)(5264KiB/3698msec)
    slat (usec): min=13, max=1524, avg=85.67, stdev=74.88
    clat (usec): min=5775, max=41698, avg=22631.29, stdev=4415.02
     lat (usec): min=5894, max=41756, avg=22718.88, stdev=4415.64
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[16909], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[20055], 40.00th=[21103], 50.00th=[21890], 60.00th=[22938],
     | 70.00th=[23987], 80.00th=[25560], 90.00th=[27657], 95.00th=[31065],
     | 99.00th=[38011], 99.50th=[39060], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1008, max= 1576, per=25.47%, avg=1428.57, stdev=206.60, samples=7
   iops        : min=  252, max=  394, avg=357.14, stdev=51.65, samples=7
  lat (msec)   : 4=0.04%, 10=0.08%, 20=34.92%, 50=64.96%
  cpu          : usr=1.08%, sys=9.76%, ctx=2809, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13012: Wed Oct 16 12:56:33 2024
   read: IOPS=334, BW=1338KiB/s (1370kB/s)(4984KiB/3725msec)
    slat (usec): min=670, max=3439, avg=1570.11, stdev=381.47
    clat (usec): min=1228, max=40205, avg=21786.34, stdev=4479.27
     lat (usec): min=2909, max=42243, avg=23359.91, stdev=4631.78
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15926], 10.00th=[16909], 20.00th=[18220],
     | 30.00th=[19268], 40.00th=[20055], 50.00th=[21103], 60.00th=[22152],
     | 70.00th=[23200], 80.00th=[24773], 90.00th=[27395], 95.00th=[31327],
     | 99.00th=[35914], 99.50th=[36963], 99.90th=[38536], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min=  880, max= 1512, per=24.78%, avg=1334.86, stdev=211.53, samples=7
   iops        : min=  220, max=  378, avg=333.71, stdev=52.88, samples=7
  write: IOPS=352, BW=1411KiB/s (1445kB/s)(5256KiB/3725msec)
    slat (usec): min=14, max=788, avg=84.68, stdev=51.79
    clat (usec): min=13057, max=40689, avg=22651.16, stdev=4394.78
     lat (usec): min=13134, max=40722, avg=22737.83, stdev=4393.19
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17171], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20317], 40.00th=[21103], 50.00th=[21890], 60.00th=[22676],
     | 70.00th=[23725], 80.00th=[25035], 90.00th=[28181], 95.00th=[32375],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[39060], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min=  920, max= 1616, per=24.82%, avg=1392.00, stdev=261.69, samples=7
   iops        : min=  230, max=  404, avg=348.00, stdev=65.42, samples=7
  lat (msec)   : 2=0.04%, 20=32.97%, 50=66.99%
  cpu          : usr=1.75%, sys=9.16%, ctx=2912, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13013: Wed Oct 16 12:56:33 2024
   read: IOPS=340, BW=1360KiB/s (1393kB/s)(5060KiB/3720msec)
    slat (usec): min=703, max=4118, avg=1578.25, stdev=418.08
    clat (usec): min=1332, max=43353, avg=21765.21, stdev=4639.83
     lat (usec): min=2604, max=46207, avg=23346.71, stdev=4835.21
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15533], 10.00th=[16909], 20.00th=[18220],
     | 30.00th=[19268], 40.00th=[20055], 50.00th=[21103], 60.00th=[22152],
     | 70.00th=[23200], 80.00th=[24773], 90.00th=[27395], 95.00th=[30802],
     | 99.00th=[37487], 99.50th=[38011], 99.90th=[41157], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  904, max= 1576, per=25.10%, avg=1352.00, stdev=231.91, samples=7
   iops        : min=  226, max=  394, avg=338.00, stdev=57.98, samples=7
  write: IOPS=348, BW=1392KiB/s (1426kB/s)(5180KiB/3720msec)
    slat (usec): min=16, max=407, avg=80.68, stdev=44.05
    clat (usec): min=13638, max=43522, avg=22666.89, stdev=4260.30
     lat (usec): min=13698, max=43630, avg=22749.28, stdev=4260.92
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[16909], 10.00th=[17957], 20.00th=[19530],
     | 30.00th=[20317], 40.00th=[21103], 50.00th=[22152], 60.00th=[22676],
     | 70.00th=[23987], 80.00th=[25297], 90.00th=[28181], 95.00th=[32113],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[41681], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  928, max= 1560, per=24.70%, avg=1385.14, stdev=224.54, samples=7
   iops        : min=  232, max=  390, avg=346.29, stdev=56.14, samples=7
  lat (msec)   : 2=0.04%, 10=0.04%, 20=31.80%, 50=68.12%
  cpu          : usr=1.10%, sys=9.52%, ctx=2823, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5387KiB/s (5517kB/s), 1338KiB/s-1366KiB/s (1370kB/s-1399kB/s), io=19.6MiB (20.5MB), run=3696-3725msec
  WRITE: bw=5609KiB/s (5743kB/s), 1392KiB/s-1423KiB/s (1426kB/s-1458kB/s), io=20.4MiB (21.4MB), run=3696-3725msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=13020: Wed Oct 16 12:56:35 2024
   read: IOPS=7, BW=313MiB/s (328MB/s)(40.0MiB/128msec)
    slat (nsec): min=5893.8k, max=5893.8k, avg=5893752.00, stdev= 0.00
    clat (nsec): min=114278k, max=114278k, avg=114277656.00, stdev= 0.00
     lat (nsec): min=120182k, max=120182k, avg=120182283.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  114],  5.00th=[  114], 10.00th=[  114], 20.00th=[  114],
     | 30.00th=[  114], 40.00th=[  114], 50.00th=[  114], 60.00th=[  114],
     | 70.00th=[  114], 80.00th=[  114], 90.00th=[  114], 95.00th=[  114],
     | 99.00th=[  114], 99.50th=[  114], 99.90th=[  114], 99.95th=[  114],
     | 99.99th=[  114]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.72%, ctx=6, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=40.0MiB (41.9MB), run=128-128msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout