== sanity test 398c: run fio to test AIO ================= 04:49:30 (1733651370)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0840375 s, 499 MB/s
osc.lustre-OST0000-osc-ffff8800a9b97800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=22928: Sun Dec  8 04:49:35 2024
  write: IOPS=782, BW=3130KiB/s (3205kB/s)(10.0MiB/3272msec)
    slat (usec): min=12, max=514, avg=45.33, stdev=26.17
    clat (usec): min=5152, max=40115, avg=20264.74, stdev=2592.19
     lat (usec): min=5213, max=40137, avg=20311.26, stdev=2594.89
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[16188], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20317], 60.00th=[20841],
     | 70.00th=[21365], 80.00th=[21890], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[24773], 99.50th=[35914], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2968, max= 3376, per=25.18%, avg=3152.00, stdev=166.58, samples=6
   iops        : min=  742, max=  844, avg=788.00, stdev=41.65, samples=6
  lat (msec)   : 10=0.04%, 20=44.18%, 50=55.78%
  cpu          : usr=0.86%, sys=3.58%, ctx=1452, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22929: Sun Dec  8 04:49:35 2024
  write: IOPS=785, BW=3142KiB/s (3217kB/s)(10.0MiB/3259msec)
    slat (usec): min=11, max=363, avg=48.07, stdev=24.48
    clat (usec): min=2423, max=39685, avg=20221.70, stdev=2806.24
     lat (usec): min=2472, max=39765, avg=20271.03, stdev=2806.81
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15926], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20317], 60.00th=[20841],
     | 70.00th=[21627], 80.00th=[21890], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[26608], 99.50th=[36963], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 2992, max= 3504, per=25.36%, avg=3174.83, stdev=206.62, samples=6
   iops        : min=  748, max=  876, avg=793.67, stdev=51.68, samples=6
  lat (msec)   : 4=0.04%, 10=0.23%, 20=42.89%, 50=56.84%
  cpu          : usr=0.64%, sys=4.14%, ctx=1689, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22930: Sun Dec  8 04:49:35 2024
  write: IOPS=783, BW=3133KiB/s (3209kB/s)(10.0MiB/3268msec)
    slat (usec): min=12, max=340, avg=49.16, stdev=24.91
    clat (usec): min=4172, max=40067, avg=20269.39, stdev=2719.20
     lat (usec): min=4235, max=40111, avg=20319.89, stdev=2720.37
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[16188], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20317], 60.00th=[20841],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22676], 95.00th=[23200],
     | 99.00th=[30016], 99.50th=[34341], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2992, max= 3440, per=25.30%, avg=3166.83, stdev=191.84, samples=6
   iops        : min=  748, max=  860, avg=791.67, stdev=47.99, samples=6
  lat (msec)   : 10=0.12%, 20=43.95%, 50=55.94%
  cpu          : usr=0.98%, sys=4.10%, ctx=1777, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=22931: Sun Dec  8 04:49:35 2024
  write: IOPS=782, BW=3131KiB/s (3207kB/s)(10.0MiB/3270msec)
    slat (usec): min=13, max=880, avg=49.67, stdev=31.17
    clat (usec): min=4224, max=40042, avg=20264.40, stdev=2695.88
     lat (usec): min=4281, max=40108, avg=20315.35, stdev=2697.25
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[16450], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22152], 90.00th=[22676], 95.00th=[22938],
     | 99.00th=[28443], 99.50th=[34341], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2984, max= 3424, per=25.24%, avg=3160.17, stdev=183.43, samples=6
   iops        : min=  746, max=  856, avg=790.00, stdev=45.89, samples=6
  lat (msec)   : 10=0.04%, 20=44.18%, 50=55.78%
  cpu          : usr=0.64%, sys=4.04%, ctx=1875, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.2MiB/s (12.8MB/s), 3130KiB/s-3142KiB/s (3205kB/s-3217kB/s), io=40.0MiB (41.9MB), run=3259-3272msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=22943: Sun Dec  8 04:49:39 2024
   read: IOPS=432, BW=1728KiB/s (1770kB/s)(5048KiB/2921msec)
    slat (usec): min=714, max=2665, avg=1179.83, stdev=226.84
    clat (usec): min=5584, max=32028, avg=17169.37, stdev=2316.77
     lat (usec): min=6785, max=33116, avg=18351.00, stdev=2322.30
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13566], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22676], 99.50th=[23462], 99.90th=[31065], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1600, max= 1800, per=25.06%, avg=1720.00, stdev=73.32, samples=5
   iops        : min=  400, max=  450, avg=430.00, stdev=18.33, samples=5
  write: IOPS=444, BW=1777KiB/s (1820kB/s)(5192KiB/2921msec)
    slat (usec): min=12, max=946, avg=46.65, stdev=35.84
    clat (usec): min=12418, max=31059, avg=17847.16, stdev=1716.10
     lat (usec): min=12497, max=31095, avg=17895.14, stdev=1715.70
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15270], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22152], 99.50th=[23200], 99.90th=[28181], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1704, max= 1928, per=25.15%, avg=1796.80, stdev=88.47, samples=5
   iops        : min=  426, max=  482, avg=449.20, stdev=22.12, samples=5
  lat (msec)   : 10=0.04%, 20=90.66%, 50=9.30%
  cpu          : usr=0.68%, sys=7.50%, ctx=2255, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22944: Sun Dec  8 04:49:39 2024
   read: IOPS=425, BW=1704KiB/s (1744kB/s)(4976KiB/2921msec)
    slat (usec): min=729, max=2542, avg=1183.27, stdev=235.56
    clat (usec): min=4550, max=37893, avg=17304.64, stdev=2559.38
     lat (usec): min=5995, max=39308, avg=18489.67, stdev=2565.91
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13698], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20317], 95.00th=[21365],
     | 99.00th=[23725], 99.50th=[26346], 99.90th=[36963], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1568, max= 1752, per=24.78%, avg=1700.80, stdev=78.91, samples=5
   iops        : min=  392, max=  438, avg=425.20, stdev=19.73, samples=5
  write: IOPS=450, BW=1802KiB/s (1845kB/s)(5264KiB/2921msec)
    slat (usec): min=13, max=714, avg=46.93, stdev=34.00
    clat (usec): min=6385, max=37833, avg=17812.56, stdev=2000.14
     lat (usec): min=6423, max=37886, avg=17860.70, stdev=1999.28
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14877], 10.00th=[15664], 20.00th=[16319],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22152], 99.50th=[22676], 99.90th=[32113], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1736, max= 1944, per=25.66%, avg=1833.60, stdev=84.36, samples=5
   iops        : min=  434, max=  486, avg=458.40, stdev=21.09, samples=5
  lat (msec)   : 10=0.20%, 20=88.16%, 50=11.64%
  cpu          : usr=0.79%, sys=7.12%, ctx=2277, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22945: Sun Dec  8 04:49:39 2024
   read: IOPS=427, BW=1708KiB/s (1749kB/s)(4984KiB/2918msec)
    slat (usec): min=22, max=3160, avg=1173.21, stdev=258.85
    clat (usec): min=1879, max=27544, avg=17247.25, stdev=2619.00
     lat (usec): min=1903, max=29302, avg=18422.15, stdev=2654.66
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16712], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[21365],
     | 99.00th=[23462], 99.50th=[23725], 99.90th=[26346], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1544, max= 1944, per=24.95%, avg=1712.00, stdev=156.15, samples=5
   iops        : min=  386, max=  486, avg=428.00, stdev=39.04, samples=5
  write: IOPS=450, BW=1801KiB/s (1844kB/s)(5256KiB/2918msec)
    slat (usec): min=13, max=690, avg=46.62, stdev=28.13
    clat (usec): min=5885, max=32805, avg=17932.46, stdev=2033.31
     lat (usec): min=5923, max=32854, avg=17980.24, stdev=2032.45
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15008], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19530], 90.00th=[20317], 95.00th=[20841],
     | 99.00th=[22938], 99.50th=[24773], 99.90th=[32637], 99.95th=[32900],
     | 99.99th=[32900]
   bw (  KiB/s): min= 1736, max= 1968, per=25.57%, avg=1827.20, stdev=103.48, samples=5
   iops        : min=  434, max=  492, avg=456.80, stdev=25.87, samples=5
  lat (msec)   : 2=0.08%, 4=0.08%, 10=0.20%, 20=87.89%, 50=11.76%
  cpu          : usr=0.72%, sys=7.20%, ctx=2260, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=22946: Sun Dec  8 04:49:39 2024
   read: IOPS=432, BW=1731KiB/s (1772kB/s)(5060KiB/2924msec)
    slat (usec): min=718, max=2255, avg=1171.02, stdev=228.68
    clat (usec): min=4952, max=23661, avg=17170.06, stdev=2206.21
     lat (usec): min=6335, max=24671, avg=18342.88, stdev=2228.30
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13698], 10.00th=[14353], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22152], 99.50th=[22414], 99.90th=[23462], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 1640, max= 1824, per=25.16%, avg=1726.40, stdev=78.06, samples=5
   iops        : min=  410, max=  456, avg=431.60, stdev=19.51, samples=5
  write: IOPS=442, BW=1772KiB/s (1814kB/s)(5180KiB/2924msec)
    slat (usec): min=13, max=1198, avg=46.97, stdev=42.17
    clat (usec): min=9004, max=24520, avg=17866.70, stdev=1768.25
     lat (usec): min=9046, max=24628, avg=17914.87, stdev=1768.96
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15139], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[16909], 40.00th=[17433], 50.00th=[17957], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20317], 95.00th=[20579],
     | 99.00th=[21890], 99.50th=[22938], 99.90th=[24511], 99.95th=[24511],
     | 99.99th=[24511]
   bw (  KiB/s): min= 1640, max= 1928, per=24.95%, avg=1782.40, stdev=103.44, samples=5
   iops        : min=  410, max=  482, avg=445.60, stdev=25.86, samples=5
  lat (msec)   : 10=0.12%, 20=89.61%, 50=10.27%
  cpu          : usr=1.03%, sys=6.98%, ctx=2294, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6863KiB/s (7028kB/s), 1704KiB/s-1731KiB/s (1744kB/s-1772kB/s), io=19.6MiB (20.5MB), run=2918-2924msec
  WRITE: bw=7145KiB/s (7316kB/s), 1772KiB/s-1802KiB/s (1814kB/s-1845kB/s), io=20.4MiB (21.4MB), run=2918-2924msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=22952: Sun Dec  8 04:49:40 2024
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=5124.9k, max=5124.9k, avg=5124902.00, stdev= 0.00
    clat (nsec): min=66890k, max=66890k, avg=66890011.00, stdev= 0.00
     lat (nsec): min=72025k, max=72025k, avg=72025448.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[66847],  5.00th=[66847], 10.00th=[66847], 20.00th=[66847],
     | 30.00th=[66847], 40.00th=[66847], 50.00th=[66847], 60.00th=[66847],
     | 70.00th=[66847], 80.00th=[66847], 90.00th=[66847], 95.00th=[66847],
     | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847],
     | 99.99th=[66847]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.67%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout