== sanity test 398c: run fio to test AIO ================= 08:23:51 (1744028631)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.095789 s, 438 MB/s
osc.lustre-OST0000-osc-ffff880097de4000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=504: Mon Apr  7 08:23:56 2025
  write: IOPS=677, BW=2709KiB/s (2774kB/s)(10.0MiB/3780msec)
    slat (usec): min=13, max=487, avg=65.74, stdev=46.96
    clat (usec): min=5689, max=51791, avg=23486.31, stdev=4917.99
     lat (usec): min=5831, max=51926, avg=23553.53, stdev=4927.99
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[14091], 10.00th=[17171], 20.00th=[20579],
     | 30.00th=[21365], 40.00th=[22676], 50.00th=[23462], 60.00th=[24249],
     | 70.00th=[25035], 80.00th=[27132], 90.00th=[29754], 95.00th=[32113],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[48497], 99.95th=[48497],
     | 99.99th=[51643]
   bw (  KiB/s): min= 2424, max= 3200, per=25.43%, avg=2749.71, stdev=286.84, samples=7
   iops        : min=  606, max=  800, avg=687.43, stdev=71.71, samples=7
  lat (msec)   : 10=0.43%, 20=14.18%, 50=85.35%, 100=0.04%
  cpu          : usr=0.98%, sys=4.66%, ctx=1781, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=505: Mon Apr  7 08:23:56 2025
  write: IOPS=676, BW=2706KiB/s (2771kB/s)(10.0MiB/3784msec)
    slat (usec): min=12, max=473, avg=66.11, stdev=44.91
    clat (usec): min=2931, max=51972, avg=23494.30, stdev=5040.65
     lat (usec): min=2990, max=52096, avg=23561.90, stdev=5052.79
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[14091], 10.00th=[16909], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23462], 60.00th=[23987],
     | 70.00th=[25035], 80.00th=[27132], 90.00th=[30016], 95.00th=[32113],
     | 99.00th=[35390], 99.50th=[37487], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[52167]
   bw (  KiB/s): min= 2432, max= 3200, per=25.45%, avg=2752.00, stdev=287.44, samples=7
   iops        : min=  608, max=  800, avg=688.00, stdev=71.86, samples=7
  lat (msec)   : 4=0.04%, 10=0.90%, 20=13.98%, 50=84.96%, 100=0.12%
  cpu          : usr=0.77%, sys=4.86%, ctx=1736, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=506: Mon Apr  7 08:23:56 2025
  write: IOPS=675, BW=2704KiB/s (2769kB/s)(10.0MiB/3787msec)
    slat (usec): min=13, max=532, avg=62.66, stdev=42.74
    clat (usec): min=2448, max=43331, avg=23500.72, stdev=4823.56
     lat (usec): min=2499, max=43364, avg=23564.74, stdev=4836.36
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[14353], 10.00th=[17695], 20.00th=[20317],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23200], 60.00th=[24249],
     | 70.00th=[25035], 80.00th=[27132], 90.00th=[30016], 95.00th=[32113],
     | 99.00th=[34866], 99.50th=[36963], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 2432, max= 3184, per=25.41%, avg=2748.57, stdev=289.07, samples=7
   iops        : min=  608, max=  796, avg=687.14, stdev=72.27, samples=7
  lat (msec)   : 4=0.04%, 10=0.08%, 20=15.16%, 50=84.73%
  cpu          : usr=0.85%, sys=4.54%, ctx=1588, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=507: Mon Apr  7 08:23:56 2025
  write: IOPS=678, BW=2715KiB/s (2781kB/s)(10.0MiB/3771msec)
    slat (usec): min=12, max=478, avg=70.82, stdev=46.70
    clat (usec): min=2414, max=57534, avg=23427.25, stdev=5258.57
     lat (usec): min=2482, max=57602, avg=23499.62, stdev=5268.43
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[14091], 10.00th=[16712], 20.00th=[20317],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23200], 60.00th=[23987],
     | 70.00th=[25035], 80.00th=[27132], 90.00th=[29754], 95.00th=[32113],
     | 99.00th=[35914], 99.50th=[37487], 99.90th=[52167], 99.95th=[52691],
     | 99.99th=[57410]
   bw (  KiB/s): min= 2424, max= 3256, per=25.55%, avg=2763.43, stdev=317.85, samples=7
   iops        : min=  606, max=  814, avg=690.86, stdev=79.46, samples=7
  lat (msec)   : 4=0.04%, 10=1.09%, 20=14.26%, 50=84.49%, 100=0.12%
  cpu          : usr=0.58%, sys=5.17%, ctx=2198, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.6MiB/s (11.1MB/s), 2704KiB/s-2715KiB/s (2769kB/s-2781kB/s), io=40.0MiB (41.9MB), run=3771-3787msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=520: Mon Apr  7 08:24:00 2025
   read: IOPS=438, BW=1753KiB/s (1795kB/s)(5048KiB/2879msec)
    slat (usec): min=649, max=2536, avg=1177.82, stdev=245.49
    clat (usec): min=3309, max=29899, avg=16986.24, stdev=2549.77
     lat (usec): min=4430, max=31439, avg=18166.18, stdev=2580.94
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13173], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15533], 40.00th=[16319], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[20317], 95.00th=[21365],
     | 99.00th=[23462], 99.50th=[24249], 99.90th=[28705], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1640, max= 1860, per=24.81%, avg=1728.80, stdev=88.02, samples=5
   iops        : min=  410, max=  465, avg=432.20, stdev=22.00, samples=5
  write: IOPS=450, BW=1803KiB/s (1847kB/s)(5192KiB/2879msec)
    slat (usec): min=14, max=318, avg=54.36, stdev=33.17
    clat (usec): min=11405, max=28580, avg=17500.71, stdev=1972.79
     lat (usec): min=11445, max=28636, avg=17556.40, stdev=1972.76
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14484], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16450], 40.00th=[16909], 50.00th=[17433], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[27395], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1760, max= 1832, per=24.87%, avg=1804.00, stdev=33.70, samples=5
   iops        : min=  440, max=  458, avg=451.00, stdev= 8.43, samples=5
  lat (msec)   : 4=0.04%, 20=89.26%, 50=10.70%
  cpu          : usr=1.29%, sys=7.71%, ctx=3000, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=521: Mon Apr  7 08:24:00 2025
   read: IOPS=434, BW=1737KiB/s (1779kB/s)(4976KiB/2864msec)
    slat (usec): min=24, max=2642, avg=1169.84, stdev=252.48
    clat (usec): min=1659, max=35453, avg=16930.23, stdev=2651.78
     lat (usec): min=1824, max=36702, avg=18102.23, stdev=2686.88
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13173], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15533], 40.00th=[16188], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[24249], 99.50th=[26608], 99.90th=[30802], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1664, max= 1828, per=24.54%, avg=1709.60, stdev=67.86, samples=5
   iops        : min=  416, max=  457, avg=427.40, stdev=16.96, samples=5
  write: IOPS=459, BW=1838KiB/s (1882kB/s)(5264KiB/2864msec)
    slat (usec): min=13, max=261, avg=51.89, stdev=27.85
    clat (usec): min=6877, max=31800, avg=17568.27, stdev=2233.64
     lat (usec): min=6899, max=31829, avg=17621.44, stdev=2234.29
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14353], 10.00th=[15008], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[25560], 99.90th=[29492], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 1736, max= 1912, per=25.46%, avg=1847.20, stdev=65.87, samples=5
   iops        : min=  434, max=  478, avg=461.80, stdev=16.47, samples=5
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.27%, 20=88.87%, 50=10.74%
  cpu          : usr=1.05%, sys=7.72%, ctx=2919, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=522: Mon Apr  7 08:24:00 2025
   read: IOPS=432, BW=1731KiB/s (1772kB/s)(4984KiB/2880msec)
    slat (usec): min=650, max=2552, avg=1164.06, stdev=242.27
    clat (usec): min=2658, max=24997, avg=16948.53, stdev=2423.52
     lat (usec): min=4108, max=26480, avg=18114.61, stdev=2451.36
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13173], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[23200], 99.50th=[23987], 99.90th=[25035], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1592, max= 1872, per=24.43%, avg=1702.40, stdev=118.31, samples=5
   iops        : min=  398, max=  468, avg=425.60, stdev=29.58, samples=5
  write: IOPS=456, BW=1825KiB/s (1869kB/s)(5256KiB/2880msec)
    slat (usec): min=13, max=902, avg=52.99, stdev=36.08
    clat (usec): min=9556, max=25560, avg=17547.51, stdev=2114.71
     lat (usec): min=9598, max=25616, avg=17601.63, stdev=2115.67
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14353], 10.00th=[15008], 20.00th=[15795],
     | 30.00th=[16450], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20317], 95.00th=[21365],
     | 99.00th=[22938], 99.50th=[23725], 99.90th=[24249], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 1680, max= 2064, per=24.97%, avg=1811.20, stdev=151.22, samples=5
   iops        : min=  420, max=  516, avg=452.80, stdev=37.80, samples=5
  lat (msec)   : 4=0.04%, 10=0.12%, 20=88.24%, 50=11.60%
  cpu          : usr=1.18%, sys=7.61%, ctx=2830, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=523: Mon Apr  7 08:24:00 2025
   read: IOPS=440, BW=1761KiB/s (1803kB/s)(5060KiB/2874msec)
    slat (usec): min=711, max=2313, avg=1184.76, stdev=242.90
    clat (usec): min=9582, max=27562, avg=17024.57, stdev=2421.68
     lat (usec): min=11750, max=28877, avg=18211.28, stdev=2446.80
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13304], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21365],
     | 99.00th=[23462], 99.50th=[24249], 99.90th=[26084], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1688, max= 1800, per=24.97%, avg=1740.00, stdev=47.67, samples=5
   iops        : min=  422, max=  450, avg=435.00, stdev=11.92, samples=5
  write: IOPS=450, BW=1802KiB/s (1846kB/s)(5180KiB/2874msec)
    slat (usec): min=13, max=358, avg=54.04, stdev=31.98
    clat (usec): min=10704, max=29056, avg=17447.40, stdev=2106.52
     lat (usec): min=10898, max=29117, avg=17502.76, stdev=2106.82
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14222], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22938], 99.50th=[23725], 99.90th=[28967], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1744, max= 1884, per=24.74%, avg=1794.40, stdev=59.03, samples=5
   iops        : min=  436, max=  471, avg=448.60, stdev=14.76, samples=5
  lat (msec)   : 10=0.04%, 20=89.73%, 50=10.23%
  cpu          : usr=0.73%, sys=8.21%, ctx=3028, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6968KiB/s (7135kB/s), 1731KiB/s-1761KiB/s (1772kB/s-1803kB/s), io=19.6MiB (20.5MB), run=2864-2880msec
  WRITE: bw=7254KiB/s (7428kB/s), 1802KiB/s-1838KiB/s (1846kB/s-1882kB/s), io=20.4MiB (21.4MB), run=2864-2880msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=529: Mon Apr  7 08:24:01 2025
   read: IOPS=12, BW=500MiB/s (524MB/s)(40.0MiB/80msec)
    slat (nsec): min=6194.4k, max=6194.4k, avg=6194368.00, stdev= 0.00
    clat (nsec): min=68560k, max=68560k, avg=68559753.00, stdev= 0.00
     lat (nsec): min=74772k, max=74772k, avg=74772479.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.86%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=40.0MiB (41.9MB), run=80-80msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout