== sanity test 398c: run fio to test AIO ================= 01:00:28 (1717563628)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0954782 s, 439 MB/s
osc.lustre-OST0000-osc-ffff8800b072a800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=10308: Wed Jun  5 01:00:32 2024
  write: IOPS=739, BW=2957KiB/s (3028kB/s)(10.0MiB/3463msec)
    slat (usec): min=11, max=488, avg=59.20, stdev=42.30
    clat (usec): min=3709, max=41566, avg=21497.10, stdev=6505.87
     lat (usec): min=3753, max=41644, avg=21557.68, stdev=6520.23
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[14484], 10.00th=[15139], 20.00th=[16450],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[19268], 60.00th=[20579],
     | 70.00th=[23200], 80.00th=[26608], 90.00th=[32375], 95.00th=[35390],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2112, max= 3720, per=25.24%, avg=2985.33, stdev=696.22, samples=6
   iops        : min=  528, max=  930, avg=746.33, stdev=174.05, samples=6
  lat (msec)   : 4=0.04%, 10=0.12%, 20=55.55%, 50=44.30%
  cpu          : usr=1.16%, sys=4.13%, ctx=2089, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10309: Wed Jun  5 01:00:32 2024
  write: IOPS=741, BW=2966KiB/s (3037kB/s)(10.0MiB/3453msec)
    slat (usec): min=10, max=472, avg=61.83, stdev=41.03
    clat (usec): min=1891, max=41639, avg=21440.35, stdev=6635.04
     lat (usec): min=1943, max=41722, avg=21503.56, stdev=6650.27
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[14091], 10.00th=[15139], 20.00th=[16450],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[19268], 60.00th=[20579],
     | 70.00th=[23200], 80.00th=[26870], 90.00th=[32375], 95.00th=[35390],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2088, max= 3800, per=25.35%, avg=2998.67, stdev=712.07, samples=6
   iops        : min=  522, max=  950, avg=749.67, stdev=178.02, samples=6
  lat (msec)   : 2=0.04%, 4=0.20%, 10=1.02%, 20=54.45%, 50=44.30%
  cpu          : usr=0.90%, sys=4.52%, ctx=2294, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10310: Wed Jun  5 01:00:32 2024
  write: IOPS=739, BW=2957KiB/s (3028kB/s)(10.0MiB/3463msec)
    slat (usec): min=11, max=680, avg=60.06, stdev=46.53
    clat (usec): min=2230, max=41646, avg=21503.34, stdev=6441.06
     lat (usec): min=2340, max=41739, avg=21564.74, stdev=6459.50
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14615], 10.00th=[15270], 20.00th=[16581],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[19268], 60.00th=[20841],
     | 70.00th=[23462], 80.00th=[26608], 90.00th=[32375], 95.00th=[35390],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2072, max= 3728, per=25.20%, avg=2980.00, stdev=682.18, samples=6
   iops        : min=  518, max=  932, avg=745.00, stdev=170.55, samples=6
  lat (msec)   : 4=0.04%, 10=0.08%, 20=56.17%, 50=43.71%
  cpu          : usr=0.92%, sys=4.48%, ctx=1954, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10311: Wed Jun  5 01:00:32 2024
  write: IOPS=739, BW=2959KiB/s (3030kB/s)(10.0MiB/3461msec)
    slat (usec): min=12, max=503, avg=62.65, stdev=44.58
    clat (usec): min=3252, max=41633, avg=21485.95, stdev=6524.26
     lat (usec): min=3292, max=41688, avg=21550.06, stdev=6540.04
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[14353], 10.00th=[15139], 20.00th=[16581],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[19268], 60.00th=[20579],
     | 70.00th=[23462], 80.00th=[26608], 90.00th=[32375], 95.00th=[35390],
     | 99.00th=[39060], 99.50th=[39584], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 2096, max= 3736, per=25.26%, avg=2988.00, stdev=691.39, samples=6
   iops        : min=  524, max=  934, avg=747.00, stdev=172.85, samples=6
  lat (msec)   : 4=0.08%, 10=0.39%, 20=55.66%, 50=43.87%
  cpu          : usr=0.81%, sys=4.54%, ctx=2301, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.5MiB/s (12.1MB/s), 2957KiB/s-2966KiB/s (3028kB/s-3037kB/s), io=40.0MiB (41.9MB), run=3453-3463msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=10323: Wed Jun  5 01:00:36 2024
   read: IOPS=554, BW=2219KiB/s (2272kB/s)(5048KiB/2275msec)
    slat (usec): min=20, max=2539, avg=857.97, stdev=282.15
    clat (usec): min=1483, max=32015, avg=13460.74, stdev=3680.41
     lat (usec): min=2538, max=33450, avg=14320.81, stdev=3867.76
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[ 9372], 10.00th=[10028], 20.00th=[10814],
     | 30.00th=[11469], 40.00th=[12125], 50.00th=[12780], 60.00th=[13304],
     | 70.00th=[14091], 80.00th=[15008], 90.00th=[18220], 95.00th=[21890],
     | 99.00th=[26870], 99.50th=[27919], 99.90th=[31589], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1456, max= 2488, per=24.77%, avg=2172.00, stdev=481.35, samples=4
   iops        : min=  364, max=  622, avg=543.00, stdev=120.34, samples=4
  write: IOPS=570, BW=2282KiB/s (2337kB/s)(5192KiB/2275msec)
    slat (usec): min=12, max=918, avg=52.23, stdev=42.10
    clat (usec): min=2506, max=29089, avg=13969.91, stdev=3675.48
     lat (usec): min=2545, max=29219, avg=14023.31, stdev=3687.67
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10552], 10.00th=[10945], 20.00th=[11469],
     | 30.00th=[12125], 40.00th=[12518], 50.00th=[12911], 60.00th=[13304],
     | 70.00th=[13960], 80.00th=[15008], 90.00th=[20317], 95.00th=[22152],
     | 99.00th=[26608], 99.50th=[27657], 99.90th=[28705], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1496, max= 2616, per=24.78%, avg=2262.00, stdev=521.84, samples=4
   iops        : min=  374, max=  654, avg=565.50, stdev=130.46, samples=4
  lat (msec)   : 2=0.08%, 4=0.16%, 10=5.62%, 20=85.12%, 50=9.02%
  cpu          : usr=1.45%, sys=9.28%, ctx=2977, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10324: Wed Jun  5 01:00:36 2024
   read: IOPS=545, BW=2182KiB/s (2235kB/s)(4976KiB/2280msec)
    slat (usec): min=419, max=2233, avg=856.77, stdev=261.27
    clat (usec): min=7538, max=29772, avg=13416.76, stdev=3558.50
     lat (usec): min=8307, max=31535, avg=14275.53, stdev=3738.15
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11994], 50.00th=[12518], 60.00th=[13042],
     | 70.00th=[13960], 80.00th=[14877], 90.00th=[18744], 95.00th=[21627],
     | 99.00th=[26346], 99.50th=[27132], 99.90th=[29754], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1264, max= 2448, per=24.25%, avg=2126.00, stdev=575.27, samples=4
   iops        : min=  316, max=  612, avg=531.50, stdev=143.82, samples=4
  write: IOPS=577, BW=2309KiB/s (2364kB/s)(5264KiB/2280msec)
    slat (usec): min=12, max=331, avg=49.71, stdev=31.71
    clat (usec): min=1818, max=30870, avg=13970.78, stdev=3770.36
     lat (usec): min=1849, max=30904, avg=14021.58, stdev=3782.17
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10421], 10.00th=[10814], 20.00th=[11469],
     | 30.00th=[11994], 40.00th=[12518], 50.00th=[12911], 60.00th=[13435],
     | 70.00th=[13960], 80.00th=[15008], 90.00th=[20317], 95.00th=[22414],
     | 99.00th=[27395], 99.50th=[29230], 99.90th=[30802], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1552, max= 2704, per=25.18%, avg=2298.00, stdev=514.22, samples=4
   iops        : min=  388, max=  676, avg=574.50, stdev=128.55, samples=4
  lat (msec)   : 2=0.04%, 10=5.55%, 20=85.31%, 50=9.10%
  cpu          : usr=1.49%, sys=9.00%, ctx=2820, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10325: Wed Jun  5 01:00:36 2024
   read: IOPS=544, BW=2177KiB/s (2230kB/s)(4984KiB/2289msec)
    slat (usec): min=460, max=2408, avg=863.04, stdev=271.84
    clat (usec): min=7734, max=28805, avg=13486.21, stdev=3625.61
     lat (usec): min=8529, max=30380, avg=14351.21, stdev=3816.93
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[12125], 50.00th=[12649], 60.00th=[13173],
     | 70.00th=[13829], 80.00th=[15008], 90.00th=[18744], 95.00th=[22152],
     | 99.00th=[26870], 99.50th=[27132], 99.90th=[28705], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1384, max= 2504, per=24.20%, avg=2122.00, stdev=501.41, samples=4
   iops        : min=  346, max=  626, avg=530.50, stdev=125.35, samples=4
  write: IOPS=574, BW=2296KiB/s (2351kB/s)(5256KiB/2289msec)
    slat (usec): min=11, max=634, avg=50.38, stdev=33.27
    clat (usec): min=8712, max=29804, avg=13982.13, stdev=3613.82
     lat (usec): min=8773, max=29898, avg=14033.62, stdev=3623.22
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10552], 10.00th=[11076], 20.00th=[11600],
     | 30.00th=[12125], 40.00th=[12518], 50.00th=[12911], 60.00th=[13304],
     | 70.00th=[13960], 80.00th=[15008], 90.00th=[19530], 95.00th=[22414],
     | 99.00th=[27657], 99.50th=[28967], 99.90th=[29492], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1408, max= 2816, per=24.78%, avg=2262.00, stdev=600.35, samples=4
   iops        : min=  352, max=  704, avg=565.50, stdev=150.09, samples=4
  lat (msec)   : 10=5.35%, 20=86.29%, 50=8.36%
  cpu          : usr=1.01%, sys=9.57%, ctx=3041, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10326: Wed Jun  5 01:00:36 2024
   read: IOPS=553, BW=2213KiB/s (2266kB/s)(5060KiB/2287msec)
    slat (usec): min=439, max=2308, avg=863.13, stdev=267.82
    clat (usec): min=2439, max=30121, avg=13456.71, stdev=3727.48
     lat (usec): min=3180, max=31864, avg=14321.82, stdev=3912.01
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9372], 10.00th=[10028], 20.00th=[10814],
     | 30.00th=[11469], 40.00th=[12125], 50.00th=[12518], 60.00th=[13042],
     | 70.00th=[13960], 80.00th=[15139], 90.00th=[18482], 95.00th=[22152],
     | 99.00th=[26608], 99.50th=[27395], 99.90th=[30016], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1408, max= 2584, per=24.73%, avg=2168.00, stdev=519.82, samples=4
   iops        : min=  352, max=  646, avg=542.00, stdev=129.95, samples=4
  write: IOPS=566, BW=2265KiB/s (2319kB/s)(5180KiB/2287msec)
    slat (usec): min=11, max=540, avg=50.64, stdev=32.89
    clat (usec): min=8772, max=32022, avg=13984.13, stdev=3638.89
     lat (usec): min=8831, max=32088, avg=14035.87, stdev=3652.35
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10552], 10.00th=[10945], 20.00th=[11600],
     | 30.00th=[12125], 40.00th=[12518], 50.00th=[12911], 60.00th=[13435],
     | 70.00th=[14091], 80.00th=[14877], 90.00th=[20317], 95.00th=[22414],
     | 99.00th=[27132], 99.50th=[28705], 99.90th=[30540], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 1392, max= 2616, per=24.48%, avg=2234.00, stdev=567.24, samples=4
   iops        : min=  348, max=  654, avg=558.50, stdev=141.81, samples=4
  lat (msec)   : 4=0.12%, 10=5.90%, 20=85.08%, 50=8.91%
  cpu          : usr=1.40%, sys=9.06%, ctx=3003, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=8767KiB/s (8978kB/s), 2177KiB/s-2219KiB/s (2230kB/s-2272kB/s), io=19.6MiB (20.5MB), run=2275-2289msec
  WRITE: bw=9127KiB/s (9346kB/s), 2265KiB/s-2309KiB/s (2319kB/s-2364kB/s), io=20.4MiB (21.4MB), run=2275-2289msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=10332: Wed Jun  5 01:00:37 2024
   read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=5421.5k, max=5421.5k, avg=5421455.00, stdev= 0.00
    clat (nsec): min=93160k, max=93160k, avg=93159825.00, stdev= 0.00
     lat (nsec): min=98590k, max=98590k, avg=98590249.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[92799],  5.00th=[92799], 10.00th=[92799], 20.00th=[92799],
     | 30.00th=[92799], 40.00th=[92799], 50.00th=[92799], 60.00th=[92799],
     | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799],
     | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799],
     | 99.99th=[92799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.88%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout