== sanity test 398c: run fio to test AIO ================= 18:31:34 (1715207494)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.133204 s, 315 MB/s
osc.lustre-OST0000-osc-ffff8800b662e000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16130: Wed May  8 18:31:42 2024
  write: IOPS=466, BW=1866KiB/s (1910kB/s)(10.0MiB/5489msec)
    slat (usec): min=16, max=475, avg=87.31, stdev=47.27
    clat (usec): min=3961, max=73939, avg=34165.88, stdev=4496.84
     lat (usec): min=4088, max=74033, avg=34255.23, stdev=4496.89
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26608], 10.00th=[29230], 20.00th=[31851],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[41681], 99.50th=[49021], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1776, max= 2072, per=25.23%, avg=1874.00, stdev=92.71, samples=10
   iops        : min=  444, max=  518, avg=468.50, stdev=23.18, samples=10
  lat (msec)   : 4=0.04%, 10=0.16%, 20=0.47%, 50=98.87%, 100=0.47%
  cpu          : usr=0.67%, sys=5.01%, ctx=1853, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16131: Wed May  8 18:31:42 2024
  write: IOPS=464, BW=1858KiB/s (1903kB/s)(10.0MiB/5511msec)
    slat (usec): min=15, max=34217, avg=94.02, stdev=676.06
    clat (usec): min=3900, max=68158, avg=34256.97, stdev=4188.03
     lat (usec): min=4016, max=68242, avg=34352.76, stdev=4177.65
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26870], 10.00th=[29492], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[45351], 99.50th=[47449], 99.90th=[60556], 99.95th=[64750],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1720, max= 1968, per=24.91%, avg=1850.18, stdev=77.43, samples=11
   iops        : min=  430, max=  492, avg=462.55, stdev=19.36, samples=11
  lat (msec)   : 4=0.04%, 10=0.12%, 20=0.23%, 50=99.30%, 100=0.31%
  cpu          : usr=0.87%, sys=4.95%, ctx=1638, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16132: Wed May  8 18:31:42 2024
  write: IOPS=464, BW=1857KiB/s (1902kB/s)(10.0MiB/5514msec)
    slat (usec): min=13, max=454, avg=82.47, stdev=45.36
    clat (usec): min=7360, max=59395, avg=34322.67, stdev=4084.32
     lat (usec): min=7497, max=59524, avg=34407.12, stdev=4086.35
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26870], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[47449], 99.50th=[50594], 99.90th=[55313], 99.95th=[58983],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1728, max= 1976, per=24.91%, avg=1850.55, stdev=77.07, samples=11
   iops        : min=  432, max=  494, avg=462.64, stdev=19.27, samples=11
  lat (msec)   : 10=0.04%, 20=0.20%, 50=98.87%, 100=0.90%
  cpu          : usr=0.91%, sys=4.52%, ctx=1574, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16133: Wed May  8 18:31:42 2024
  write: IOPS=464, BW=1857KiB/s (1902kB/s)(10.0MiB/5514msec)
    slat (usec): min=16, max=34246, avg=99.70, stdev=677.10
    clat (usec): min=3891, max=62552, avg=34241.03, stdev=3981.71
     lat (usec): min=3923, max=62660, avg=34342.75, stdev=3970.07
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[27132], 10.00th=[29492], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[41681], 99.50th=[47973], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1752, max= 1976, per=24.91%, avg=1850.18, stdev=70.96, samples=11
   iops        : min=  438, max=  494, avg=462.55, stdev=17.74, samples=11
  lat (msec)   : 4=0.04%, 20=0.31%, 50=99.41%, 100=0.23%
  cpu          : usr=0.63%, sys=5.51%, ctx=1756, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7428KiB/s (7607kB/s), 1857KiB/s-1866KiB/s (1902kB/s-1910kB/s), io=40.0MiB (41.9MB), run=5489-5514msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16147: Wed May  8 18:31:45 2024
   read: IOPS=588, BW=2354KiB/s (2411kB/s)(5048KiB/2144msec)
    slat (usec): min=407, max=2314, avg=804.63, stdev=288.45
    clat (usec): min=7356, max=29624, avg=12591.32, stdev=3418.23
     lat (usec): min=8092, max=31431, avg=13397.96, stdev=3631.09
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9110], 10.00th=[ 9503], 20.00th=[10290],
     | 30.00th=[10814], 40.00th=[11338], 50.00th=[11731], 60.00th=[12256],
     | 70.00th=[12780], 80.00th=[13698], 90.00th=[16909], 95.00th=[21103],
     | 99.00th=[25035], 99.50th=[26084], 99.90th=[27919], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1512, max= 2816, per=24.88%, avg=2324.00, stdev=574.98, samples=4
   iops        : min=  378, max=  704, avg=581.00, stdev=143.75, samples=4
  write: IOPS=605, BW=2422KiB/s (2480kB/s)(5192KiB/2144msec)
    slat (usec): min=12, max=342, avg=41.29, stdev=27.10
    clat (usec): min=7790, max=28334, avg=13114.98, stdev=3449.50
     lat (usec): min=7823, max=28365, avg=13157.28, stdev=3459.31
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10028], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[13042], 80.00th=[14353], 90.00th=[17433], 95.00th=[21365],
     | 99.00th=[25822], 99.50th=[26608], 99.90th=[27132], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1544, max= 2800, per=24.82%, avg=2414.00, stdev=585.27, samples=4
   iops        : min=  386, max=  700, avg=603.50, stdev=146.32, samples=4
  lat (msec)   : 10=10.47%, 20=82.38%, 50=7.15%
  cpu          : usr=1.77%, sys=9.38%, ctx=2196, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16148: Wed May  8 18:31:45 2024
   read: IOPS=581, BW=2327KiB/s (2383kB/s)(4976KiB/2138msec)
    slat (usec): min=24, max=2324, avg=803.96, stdev=270.42
    clat (usec): min=1847, max=29998, avg=12635.18, stdev=3741.79
     lat (usec): min=2511, max=31469, avg=13440.94, stdev=3930.99
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 9110], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256],
     | 70.00th=[12911], 80.00th=[13829], 90.00th=[16581], 95.00th=[21627],
     | 99.00th=[26870], 99.50th=[27657], 99.90th=[29492], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1464, max= 2744, per=24.51%, avg=2290.00, stdev=575.16, samples=4
   iops        : min=  366, max=  686, avg=572.50, stdev=143.79, samples=4
  write: IOPS=615, BW=2462KiB/s (2521kB/s)(5264KiB/2138msec)
    slat (usec): min=11, max=646, avg=40.52, stdev=29.04
    clat (usec): min=7254, max=30683, avg=13160.21, stdev=3603.89
     lat (usec): min=7386, max=30721, avg=13201.78, stdev=3613.65
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125], 60.00th=[12518],
     | 70.00th=[13173], 80.00th=[14353], 90.00th=[17695], 95.00th=[21890],
     | 99.00th=[26870], 99.50th=[27919], 99.90th=[30278], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1680, max= 2880, per=25.35%, avg=2466.00, stdev=541.11, samples=4
   iops        : min=  420, max=  720, avg=616.50, stdev=135.28, samples=4
  lat (msec)   : 2=0.04%, 4=0.12%, 10=11.09%, 20=80.98%, 50=7.77%
  cpu          : usr=1.68%, sys=9.22%, ctx=2187, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16149: Wed May  8 18:31:45 2024
   read: IOPS=580, BW=2320KiB/s (2376kB/s)(4984KiB/2148msec)
    slat (usec): min=416, max=2451, avg=812.18, stdev=290.10
    clat (usec): min=1033, max=27828, avg=12766.50, stdev=3581.89
     lat (usec): min=1740, max=29480, avg=13580.59, stdev=3795.13
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10290],
     | 30.00th=[10945], 40.00th=[11338], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[13173], 80.00th=[14091], 90.00th=[17433], 95.00th=[20841],
     | 99.00th=[25822], 99.50th=[26608], 99.90th=[27395], 99.95th=[27919],
     | 99.99th=[27919]
   bw (  KiB/s): min= 1512, max= 2656, per=24.60%, avg=2298.00, stdev=529.83, samples=4
   iops        : min=  378, max=  664, avg=574.50, stdev=132.46, samples=4
  write: IOPS=611, BW=2447KiB/s (2506kB/s)(5256KiB/2148msec)
    slat (usec): min=12, max=303, avg=40.98, stdev=27.78
    clat (usec): min=1843, max=29682, avg=13023.92, stdev=3514.80
     lat (usec): min=1877, max=29715, avg=13065.94, stdev=3527.43
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9896], 10.00th=[10421], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11731], 50.00th=[12125], 60.00th=[12387],
     | 70.00th=[13042], 80.00th=[14091], 90.00th=[16450], 95.00th=[21890],
     | 99.00th=[26346], 99.50th=[26608], 99.90th=[27919], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1512, max= 2992, per=24.78%, avg=2410.00, stdev=645.89, samples=4
   iops        : min=  378, max=  748, avg=602.50, stdev=161.47, samples=4
  lat (msec)   : 2=0.08%, 10=11.09%, 20=81.84%, 50=6.99%
  cpu          : usr=1.40%, sys=9.36%, ctx=2236, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16150: Wed May  8 18:31:45 2024
   read: IOPS=590, BW=2363KiB/s (2420kB/s)(5060KiB/2141msec)
    slat (usec): min=436, max=2413, avg=805.68, stdev=282.04
    clat (usec): min=7603, max=30547, avg=12617.69, stdev=3501.83
     lat (usec): min=8353, max=32219, avg=13425.13, stdev=3706.33
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9110], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256],
     | 70.00th=[12780], 80.00th=[13960], 90.00th=[17433], 95.00th=[20841],
     | 99.00th=[25035], 99.50th=[26870], 99.90th=[28705], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1528, max= 2778, per=25.03%, avg=2338.50, stdev=581.86, samples=4
   iops        : min=  382, max=  694, avg=584.50, stdev=145.34, samples=4
  write: IOPS=604, BW=2419KiB/s (2477kB/s)(5180KiB/2141msec)
    slat (usec): min=11, max=296, avg=41.29, stdev=28.78
    clat (usec): min=8275, max=30133, avg=13090.75, stdev=3460.85
     lat (usec): min=8321, max=30167, avg=13133.04, stdev=3474.53
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10945],
     | 30.00th=[11338], 40.00th=[11600], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[13173], 80.00th=[14222], 90.00th=[16909], 95.00th=[21890],
     | 99.00th=[25822], 99.50th=[26608], 99.90th=[28181], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1520, max= 2834, per=24.64%, avg=2396.50, stdev=593.30, samples=4
   iops        : min=  380, max=  708, avg=599.00, stdev=148.20, samples=4
  lat (msec)   : 10=10.51%, 20=82.30%, 50=7.19%
  cpu          : usr=1.17%, sys=9.67%, ctx=2266, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9343KiB/s (9567kB/s), 2320KiB/s-2363KiB/s (2376kB/s-2420kB/s), io=19.6MiB (20.5MB), run=2138-2148msec
  WRITE: bw=9726KiB/s (9960kB/s), 2419KiB/s-2462KiB/s (2477kB/s-2521kB/s), io=20.4MiB (21.4MB), run=2138-2148msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16155: Wed May  8 18:31:46 2024
   read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=6900.3k, max=6900.3k, avg=6900347.00, stdev= 0.00
    clat (nsec): min=78531k, max=78531k, avg=78531054.00, stdev= 0.00
     lat (nsec): min=85441k, max=85441k, avg=85441099.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[78119],  5.00th=[78119], 10.00th=[78119], 20.00th=[78119],
     | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119],
     | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119],
     | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.87%, ctx=4, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck