== sanity test 398c: run fio to test AIO ================= 20:09:16 (1736989756)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0818547 s, 512 MB/s
osc.lustre-OST0000-osc-ffff8801302dd800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=20709: Wed Jan 15 20:09:19 2025
  write: IOPS=1152, BW=4608KiB/s (4719kB/s)(10.0MiB/2222msec)
    slat (usec): min=9, max=363, avg=38.60, stdev=21.94
    clat (usec): min=1156, max=24377, avg=13822.29, stdev=2269.51
     lat (usec): min=1209, max=24387, avg=13861.93, stdev=2270.58
    clat percentiles (usec):
     |  1.00th=[ 6587],  5.00th=[10290], 10.00th=[11207], 20.00th=[12518],
     | 30.00th=[13173], 40.00th=[13566], 50.00th=[13829], 60.00th=[14222],
     | 70.00th=[14746], 80.00th=[15401], 90.00th=[16450], 95.00th=[17171],
     | 99.00th=[18744], 99.50th=[20055], 99.90th=[24249], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 4496, max= 4880, per=25.42%, avg=4646.00, stdev=164.13, samples=4
   iops        : min= 1124, max= 1220, avg=1161.50, stdev=41.03, samples=4
  lat (msec)   : 2=0.04%, 4=0.31%, 10=4.18%, 20=94.96%, 50=0.51%
  cpu          : usr=1.31%, sys=4.64%, ctx=1428, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20710: Wed Jan 15 20:09:19 2025
  write: IOPS=1145, BW=4584KiB/s (4694kB/s)(10.0MiB/2234msec)
    slat (usec): min=10, max=250, avg=37.88, stdev=20.40
    clat (usec): min=4209, max=25722, avg=13893.05, stdev=2177.29
     lat (usec): min=4249, max=25740, avg=13931.96, stdev=2179.66
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[10159], 10.00th=[11338], 20.00th=[12518],
     | 30.00th=[13173], 40.00th=[13566], 50.00th=[13829], 60.00th=[14222],
     | 70.00th=[14746], 80.00th=[15401], 90.00th=[16319], 95.00th=[17171],
     | 99.00th=[18482], 99.50th=[22676], 99.90th=[22676], 99.95th=[25822],
     | 99.99th=[25822]
   bw (  KiB/s): min= 4512, max= 4768, per=25.28%, avg=4620.00, stdev=107.83, samples=4
   iops        : min= 1128, max= 1192, avg=1155.00, stdev=26.96, samples=4
  lat (msec)   : 10=4.26%, 20=94.92%, 50=0.82%
  cpu          : usr=1.21%, sys=4.75%, ctx=1316, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20711: Wed Jan 15 20:09:19 2025
  write: IOPS=1143, BW=4573KiB/s (4683kB/s)(10.0MiB/2239msec)
    slat (usec): min=10, max=399, avg=38.74, stdev=20.82
    clat (usec): min=1459, max=27358, avg=13920.46, stdev=2196.33
     lat (usec): min=1509, max=27414, avg=13960.21, stdev=2199.21
    clat percentiles (usec):
     |  1.00th=[ 7767],  5.00th=[10290], 10.00th=[11600], 20.00th=[12649],
     | 30.00th=[13173], 40.00th=[13566], 50.00th=[13829], 60.00th=[14222],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[16319], 95.00th=[17171],
     | 99.00th=[18744], 99.50th=[22676], 99.90th=[27132], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 4544, max= 4768, per=25.27%, avg=4618.00, stdev=103.97, samples=4
   iops        : min= 1136, max= 1192, avg=1154.50, stdev=25.99, samples=4
  lat (msec)   : 2=0.04%, 10=3.79%, 20=95.23%, 50=0.94%
  cpu          : usr=1.21%, sys=4.83%, ctx=1461, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20712: Wed Jan 15 20:09:19 2025
  write: IOPS=1142, BW=4569KiB/s (4679kB/s)(10.0MiB/2241msec)
    slat (usec): min=10, max=283, avg=37.63, stdev=20.10
    clat (usec): min=2206, max=23014, avg=13933.23, stdev=2077.86
     lat (usec): min=2247, max=23035, avg=13971.90, stdev=2078.26
    clat percentiles (usec):
     |  1.00th=[ 7046],  5.00th=[10421], 10.00th=[11600], 20.00th=[12780],
     | 30.00th=[13173], 40.00th=[13566], 50.00th=[13960], 60.00th=[14353],
     | 70.00th=[14746], 80.00th=[15533], 90.00th=[16319], 95.00th=[16909],
     | 99.00th=[18482], 99.50th=[22938], 99.90th=[22938], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 4520, max= 4734, per=25.15%, avg=4597.50, stdev=99.30, samples=4
   iops        : min= 1130, max= 1183, avg=1149.25, stdev=24.60, samples=4
  lat (msec)   : 4=0.04%, 10=3.32%, 20=96.02%, 50=0.62%
  cpu          : usr=0.76%, sys=4.96%, ctx=1241, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=17.8MiB/s (18.7MB/s), 4569KiB/s-4608KiB/s (4679kB/s-4719kB/s), io=40.0MiB (41.9MB), run=2222-2241msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=20724: Wed Jan 15 20:09:22 2025
   read: IOPS=616, BW=2466KiB/s (2525kB/s)(5048KiB/2047msec)
    slat (usec): min=452, max=1740, avg=776.57, stdev=162.59
    clat (usec): min=2055, max=20504, avg=12192.83, stdev=2090.44
     lat (usec): min=2646, max=21307, avg=12971.04, stdev=2130.53
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[13042], 80.00th=[13829], 90.00th=[14746], 95.00th=[15926],
     | 99.00th=[18220], 99.50th=[19006], 99.90th=[20317], 99.95th=[20579],
     | 99.99th=[20579]
   bw (  KiB/s): min= 2392, max= 2584, per=25.05%, avg=2456.00, stdev=86.66, samples=4
   iops        : min=  598, max=  646, avg=614.00, stdev=21.66, samples=4
  write: IOPS=634, BW=2536KiB/s (2597kB/s)(5192KiB/2047msec)
    slat (usec): min=11, max=565, avg=40.85, stdev=25.77
    clat (usec): min=5635, max=19427, avg=12436.12, stdev=1813.51
     lat (usec): min=5699, max=19450, avg=12477.96, stdev=1814.54
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10945],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13173], 80.00th=[13698], 90.00th=[14877], 95.00th=[15795],
     | 99.00th=[17695], 99.50th=[18220], 99.90th=[19006], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2456, max= 2688, per=24.81%, avg=2532.00, stdev=105.83, samples=4
   iops        : min=  614, max=  672, avg=633.00, stdev=26.46, samples=4
  lat (msec)   : 4=0.04%, 10=9.34%, 20=90.55%, 50=0.08%
  cpu          : usr=0.98%, sys=9.53%, ctx=2225, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20725: Wed Jan 15 20:09:22 2025
   read: IOPS=613, BW=2455KiB/s (2514kB/s)(4976KiB/2027msec)
    slat (usec): min=9, max=1761, avg=774.70, stdev=163.24
    clat (usec): min=1080, max=20359, avg=12048.48, stdev=2017.34
     lat (usec): min=1109, max=21133, avg=12824.92, stdev=2065.05
    clat percentiles (usec):
     |  1.00th=[ 7439],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12911], 80.00th=[13566], 90.00th=[14484], 95.00th=[15533],
     | 99.00th=[17695], 99.50th=[18220], 99.90th=[19792], 99.95th=[20317],
     | 99.99th=[20317]
   bw (  KiB/s): min= 2368, max= 2536, per=24.91%, avg=2442.00, stdev=76.84, samples=4
   iops        : min=  592, max=  634, avg=610.50, stdev=19.21, samples=4
  write: IOPS=649, BW=2597KiB/s (2659kB/s)(5264KiB/2027msec)
    slat (usec): min=9, max=249, avg=39.61, stdev=18.34
    clat (usec): min=3336, max=19559, avg=12423.06, stdev=1897.90
     lat (usec): min=3357, max=19576, avg=12463.75, stdev=1899.32
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[10945],
     | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13173], 80.00th=[13698], 90.00th=[14877], 95.00th=[15664],
     | 99.00th=[17695], 99.50th=[18482], 99.90th=[19006], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2480, max= 2776, per=25.49%, avg=2602.00, stdev=124.52, samples=4
   iops        : min=  620, max=  694, avg=650.50, stdev=31.13, samples=4
  lat (msec)   : 2=0.08%, 4=0.12%, 10=10.04%, 20=89.73%, 50=0.04%
  cpu          : usr=1.04%, sys=9.77%, ctx=2246, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20726: Wed Jan 15 20:09:22 2025
   read: IOPS=610, BW=2441KiB/s (2499kB/s)(4984KiB/2042msec)
    slat (usec): min=473, max=1723, avg=780.67, stdev=159.64
    clat (usec): min=6397, max=23348, avg=12172.03, stdev=2099.24
     lat (usec): min=7274, max=23896, avg=12954.28, stdev=2131.11
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 9110], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[13042], 80.00th=[13829], 90.00th=[14746], 95.00th=[15926],
     | 99.00th=[18482], 99.50th=[19530], 99.90th=[22676], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 2328, max= 2576, per=24.91%, avg=2442.00, stdev=102.74, samples=4
   iops        : min=  582, max=  644, avg=610.50, stdev=25.68, samples=4
  write: IOPS=643, BW=2574KiB/s (2636kB/s)(5256KiB/2042msec)
    slat (usec): min=12, max=971, avg=41.85, stdev=34.96
    clat (usec): min=6014, max=21790, avg=12411.93, stdev=1780.28
     lat (usec): min=6067, max=21842, avg=12454.83, stdev=1783.66
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[10028], 10.00th=[10421], 20.00th=[11076],
     | 30.00th=[11469], 40.00th=[11863], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13698], 90.00th=[14615], 95.00th=[15533],
     | 99.00th=[17695], 99.50th=[18220], 99.90th=[21103], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 2496, max= 2656, per=25.10%, avg=2562.00, stdev=67.37, samples=4
   iops        : min=  624, max=  664, avg=640.50, stdev=16.84, samples=4
  lat (msec)   : 10=8.91%, 20=90.90%, 50=0.20%
  cpu          : usr=1.37%, sys=9.31%, ctx=2248, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20727: Wed Jan 15 20:09:22 2025
   read: IOPS=617, BW=2472KiB/s (2531kB/s)(5060KiB/2047msec)
    slat (usec): min=430, max=1788, avg=780.88, stdev=160.52
    clat (usec): min=1907, max=19710, avg=12202.06, stdev=2094.88
     lat (usec): min=2919, max=20551, avg=12984.48, stdev=2126.34
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 9241], 10.00th=[ 9896], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[13173], 80.00th=[13698], 90.00th=[15008], 95.00th=[15795],
     | 99.00th=[18220], 99.50th=[18482], 99.90th=[19530], 99.95th=[19792],
     | 99.99th=[19792]
   bw (  KiB/s): min= 2416, max= 2552, per=25.18%, avg=2468.00, stdev=58.61, samples=4
   iops        : min=  604, max=  638, avg=617.00, stdev=14.65, samples=4
  write: IOPS=632, BW=2531KiB/s (2591kB/s)(5180KiB/2047msec)
    slat (usec): min=12, max=396, avg=41.10, stdev=26.20
    clat (usec): min=6912, max=19655, avg=12439.40, stdev=1774.99
     lat (usec): min=6972, max=19695, avg=12481.48, stdev=1774.85
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9765], 10.00th=[10421], 20.00th=[11076],
     | 30.00th=[11600], 40.00th=[11863], 50.00th=[12387], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13698], 90.00th=[14615], 95.00th=[15664],
     | 99.00th=[17695], 99.50th=[18744], 99.90th=[19530], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2424, max= 2600, per=24.67%, avg=2518.00, stdev=75.72, samples=4
   iops        : min=  606, max=  650, avg=629.50, stdev=18.93, samples=4
  lat (msec)   : 2=0.04%, 10=9.10%, 20=90.86%
  cpu          : usr=0.93%, sys=9.73%, ctx=2245, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9804KiB/s (10.0MB/s), 2441KiB/s-2472KiB/s (2499kB/s-2531kB/s), io=19.6MiB (20.5MB), run=2027-2047msec
  WRITE: bw=9.97MiB/s (10.5MB/s), 2531KiB/s-2597KiB/s (2591kB/s-2659kB/s), io=20.4MiB (21.4MB), run=2027-2047msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=20732: Wed Jan 15 20:09:24 2025
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=7353.1k, max=7353.1k, avg=7353066.00, stdev= 0.00
    clat (nsec): min=66010k, max=66010k, avg=66009867.00, stdev= 0.00
     lat (nsec): min=73384k, max=73384k, avg=73384011.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[65799],  5.00th=[65799], 10.00th=[65799], 20.00th=[65799],
     | 30.00th=[65799], 40.00th=[65799], 50.00th=[65799], 60.00th=[65799],
     | 70.00th=[65799], 80.00th=[65799], 90.00th=[65799], 95.00th=[65799],
     | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.53%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck