== sanity test 398c: run fio to test AIO ================= 21:06:45 (1718932005)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0782453 s, 536 MB/s
osc.lustre-OST0000-osc-ffff88007a2fd000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=16546: Thu Jun 20 21:06:49 2024
  write: IOPS=1038, BW=4154KiB/s (4254kB/s)(10.0MiB/2465msec)
    slat (usec): min=11, max=212, avg=42.89, stdev=24.17
    clat (usec): min=5910, max=43472, avg=15303.27, stdev=2980.89
     lat (usec): min=5968, max=43547, avg=15347.33, stdev=2983.42
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11994], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14091], 40.00th=[14746], 50.00th=[15270], 60.00th=[15533],
     | 70.00th=[16057], 80.00th=[16450], 90.00th=[17171], 95.00th=[19530],
     | 99.00th=[20841], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 4000, max= 4360, per=25.49%, avg=4236.00, stdev=164.54, samples=4
   iops        : min= 1000, max= 1090, avg=1059.00, stdev=41.13, samples=4
  lat (msec)   : 10=0.82%, 20=95.98%, 50=3.20%
  cpu          : usr=1.01%, sys=4.95%, ctx=1554, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16547: Thu Jun 20 21:06:49 2024
  write: IOPS=1041, BW=4164KiB/s (4264kB/s)(10.0MiB/2459msec)
    slat (usec): min=11, max=252, avg=45.07, stdev=25.30
    clat (usec): min=2174, max=46689, avg=15286.07, stdev=3020.27
     lat (usec): min=2249, max=46744, avg=15332.34, stdev=3020.53
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[11994], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14091], 40.00th=[14615], 50.00th=[15139], 60.00th=[15533],
     | 70.00th=[15926], 80.00th=[16450], 90.00th=[17433], 95.00th=[19530],
     | 99.00th=[21365], 99.50th=[40633], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 4000, max= 4424, per=25.70%, avg=4270.00, stdev=188.45, samples=4
   iops        : min= 1000, max= 1106, avg=1067.50, stdev=47.11, samples=4
  lat (msec)   : 4=0.12%, 10=0.59%, 20=95.74%, 50=3.55%
  cpu          : usr=0.81%, sys=5.57%, ctx=1592, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16548: Thu Jun 20 21:06:49 2024
  write: IOPS=1039, BW=4158KiB/s (4257kB/s)(10.0MiB/2463msec)
    slat (usec): min=11, max=220, avg=44.95, stdev=22.99
    clat (usec): min=1646, max=43264, avg=15284.04, stdev=3002.30
     lat (usec): min=1718, max=43325, avg=15330.19, stdev=3004.95
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[12125], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14091], 40.00th=[14615], 50.00th=[15270], 60.00th=[15533],
     | 70.00th=[15926], 80.00th=[16450], 90.00th=[17171], 95.00th=[19530],
     | 99.00th=[21890], 99.50th=[42730], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 3984, max= 4368, per=25.58%, avg=4250.00, stdev=180.95, samples=4
   iops        : min=  996, max= 1092, avg=1062.50, stdev=45.24, samples=4
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.35%, 20=96.13%, 50=3.40%
  cpu          : usr=1.14%, sys=5.40%, ctx=1705, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=16549: Thu Jun 20 21:06:49 2024
  write: IOPS=1038, BW=4156KiB/s (4256kB/s)(10.0MiB/2464msec)
    slat (usec): min=11, max=321, avg=42.38, stdev=24.28
    clat (usec): min=6083, max=43371, avg=15296.73, stdev=2960.85
     lat (usec): min=6192, max=43399, avg=15340.26, stdev=2963.69
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[11994], 10.00th=[12780], 20.00th=[13566],
     | 30.00th=[14091], 40.00th=[14615], 50.00th=[15139], 60.00th=[15533],
     | 70.00th=[16057], 80.00th=[16450], 90.00th=[17171], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[42730], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 4016, max= 4352, per=25.55%, avg=4246.00, stdev=156.41, samples=4
   iops        : min= 1004, max= 1088, avg=1061.50, stdev=39.10, samples=4
  lat (msec)   : 10=0.51%, 20=96.17%, 50=3.32%
  cpu          : usr=0.89%, sys=5.16%, ctx=1549, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=16.2MiB/s (17.0MB/s), 4154KiB/s-4164KiB/s (4254kB/s-4264kB/s), io=40.0MiB (41.9MB), run=2459-2465msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=16560: Thu Jun 20 21:06:51 2024
   read: IOPS=670, BW=2684KiB/s (2748kB/s)(5048KiB/1881msec)
    slat (usec): min=12, max=1257, avg=718.78, stdev=134.13
    clat (usec): min=1047, max=20096, avg=11162.73, stdev=1788.05
     lat (usec): min=1068, max=21010, avg=11883.09, stdev=1820.71
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11207], 60.00th=[11600],
     | 70.00th=[12125], 80.00th=[12518], 90.00th=[13173], 95.00th=[13698],
     | 99.00th=[14877], 99.50th=[15401], 99.90th=[19530], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min= 2568, max= 2704, per=24.88%, avg=2637.33, stdev=68.04, samples=3
   iops        : min=  642, max=  676, avg=659.33, stdev=17.01, samples=3
  write: IOPS=690, BW=2760KiB/s (2826kB/s)(5192KiB/1881msec)
    slat (usec): min=12, max=653, avg=41.61, stdev=37.70
    clat (usec): min=2139, max=19448, avg=11529.90, stdev=1418.16
     lat (usec): min=2207, max=19485, avg=11572.64, stdev=1414.96
    clat percentiles (usec):
     |  1.00th=[ 8356],  5.00th=[ 9503], 10.00th=[ 9896], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11207], 50.00th=[11600], 60.00th=[11863],
     | 70.00th=[12125], 80.00th=[12649], 90.00th=[13042], 95.00th=[13566],
     | 99.00th=[14353], 99.50th=[14746], 99.90th=[18482], 99.95th=[19530],
     | 99.99th=[19530]
   bw (  KiB/s): min= 2712, max= 2816, per=25.18%, avg=2778.67, stdev=57.87, samples=3
   iops        : min=  678, max=  704, avg=694.67, stdev=14.47, samples=3
  lat (msec)   : 2=0.16%, 4=0.39%, 10=16.68%, 20=82.73%, 50=0.04%
  cpu          : usr=1.28%, sys=9.73%, ctx=2609, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16561: Thu Jun 20 21:06:51 2024
   read: IOPS=657, BW=2630KiB/s (2693kB/s)(4976KiB/1892msec)
    slat (usec): min=443, max=1290, avg=722.32, stdev=122.19
    clat (usec): min=7220, max=22453, avg=11269.90, stdev=1607.75
     lat (usec): min=8011, max=23229, avg=11993.89, stdev=1618.60
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 8717], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11338], 60.00th=[11731],
     | 70.00th=[12125], 80.00th=[12649], 90.00th=[13173], 95.00th=[13698],
     | 99.00th=[15008], 99.50th=[15926], 99.90th=[17957], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2464, max= 2688, per=24.45%, avg=2592.00, stdev=115.38, samples=3
   iops        : min=  616, max=  672, avg=648.00, stdev=28.84, samples=3
  write: IOPS=695, BW=2782KiB/s (2849kB/s)(5264KiB/1892msec)
    slat (usec): min=12, max=175, avg=38.81, stdev=19.59
    clat (usec): min=6357, max=20977, avg=11528.84, stdev=1263.99
     lat (usec): min=6389, max=21005, avg=11568.66, stdev=1263.28
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9634], 10.00th=[10028], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11207], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12125], 80.00th=[12518], 90.00th=[13042], 95.00th=[13435],
     | 99.00th=[14615], 99.50th=[15008], 99.90th=[20317], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min= 2760, max= 2832, per=25.30%, avg=2792.00, stdev=36.66, samples=3
   iops        : min=  690, max=  708, avg=698.00, stdev= 9.17, samples=3
  lat (msec)   : 10=16.25%, 20=83.63%, 50=0.12%
  cpu          : usr=1.11%, sys=9.62%, ctx=2628, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16562: Thu Jun 20 21:06:51 2024
   read: IOPS=658, BW=2633KiB/s (2696kB/s)(4984KiB/1893msec)
    slat (usec): min=444, max=1213, avg=724.28, stdev=121.16
    clat (usec): min=2339, max=15759, avg=11244.18, stdev=1569.39
     lat (usec): min=2981, max=16416, avg=11969.95, stdev=1578.10
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 8717], 10.00th=[ 9241], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11207], 60.00th=[11731],
     | 70.00th=[12125], 80.00th=[12649], 90.00th=[13304], 95.00th=[13698],
     | 99.00th=[14615], 99.50th=[15008], 99.90th=[15533], 99.95th=[15795],
     | 99.99th=[15795]
   bw (  KiB/s): min= 2592, max= 2664, per=24.73%, avg=2621.33, stdev=37.81, samples=3
   iops        : min=  648, max=  666, avg=655.33, stdev= 9.45, samples=3
  write: IOPS=694, BW=2777KiB/s (2843kB/s)(5256KiB/1893msec)
    slat (usec): min=12, max=726, avg=39.57, stdev=28.23
    clat (usec): min=4391, max=16933, avg=11548.80, stdev=1230.01
     lat (usec): min=4434, max=16953, avg=11589.57, stdev=1230.01
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9634], 10.00th=[10028], 20.00th=[10552],
     | 30.00th=[10945], 40.00th=[11207], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12518], 90.00th=[13042], 95.00th=[13566],
     | 99.00th=[14222], 99.50th=[14484], 99.90th=[16581], 99.95th=[16909],
     | 99.99th=[16909]
   bw (  KiB/s): min= 2584, max= 2808, per=24.65%, avg=2720.00, stdev=119.47, samples=3
   iops        : min=  646, max=  702, avg=680.00, stdev=29.87, samples=3
  lat (msec)   : 4=0.04%, 10=16.17%, 20=83.79%
  cpu          : usr=0.95%, sys=9.83%, ctx=2625, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=16563: Thu Jun 20 21:06:51 2024
   read: IOPS=668, BW=2674KiB/s (2739kB/s)(5060KiB/1892msec)
    slat (usec): min=448, max=1554, avg=720.50, stdev=121.11
    clat (usec): min=1616, max=16778, avg=11234.03, stdev=1628.03
     lat (usec): min=2330, max=17479, avg=11956.27, stdev=1639.32
    clat percentiles (usec):
     |  1.00th=[ 7898],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11338], 60.00th=[11731],
     | 70.00th=[12125], 80.00th=[12649], 90.00th=[13304], 95.00th=[13698],
     | 99.00th=[14484], 99.50th=[15139], 99.90th=[16188], 99.95th=[16909],
     | 99.99th=[16909]
   bw (  KiB/s): min= 2536, max= 2696, per=24.88%, avg=2637.33, stdev=88.12, samples=3
   iops        : min=  634, max=  674, avg=659.33, stdev=22.03, samples=3
  write: IOPS=684, BW=2738KiB/s (2804kB/s)(5180KiB/1892msec)
    slat (usec): min=12, max=549, avg=40.82, stdev=27.31
    clat (usec): min=5098, max=16672, avg=11540.78, stdev=1252.11
     lat (usec): min=5251, max=16729, avg=11582.60, stdev=1250.39
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10552],
     | 30.00th=[10814], 40.00th=[11207], 50.00th=[11600], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12649], 90.00th=[13042], 95.00th=[13566],
     | 99.00th=[14615], 99.50th=[14877], 99.90th=[15270], 99.95th=[16712],
     | 99.99th=[16712]
   bw (  KiB/s): min= 2656, max= 2784, per=24.60%, avg=2714.67, stdev=64.66, samples=3
   iops        : min=  664, max=  696, avg=678.67, stdev=16.17, samples=3
  lat (msec)   : 2=0.04%, 4=0.04%, 10=17.11%, 20=82.81%
  cpu          : usr=1.22%, sys=9.68%, ctx=2672, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.4MiB/s (10.9MB/s), 2630KiB/s-2684KiB/s (2693kB/s-2748kB/s), io=19.6MiB (20.5MB), run=1881-1893msec
  WRITE: bw=10.8MiB/s (11.3MB/s), 2738KiB/s-2782KiB/s (2804kB/s-2849kB/s), io=20.4MiB (21.4MB), run=1881-1893msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=16569: Thu Jun 20 21:06:53 2024
   read: IOPS=11, BW=455MiB/s (477MB/s)(40.0MiB/88msec)
    slat (nsec): min=7075.3k, max=7075.3k, avg=7075277.00, stdev= 0.00
    clat (nsec): min=77280k, max=77280k, avg=77280382.00, stdev= 0.00
     lat (nsec): min=84370k, max=84370k, avg=84369807.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[77071],  5.00th=[77071], 10.00th=[77071], 20.00th=[77071],
     | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[77071],
     | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071],
     | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.05%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=40.0MiB (41.9MB), run=88-88msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck