== sanity test 398c: run fio to test AIO ================= 19:32:52 (1736987572)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0763754 s, 549 MB/s
osc.lustre-OST0000-osc-ffff8800a8909800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18139: Wed Jan 15 19:32:56 2025
  write: IOPS=1033, BW=4134KiB/s (4233kB/s)(10.0MiB/2477msec)
    slat (usec): min=12, max=340, avg=42.71, stdev=22.69
    clat (usec): min=1640, max=27137, avg=15408.26, stdev=2331.50
     lat (usec): min=1662, max=27296, avg=15452.13, stdev=2330.59
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10683], 10.00th=[12125], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[16909], 90.00th=[17695], 95.00th=[18220],
     | 99.00th=[19792], 99.50th=[23987], 99.90th=[27132], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 3816, max= 4672, per=25.38%, avg=4170.00, stdev=374.63, samples=4
   iops        : min=  954, max= 1168, avg=1042.50, stdev=93.66, samples=4
  lat (msec)   : 2=0.08%, 4=0.23%, 10=2.11%, 20=96.72%, 50=0.86%
  cpu          : usr=0.65%, sys=5.21%, ctx=1399, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18140: Wed Jan 15 19:32:56 2025
  write: IOPS=1027, BW=4111KiB/s (4209kB/s)(10.0MiB/2491msec)
    slat (usec): min=12, max=289, avg=41.53, stdev=22.04
    clat (usec): min=1513, max=28962, avg=15480.57, stdev=2295.53
     lat (usec): min=1577, max=28988, avg=15523.29, stdev=2296.29
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10945], 10.00th=[12256], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17695], 95.00th=[18220],
     | 99.00th=[20055], 99.50th=[25560], 99.90th=[25560], 99.95th=[25822],
     | 99.99th=[28967]
   bw (  KiB/s): min= 3832, max= 4544, per=25.17%, avg=4136.00, stdev=307.84, samples=4
   iops        : min=  958, max= 1136, avg=1034.00, stdev=76.96, samples=4
  lat (msec)   : 2=0.08%, 4=0.08%, 10=1.72%, 20=97.15%, 50=0.98%
  cpu          : usr=0.96%, sys=4.62%, ctx=1391, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18141: Wed Jan 15 19:32:56 2025
  write: IOPS=1026, BW=4108KiB/s (4206kB/s)(10.0MiB/2493msec)
    slat (usec): min=12, max=564, avg=42.46, stdev=24.15
    clat (usec): min=6967, max=25812, avg=15490.04, stdev=2154.17
     lat (usec): min=7013, max=25835, avg=15533.62, stdev=2155.57
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10945], 10.00th=[12387], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15795], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17171], 90.00th=[17433], 95.00th=[18220],
     | 99.00th=[19792], 99.50th=[25822], 99.90th=[25822], 99.95th=[25822],
     | 99.99th=[25822]
   bw (  KiB/s): min= 3824, max= 4496, per=25.11%, avg=4126.00, stdev=288.92, samples=4
   iops        : min=  956, max= 1124, avg=1031.50, stdev=72.23, samples=4
  lat (msec)   : 10=1.45%, 20=97.81%, 50=0.74%
  cpu          : usr=0.88%, sys=4.94%, ctx=1434, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18142: Wed Jan 15 19:32:56 2025
  write: IOPS=1027, BW=4111KiB/s (4209kB/s)(10.0MiB/2491msec)
    slat (usec): min=12, max=369, avg=42.92, stdev=23.72
    clat (usec): min=1695, max=29019, avg=15479.27, stdev=2270.67
     lat (usec): min=1741, max=29035, avg=15523.38, stdev=2269.57
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10945], 10.00th=[12125], 20.00th=[14222],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16581], 80.00th=[16909], 90.00th=[17695], 95.00th=[18220],
     | 99.00th=[20055], 99.50th=[25560], 99.90th=[27395], 99.95th=[27657],
     | 99.99th=[28967]
   bw (  KiB/s): min= 3808, max= 4568, per=25.21%, avg=4142.00, stdev=327.80, samples=4
   iops        : min=  952, max= 1142, avg=1035.50, stdev=81.95, samples=4
  lat (msec)   : 2=0.08%, 10=2.27%, 20=96.68%, 50=0.98%
  cpu          : usr=1.04%, sys=4.66%, ctx=1383, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=16.0MiB/s (16.8MB/s), 4108KiB/s-4134KiB/s (4206kB/s-4233kB/s), io=40.0MiB (41.9MB), run=2477-2493msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=18154: Wed Jan 15 19:32:58 2025
   read: IOPS=655, BW=2622KiB/s (2685kB/s)(5048KiB/1925msec)
    slat (usec): min=14, max=1395, avg=749.10, stdev=139.84
    clat (usec): min=818, max=25027, avg=11433.26, stdev=1841.14
     lat (usec): min=840, max=25820, avg=12183.96, stdev=1869.51
    clat percentiles (usec):
     |  1.00th=[ 5669],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[11076], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13435], 95.00th=[14091],
     | 99.00th=[15795], 99.50th=[16188], 99.90th=[23725], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2576, max= 2648, per=25.19%, avg=2608.00, stdev=36.66, samples=3
   iops        : min=  644, max=  662, avg=652.00, stdev= 9.17, samples=3
  write: IOPS=674, BW=2697KiB/s (2762kB/s)(5192KiB/1925msec)
    slat (usec): min=13, max=383, avg=40.55, stdev=22.02
    clat (usec): min=2254, max=22388, avg=11776.30, stdev=1427.87
     lat (usec): min=2361, max=22416, avg=11817.90, stdev=1426.42
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 9765], 10.00th=[10159], 20.00th=[10683],
     | 30.00th=[11207], 40.00th=[11469], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[12780], 90.00th=[13304], 95.00th=[13829],
     | 99.00th=[14877], 99.50th=[15008], 99.90th=[19268], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2688, max= 2800, per=25.50%, avg=2749.33, stdev=56.76, samples=3
   iops        : min=  672, max=  700, avg=687.33, stdev=14.19, samples=3
  lat (usec)   : 1000=0.08%
  lat (msec)   : 2=0.08%, 4=0.31%, 10=12.03%, 20=87.38%, 50=0.12%
  cpu          : usr=1.04%, sys=10.03%, ctx=2200, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18155: Wed Jan 15 19:32:58 2025
   read: IOPS=642, BW=2572KiB/s (2633kB/s)(4976KiB/1935msec)
    slat (usec): min=477, max=1635, avg=759.13, stdev=130.99
    clat (usec): min=3943, max=19961, avg=11418.77, stdev=1549.73
     lat (usec): min=4512, max=20656, avg=12179.80, stdev=1557.35
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[10159],
     | 30.00th=[10552], 40.00th=[10945], 50.00th=[11338], 60.00th=[11731],
     | 70.00th=[12125], 80.00th=[12780], 90.00th=[13435], 95.00th=[13829],
     | 99.00th=[15270], 99.50th=[15926], 99.90th=[19268], 99.95th=[20055],
     | 99.99th=[20055]
   bw (  KiB/s): min= 2448, max= 2616, per=24.44%, avg=2530.67, stdev=84.03, samples=3
   iops        : min=  612, max=  654, avg=632.67, stdev=21.01, samples=3
  write: IOPS=680, BW=2720KiB/s (2786kB/s)(5264KiB/1935msec)
    slat (usec): min=12, max=371, avg=42.17, stdev=24.37
    clat (usec): min=7456, max=22921, avg=11850.33, stdev=1299.52
     lat (usec): min=7522, max=22950, avg=11893.66, stdev=1298.14
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11207], 40.00th=[11469], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[12911], 90.00th=[13435], 95.00th=[13960],
     | 99.00th=[14615], 99.50th=[15270], 99.90th=[20579], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 2720, max= 2800, per=25.58%, avg=2757.33, stdev=40.27, samples=3
   iops        : min=  680, max=  700, avg=689.33, stdev=10.07, samples=3
  lat (msec)   : 4=0.04%, 10=11.64%, 20=88.24%, 50=0.08%
  cpu          : usr=1.19%, sys=10.03%, ctx=2248, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18156: Wed Jan 15 19:32:58 2025
   read: IOPS=643, BW=2573KiB/s (2635kB/s)(4984KiB/1937msec)
    slat (usec): min=447, max=1281, avg=753.39, stdev=128.49
    clat (usec): min=3088, max=18422, avg=11440.76, stdev=1617.52
     lat (usec): min=3837, max=19238, avg=12195.82, stdev=1620.92
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8979], 10.00th=[ 9372], 20.00th=[10159],
     | 30.00th=[10683], 40.00th=[11076], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13435], 95.00th=[13960],
     | 99.00th=[15401], 99.50th=[16319], 99.90th=[16909], 99.95th=[18482],
     | 99.99th=[18482]
   bw (  KiB/s): min= 2592, max= 2600, per=25.06%, avg=2594.67, stdev= 4.62, samples=3
   iops        : min=  648, max=  650, avg=648.67, stdev= 1.15, samples=3
  write: IOPS=678, BW=2713KiB/s (2779kB/s)(5256KiB/1937msec)
    slat (usec): min=13, max=855, avg=41.14, stdev=31.22
    clat (usec): min=6100, max=22251, avg=11857.25, stdev=1381.76
     lat (usec): min=6172, max=22367, avg=11899.47, stdev=1380.93
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9896], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11207], 40.00th=[11469], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12518], 80.00th=[12911], 90.00th=[13435], 95.00th=[13829],
     | 99.00th=[15139], 99.50th=[15533], 99.90th=[22152], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2576, max= 2800, per=24.91%, avg=2685.33, stdev=112.10, samples=3
   iops        : min=  644, max=  700, avg=671.33, stdev=28.02, samples=3
  lat (msec)   : 4=0.08%, 10=11.84%, 20=87.93%, 50=0.16%
  cpu          : usr=1.34%, sys=9.76%, ctx=2240, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=18157: Wed Jan 15 19:32:58 2025
   read: IOPS=652, BW=2611KiB/s (2674kB/s)(5060KiB/1938msec)
    slat (usec): min=439, max=1344, avg=754.83, stdev=126.01
    clat (usec): min=1278, max=16345, avg=11448.02, stdev=1599.52
     lat (usec): min=2099, max=17631, avg=12204.59, stdev=1598.94
    clat percentiles (usec):
     |  1.00th=[ 7767],  5.00th=[ 8979], 10.00th=[ 9503], 20.00th=[10028],
     | 30.00th=[10552], 40.00th=[10945], 50.00th=[11469], 60.00th=[11863],
     | 70.00th=[12256], 80.00th=[12780], 90.00th=[13566], 95.00th=[14091],
     | 99.00th=[15401], 99.50th=[15795], 99.90th=[16319], 99.95th=[16319],
     | 99.99th=[16319]
   bw (  KiB/s): min= 2520, max= 2696, per=25.16%, avg=2605.33, stdev=88.12, samples=3
   iops        : min=  630, max=  674, avg=651.33, stdev=22.03, samples=3
  write: IOPS=668, BW=2673KiB/s (2737kB/s)(5180KiB/1938msec)
    slat (usec): min=13, max=319, avg=40.72, stdev=20.90
    clat (usec): min=6148, max=17102, avg=11857.57, stdev=1285.61
     lat (usec): min=6227, max=17158, avg=11899.35, stdev=1283.84
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[ 9765], 10.00th=[10290], 20.00th=[10814],
     | 30.00th=[11207], 40.00th=[11469], 50.00th=[11863], 60.00th=[12125],
     | 70.00th=[12387], 80.00th=[12911], 90.00th=[13435], 95.00th=[13960],
     | 99.00th=[15139], 99.50th=[15533], 99.90th=[17171], 99.95th=[17171],
     | 99.99th=[17171]
   bw (  KiB/s): min= 2648, max= 2704, per=24.76%, avg=2669.33, stdev=30.29, samples=3
   iops        : min=  662, max=  676, avg=667.33, stdev= 7.57, samples=3
  lat (msec)   : 2=0.04%, 10=12.15%, 20=87.81%
  cpu          : usr=0.98%, sys=9.91%, ctx=2252, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.1MiB/s (10.6MB/s), 2572KiB/s-2622KiB/s (2633kB/s-2685kB/s), io=19.6MiB (20.5MB), run=1925-1938msec
  WRITE: bw=10.5MiB/s (11.0MB/s), 2673KiB/s-2720KiB/s (2737kB/s-2786kB/s), io=20.4MiB (21.4MB), run=1925-1938msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=18162: Wed Jan 15 19:33:00 2025
   read: IOPS=13, BW=541MiB/s (567MB/s)(40.0MiB/74msec)
    slat (nsec): min=7014.1k, max=7014.1k, avg=7014068.00, stdev= 0.00
    clat (nsec): min=63622k, max=63622k, avg=63621775.00, stdev= 0.00
     lat (nsec): min=70645k, max=70645k, avg=70645494.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[63701],  5.00th=[63701], 10.00th=[63701], 20.00th=[63701],
     | 30.00th=[63701], 40.00th=[63701], 50.00th=[63701], 60.00th=[63701],
     | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701],
     | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.59%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=541MiB/s (567MB/s), 541MiB/s-541MiB/s (567MB/s-567MB/s), io=40.0MiB (41.9MB), run=74-74msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck