== sanity test 398c: run fio to test AIO ================= 18:43:21 (1748990601)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0825476 s, 508 MB/s
osc.lustre-OST0000-osc-ffff8800aaf84000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30601: Tue Jun  3 18:43:24 2025
  write: IOPS=1045, BW=4183KiB/s (4283kB/s)(10.0MiB/2448msec)
    slat (usec): min=11, max=27198, avg=50.24, stdev=537.21
    clat (usec): min=2501, max=35756, avg=15165.82, stdev=2272.82
     lat (usec): min=2552, max=35854, avg=15217.18, stdev=2305.51
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12649], 10.00th=[13173], 20.00th=[13698],
     | 30.00th=[14091], 40.00th=[14615], 50.00th=[15008], 60.00th=[15401],
     | 70.00th=[15795], 80.00th=[16319], 90.00th=[17171], 95.00th=[18220],
     | 99.00th=[21103], 99.50th=[35390], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 4000, max= 4200, per=24.59%, avg=4108.00, stdev=89.68, samples=4
   iops        : min= 1000, max= 1050, avg=1027.00, stdev=22.42, samples=4
  lat (msec)   : 4=0.08%, 10=0.12%, 20=98.01%, 50=1.80%
  cpu          : usr=0.78%, sys=6.33%, ctx=1457, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30602: Tue Jun  3 18:43:24 2025
  write: IOPS=1044, BW=4178KiB/s (4278kB/s)(10.0MiB/2451msec)
    slat (usec): min=11, max=252, avg=37.85, stdev=20.50
    clat (usec): min=1488, max=35927, avg=15221.29, stdev=2348.50
     lat (usec): min=1536, max=35995, avg=15260.32, stdev=2349.03
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12649], 10.00th=[13173], 20.00th=[13698],
     | 30.00th=[14222], 40.00th=[14615], 50.00th=[15008], 60.00th=[15401],
     | 70.00th=[15795], 80.00th=[16319], 90.00th=[17171], 95.00th=[18220],
     | 99.00th=[21103], 99.50th=[35914], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 4008, max= 4176, per=24.53%, avg=4097.75, stdev=82.92, samples=4
   iops        : min= 1002, max= 1044, avg=1024.25, stdev=20.89, samples=4
  lat (msec)   : 2=0.04%, 20=98.09%, 50=1.88%
  cpu          : usr=0.86%, sys=4.82%, ctx=1291, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30603: Tue Jun  3 18:43:24 2025
  write: IOPS=1048, BW=4195KiB/s (4296kB/s)(10.0MiB/2441msec)
    slat (usec): min=12, max=194, avg=38.18, stdev=20.17
    clat (usec): min=1895, max=40413, avg=15191.37, stdev=2514.27
     lat (usec): min=1966, max=40444, avg=15230.72, stdev=2514.15
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[12256], 10.00th=[13173], 20.00th=[13698],
     | 30.00th=[14091], 40.00th=[14615], 50.00th=[15008], 60.00th=[15401],
     | 70.00th=[15795], 80.00th=[16450], 90.00th=[17433], 95.00th=[18482],
     | 99.00th=[21890], 99.50th=[33162], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 3952, max= 4192, per=24.70%, avg=4126.00, stdev=116.55, samples=4
   iops        : min=  988, max= 1048, avg=1031.50, stdev=29.14, samples=4
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.39%, 20=97.34%, 50=2.15%
  cpu          : usr=1.07%, sys=4.75%, ctx=1392, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30604: Tue Jun  3 18:43:24 2025
  write: IOPS=1044, BW=4176KiB/s (4276kB/s)(10.0MiB/2452msec)
    slat (usec): min=12, max=521, avg=39.74, stdev=24.72
    clat (usec): min=9757, max=39638, avg=15249.79, stdev=2581.36
     lat (usec): min=9865, max=39742, avg=15290.62, stdev=2582.13
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12518], 10.00th=[13304], 20.00th=[13698],
     | 30.00th=[14222], 40.00th=[14615], 50.00th=[15008], 60.00th=[15401],
     | 70.00th=[15795], 80.00th=[16319], 90.00th=[17171], 95.00th=[18220],
     | 99.00th=[21103], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 3944, max= 4175, per=24.51%, avg=4093.75, stdev=107.35, samples=4
   iops        : min=  986, max= 1043, avg=1023.25, stdev=26.65, samples=4
  lat (msec)   : 10=0.12%, 20=97.58%, 50=2.30%
  cpu          : usr=0.78%, sys=5.06%, ctx=1453, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=16.3MiB/s (17.1MB/s), 4176KiB/s-4195KiB/s (4276kB/s-4296kB/s), io=40.0MiB (41.9MB), run=2441-2452msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30616: Tue Jun  3 18:43:27 2025
   read: IOPS=675, BW=2704KiB/s (2769kB/s)(5048KiB/1867msec)
    slat (usec): min=410, max=1273, avg=720.42, stdev=120.76
    clat (usec): min=907, max=16369, avg=11085.07, stdev=1641.04
     lat (usec): min=1583, max=16973, avg=11807.21, stdev=1639.58
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10683], 50.00th=[11076], 60.00th=[11469],
     | 70.00th=[11863], 80.00th=[12387], 90.00th=[13173], 95.00th=[13698],
     | 99.00th=[15270], 99.50th=[15926], 99.90th=[16188], 99.95th=[16319],
     | 99.99th=[16319]
   bw (  KiB/s): min= 2552, max= 2800, per=24.71%, avg=2656.00, stdev=128.75, samples=3
   iops        : min=  638, max=  700, avg=664.00, stdev=32.19, samples=3
  write: IOPS=695, BW=2781KiB/s (2848kB/s)(5192KiB/1867msec)
    slat (usec): min=11, max=666, avg=36.05, stdev=30.06
    clat (usec): min=2155, max=15559, avg=11383.83, stdev=1329.64
     lat (usec): min=2183, max=15584, avg=11420.92, stdev=1328.27
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10290],
     | 30.00th=[10683], 40.00th=[11076], 50.00th=[11338], 60.00th=[11600],
     | 70.00th=[11994], 80.00th=[12518], 90.00th=[13042], 95.00th=[13566],
     | 99.00th=[14615], 99.50th=[15008], 99.90th=[15401], 99.95th=[15533],
     | 99.99th=[15533]
   bw (  KiB/s): min= 2712, max= 2880, per=25.21%, avg=2821.33, stdev=94.77, samples=3
   iops        : min=  678, max=  720, avg=705.33, stdev=23.69, samples=3
  lat (usec)   : 1000=0.04%
  lat (msec)   : 4=0.12%, 10=18.95%, 20=80.90%
  cpu          : usr=1.50%, sys=9.27%, ctx=2404, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30617: Tue Jun  3 18:43:27 2025
   read: IOPS=667, BW=2670KiB/s (2734kB/s)(4976KiB/1864msec)
    slat (usec): min=438, max=1710, avg=721.83, stdev=125.95
    clat (usec): min=2284, max=21574, avg=11072.27, stdev=1630.77
     lat (usec): min=2943, max=22143, avg=11795.68, stdev=1637.87
    clat percentiles (usec):
     |  1.00th=[ 7767],  5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10159], 40.00th=[10683], 50.00th=[11076], 60.00th=[11469],
     | 70.00th=[11863], 80.00th=[12387], 90.00th=[13042], 95.00th=[13698],
     | 99.00th=[15270], 99.50th=[15401], 99.90th=[20841], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2584, max= 2816, per=24.81%, avg=2666.67, stdev=129.57, samples=3
   iops        : min=  646, max=  704, avg=666.67, stdev=32.39, samples=3
  write: IOPS=706, BW=2824KiB/s (2892kB/s)(5264KiB/1864msec)
    slat (usec): min=12, max=728, avg=36.44, stdev=27.71
    clat (usec): min=1664, max=21534, avg=11379.18, stdev=1361.41
     lat (usec): min=1700, max=21566, avg=11416.57, stdev=1361.48
    clat percentiles (usec):
     |  1.00th=[ 7963],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10290],
     | 30.00th=[10683], 40.00th=[11076], 50.00th=[11338], 60.00th=[11731],
     | 70.00th=[11994], 80.00th=[12387], 90.00th=[13042], 95.00th=[13566],
     | 99.00th=[14615], 99.50th=[15139], 99.90th=[17957], 99.95th=[21627],
     | 99.99th=[21627]
   bw (  KiB/s): min= 2728, max= 2936, per=25.48%, avg=2850.67, stdev=108.91, samples=3
   iops        : min=  682, max=  734, avg=712.67, stdev=27.23, samples=3
  lat (msec)   : 2=0.04%, 4=0.04%, 10=19.34%, 20=80.47%, 50=0.12%
  cpu          : usr=1.93%, sys=8.70%, ctx=2407, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30618: Tue Jun  3 18:43:27 2025
   read: IOPS=669, BW=2677KiB/s (2741kB/s)(4984KiB/1862msec)
    slat (usec): min=12, max=1254, avg=716.33, stdev=131.13
    clat (usec): min=2046, max=18615, avg=11100.27, stdev=1646.56
     lat (usec): min=2851, max=19215, avg=11818.22, stdev=1658.55
    clat percentiles (usec):
     |  1.00th=[ 7439],  5.00th=[ 8455], 10.00th=[ 8979], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10814], 50.00th=[11076], 60.00th=[11469],
     | 70.00th=[11863], 80.00th=[12387], 90.00th=[13042], 95.00th=[13698],
     | 99.00th=[15401], 99.50th=[15795], 99.90th=[17957], 99.95th=[18744],
     | 99.99th=[18744]
   bw (  KiB/s): min= 2496, max= 2864, per=25.08%, avg=2696.00, stdev=186.08, samples=3
   iops        : min=  624, max=  716, avg=674.00, stdev=46.52, samples=3
  write: IOPS=705, BW=2823KiB/s (2891kB/s)(5256KiB/1862msec)
    slat (usec): min=11, max=784, avg=37.47, stdev=29.14
    clat (usec): min=3066, max=21253, avg=11368.22, stdev=1512.82
     lat (usec): min=3079, max=21284, avg=11406.73, stdev=1510.97
    clat percentiles (usec):
     |  1.00th=[ 8094],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10290],
     | 30.00th=[10683], 40.00th=[10945], 50.00th=[11338], 60.00th=[11600],
     | 70.00th=[11994], 80.00th=[12518], 90.00th=[13042], 95.00th=[13566],
     | 99.00th=[15139], 99.50th=[15795], 99.90th=[21103], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2720, max= 2968, per=25.28%, avg=2829.33, stdev=126.58, samples=3
   iops        : min=  680, max=  742, avg=707.33, stdev=31.64, samples=3
  lat (msec)   : 4=0.16%, 10=18.67%, 20=81.02%, 50=0.16%
  cpu          : usr=1.29%, sys=9.35%, ctx=2402, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30619: Tue Jun  3 18:43:27 2025
   read: IOPS=677, BW=2710KiB/s (2775kB/s)(5060KiB/1867msec)
    slat (usec): min=460, max=1448, avg=718.96, stdev=121.80
    clat (usec): min=2695, max=17196, avg=11135.86, stdev=1641.83
     lat (usec): min=3513, max=17836, avg=11856.38, stdev=1643.95
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10290], 40.00th=[10683], 50.00th=[11207], 60.00th=[11469],
     | 70.00th=[11994], 80.00th=[12387], 90.00th=[13173], 95.00th=[13960],
     | 99.00th=[15270], 99.50th=[15664], 99.90th=[16909], 99.95th=[17171],
     | 99.99th=[17171]
   bw (  KiB/s): min= 2616, max= 2808, per=25.16%, avg=2704.00, stdev=96.99, samples=3
   iops        : min=  654, max=  702, avg=676.00, stdev=24.25, samples=3
  write: IOPS=693, BW=2775KiB/s (2841kB/s)(5180KiB/1867msec)
    slat (usec): min=12, max=515, avg=36.74, stdev=25.74
    clat (usec): min=3516, max=15821, avg=11348.64, stdev=1391.76
     lat (usec): min=3529, max=15840, avg=11386.30, stdev=1389.47
    clat percentiles (usec):
     |  1.00th=[ 8225],  5.00th=[ 9372], 10.00th=[ 9765], 20.00th=[10290],
     | 30.00th=[10683], 40.00th=[11076], 50.00th=[11338], 60.00th=[11600],
     | 70.00th=[11994], 80.00th=[12387], 90.00th=[13042], 95.00th=[13566],
     | 99.00th=[14746], 99.50th=[15008], 99.90th=[15533], 99.95th=[15795],
     | 99.99th=[15795]
   bw (  KiB/s): min= 2624, max= 2904, per=24.93%, avg=2789.33, stdev=146.72, samples=3
   iops        : min=  656, max=  726, avg=697.33, stdev=36.68, samples=3
  lat (msec)   : 4=0.16%, 10=19.45%, 20=80.39%
  cpu          : usr=0.75%, sys=9.86%, ctx=2350, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=10.5MiB/s (11.0MB/s), 2670KiB/s-2710KiB/s (2734kB/s-2775kB/s), io=19.6MiB (20.5MB), run=1862-1867msec
  WRITE: bw=10.9MiB/s (11.5MB/s), 2775KiB/s-2824KiB/s (2841kB/s-2892kB/s), io=20.4MiB (21.4MB), run=1862-1867msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30624: Tue Jun  3 18:43:28 2025
   read: IOPS=12, BW=513MiB/s (538MB/s)(40.0MiB/78msec)
    slat (nsec): min=5852.2k, max=5852.2k, avg=5852180.00, stdev= 0.00
    clat (nsec): min=68501k, max=68501k, avg=68501480.00, stdev= 0.00
     lat (nsec): min=74363k, max=74363k, avg=74363050.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.49%, ctx=10, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=513MiB/s (538MB/s), 513MiB/s-513MiB/s (538MB/s-538MB/s), io=40.0MiB (41.9MB), run=78-78msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck