== sanity test 398c: run fio to test AIO ================= 14:00:22 (1736362822)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.105816 s, 396 MB/s
osc.lustre-OST0000-osc-ffff8800b6cfc000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=17916: Wed Jan  8 14:00:28 2025
  write: IOPS=509, BW=2039KiB/s (2088kB/s)(10.0MiB/5021msec)
    slat (usec): min=11, max=727, avg=81.76, stdev=48.55
    clat (usec): min=3899, max=60721, avg=31264.09, stdev=4752.80
     lat (usec): min=3927, max=60758, avg=31347.69, stdev=4751.98
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[22676], 10.00th=[25560], 20.00th=[28443],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[40109], 99.50th=[42206], 99.90th=[51119], 99.95th=[51119],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1880, max= 2512, per=25.14%, avg=2035.20, stdev=184.58, samples=10
   iops        : min=  470, max=  628, avg=508.80, stdev=46.14, samples=10
  lat (msec)   : 4=0.04%, 10=0.39%, 20=2.23%, 50=97.15%, 100=0.20%
  cpu          : usr=0.86%, sys=4.64%, ctx=1699, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17917: Wed Jan  8 14:00:28 2025
  write: IOPS=505, BW=2024KiB/s (2072kB/s)(10.0MiB/5060msec)
    slat (usec): min=14, max=1055, avg=77.78, stdev=51.68
    clat (usec): min=15310, max=55804, avg=31494.37, stdev=4450.69
     lat (usec): min=15438, max=55846, avg=31573.99, stdev=4450.92
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[22676], 10.00th=[26084], 20.00th=[28443],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[37487],
     | 99.00th=[39584], 99.50th=[49546], 99.90th=[50070], 99.95th=[50070],
     | 99.99th=[55837]
   bw (  KiB/s): min= 1800, max= 2328, per=24.92%, avg=2016.70, stdev=139.69, samples=10
   iops        : min=  450, max=  582, avg=504.10, stdev=34.92, samples=10
  lat (msec)   : 20=1.45%, 50=98.52%, 100=0.04%
  cpu          : usr=0.59%, sys=4.37%, ctx=1689, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17918: Wed Jan  8 14:00:28 2025
  write: IOPS=507, BW=2029KiB/s (2078kB/s)(10.0MiB/5047msec)
    slat (usec): min=21, max=351, avg=75.34, stdev=43.83
    clat (usec): min=4459, max=51821, avg=31411.80, stdev=4488.37
     lat (usec): min=4567, max=51863, avg=31488.90, stdev=4492.30
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[22414], 10.00th=[25822], 20.00th=[28443],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[48497], 99.90th=[49021], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min= 1864, max= 2384, per=25.01%, avg=2024.00, stdev=148.03, samples=10
   iops        : min=  466, max=  596, avg=506.00, stdev=37.01, samples=10
  lat (msec)   : 10=0.04%, 20=1.72%, 50=98.16%, 100=0.08%
  cpu          : usr=0.67%, sys=4.42%, ctx=1647, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17919: Wed Jan  8 14:00:28 2025
  write: IOPS=505, BW=2024KiB/s (2072kB/s)(10.0MiB/5060msec)
    slat (usec): min=14, max=693, avg=70.69, stdev=44.85
    clat (usec): min=6488, max=56016, avg=31502.95, stdev=4487.10
     lat (usec): min=6601, max=56070, avg=31575.29, stdev=4487.28
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[22414], 10.00th=[26084], 20.00th=[28443],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31589], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[39584], 99.50th=[49021], 99.90th=[50070], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min= 1816, max= 2352, per=24.94%, avg=2018.40, stdev=142.85, samples=10
   iops        : min=  454, max=  588, avg=504.60, stdev=35.71, samples=10
  lat (msec)   : 10=0.04%, 20=1.09%, 50=98.79%, 100=0.08%
  cpu          : usr=0.81%, sys=3.85%, ctx=1413, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8095KiB/s (8289kB/s), 2024KiB/s-2039KiB/s (2072kB/s-2088kB/s), io=40.0MiB (41.9MB), run=5021-5060msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17933: Wed Jan  8 14:00:31 2025
   read: IOPS=524, BW=2097KiB/s (2148kB/s)(5048KiB/2407msec)
    slat (usec): min=457, max=1975, avg=932.04, stdev=245.64
    clat (usec): min=6792, max=28920, avg=14154.17, stdev=2887.30
     lat (usec): min=7437, max=30505, avg=15088.42, stdev=3010.97
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[10028], 10.00th=[10945], 20.00th=[11731],
     | 30.00th=[12518], 40.00th=[13173], 50.00th=[13698], 60.00th=[14615],
     | 70.00th=[15401], 80.00th=[16319], 90.00th=[17957], 95.00th=[19006],
     | 99.00th=[22938], 99.50th=[25297], 99.90th=[28181], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1584, max= 2216, per=23.92%, avg=1994.00, stdev=292.51, samples=4
   iops        : min=  396, max=  554, avg=498.50, stdev=73.13, samples=4
  write: IOPS=539, BW=2157KiB/s (2209kB/s)(5192KiB/2407msec)
    slat (usec): min=15, max=812, avg=54.76, stdev=37.68
    clat (usec): min=7707, max=29918, avg=14754.14, stdev=2736.50
     lat (usec): min=7733, max=29970, avg=14810.06, stdev=2745.25
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[11207], 10.00th=[11863], 20.00th=[12518],
     | 30.00th=[13173], 40.00th=[13829], 50.00th=[14353], 60.00th=[14877],
     | 70.00th=[15664], 80.00th=[16712], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[23987], 99.50th=[26346], 99.90th=[29754], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1688, max= 2472, per=24.36%, avg=2114.00, stdev=323.27, samples=4
   iops        : min=  422, max=  618, avg=528.50, stdev=80.82, samples=4
  lat (msec)   : 10=2.85%, 20=93.91%, 50=3.24%
  cpu          : usr=1.33%, sys=10.22%, ctx=2653, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17934: Wed Jan  8 14:00:31 2025
   read: IOPS=517, BW=2072KiB/s (2121kB/s)(4976KiB/2402msec)
    slat (usec): min=30, max=2144, avg=931.80, stdev=250.73
    clat (usec): min=1358, max=29920, avg=14298.32, stdev=3002.49
     lat (usec): min=1515, max=31686, avg=15232.39, stdev=3129.29
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10159], 10.00th=[11076], 20.00th=[11863],
     | 30.00th=[12780], 40.00th=[13304], 50.00th=[13960], 60.00th=[14615],
     | 70.00th=[15401], 80.00th=[16319], 90.00th=[17695], 95.00th=[19530],
     | 99.00th=[23725], 99.50th=[27395], 99.90th=[28181], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1560, max= 2227, per=23.85%, avg=1988.75, stdev=304.81, samples=4
   iops        : min=  390, max=  556, avg=497.00, stdev=76.01, samples=4
  write: IOPS=547, BW=2192KiB/s (2244kB/s)(5264KiB/2402msec)
    slat (usec): min=16, max=764, avg=55.38, stdev=42.37
    clat (usec): min=4554, max=28737, avg=14662.15, stdev=2824.01
     lat (usec): min=4608, max=28877, avg=14718.87, stdev=2831.70
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10945], 10.00th=[11731], 20.00th=[12518],
     | 30.00th=[13042], 40.00th=[13566], 50.00th=[14222], 60.00th=[14877],
     | 70.00th=[15533], 80.00th=[16712], 90.00th=[18482], 95.00th=[19792],
     | 99.00th=[23462], 99.50th=[25297], 99.90th=[26870], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1784, max= 2443, per=24.62%, avg=2136.75, stdev=275.38, samples=4
   iops        : min=  446, max=  610, avg=534.00, stdev=68.57, samples=4
  lat (msec)   : 2=0.08%, 10=2.85%, 20=93.12%, 50=3.95%
  cpu          : usr=1.21%, sys=10.33%, ctx=2568, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17935: Wed Jan  8 14:00:31 2025
   read: IOPS=518, BW=2075KiB/s (2125kB/s)(4984KiB/2402msec)
    slat (usec): min=467, max=1906, avg=932.73, stdev=237.28
    clat (usec): min=7710, max=28322, avg=14198.35, stdev=2844.41
     lat (usec): min=8325, max=30049, avg=15133.24, stdev=2968.95
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10159], 10.00th=[10945], 20.00th=[11863],
     | 30.00th=[12649], 40.00th=[13304], 50.00th=[13829], 60.00th=[14484],
     | 70.00th=[15401], 80.00th=[16188], 90.00th=[17695], 95.00th=[19006],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[28181], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1600, max= 2187, per=24.02%, avg=2002.75, stdev=272.78, samples=4
   iops        : min=  400, max=  546, avg=500.50, stdev=68.03, samples=4
  write: IOPS=547, BW=2188KiB/s (2241kB/s)(5256KiB/2402msec)
    slat (usec): min=15, max=904, avg=55.34, stdev=38.48
    clat (usec): min=3265, max=29083, avg=14647.61, stdev=2850.45
     lat (usec): min=3354, max=29171, avg=14704.38, stdev=2856.86
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10945], 10.00th=[11600], 20.00th=[12387],
     | 30.00th=[13042], 40.00th=[13566], 50.00th=[14222], 60.00th=[14746],
     | 70.00th=[15664], 80.00th=[16712], 90.00th=[18482], 95.00th=[19792],
     | 99.00th=[23462], 99.50th=[26346], 99.90th=[28967], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1664, max= 2506, per=24.29%, avg=2108.50, stdev=357.59, samples=4
   iops        : min=  416, max=  626, avg=527.00, stdev=89.21, samples=4
  lat (msec)   : 4=0.04%, 10=2.66%, 20=93.32%, 50=3.98%
  cpu          : usr=1.33%, sys=10.45%, ctx=2571, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17936: Wed Jan  8 14:00:31 2025
   read: IOPS=525, BW=2104KiB/s (2154kB/s)(5060KiB/2405msec)
    slat (usec): min=483, max=1978, avg=933.03, stdev=239.49
    clat (usec): min=2296, max=30024, avg=14172.85, stdev=2931.81
     lat (usec): min=3204, max=31523, avg=15108.01, stdev=3050.83
    clat percentiles (usec):
     |  1.00th=[ 8848],  5.00th=[10028], 10.00th=[10814], 20.00th=[11863],
     | 30.00th=[12518], 40.00th=[13173], 50.00th=[13829], 60.00th=[14484],
     | 70.00th=[15270], 80.00th=[16188], 90.00th=[17695], 95.00th=[19268],
     | 99.00th=[22938], 99.50th=[26608], 99.90th=[28181], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1632, max= 2307, per=24.21%, avg=2018.75, stdev=299.04, samples=4
   iops        : min=  408, max=  576, avg=504.50, stdev=74.52, samples=4
  write: IOPS=538, BW=2154KiB/s (2206kB/s)(5180KiB/2405msec)
    slat (usec): min=16, max=437, avg=52.07, stdev=30.52
    clat (usec): min=7493, max=30237, avg=14715.53, stdev=2768.94
     lat (usec): min=7526, max=30318, avg=14768.95, stdev=2776.57
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[11076], 10.00th=[11600], 20.00th=[12518],
     | 30.00th=[13173], 40.00th=[13698], 50.00th=[14222], 60.00th=[14877],
     | 70.00th=[15664], 80.00th=[16712], 90.00th=[18482], 95.00th=[19792],
     | 99.00th=[22676], 99.50th=[26608], 99.90th=[29230], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1632, max= 2395, per=24.02%, avg=2084.75, stdev=324.96, samples=4
   iops        : min=  408, max=  598, avg=521.00, stdev=81.00, samples=4
  lat (msec)   : 4=0.04%, 10=2.97%, 20=93.05%, 50=3.95%
  cpu          : usr=0.87%, sys=10.73%, ctx=2603, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=8337KiB/s (8537kB/s), 2072KiB/s-2104KiB/s (2121kB/s-2154kB/s), io=19.6MiB (20.5MB), run=2402-2407msec
  WRITE: bw=8680KiB/s (8888kB/s), 2154KiB/s-2192KiB/s (2206kB/s-2244kB/s), io=20.4MiB (21.4MB), run=2402-2407msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=17942: Wed Jan  8 14:00:33 2025
   read: IOPS=9, BW=392MiB/s (411MB/s)(40.0MiB/102msec)
    slat (nsec): min=6807.6k, max=6807.6k, avg=6807626.00, stdev= 0.00
    clat (nsec): min=90045k, max=90045k, avg=90045441.00, stdev= 0.00
     lat (nsec): min=96920k, max=96920k, avg=96920171.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[89654],  5.00th=[89654], 10.00th=[89654], 20.00th=[89654],
     | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654],
     | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654],
     | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.94%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=392MiB/s (411MB/s), 392MiB/s-392MiB/s (411MB/s-411MB/s), io=40.0MiB (41.9MB), run=102-102msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck