== sanity test 398c: run fio to test AIO ================= 07:14:33 (1747826073)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0993126 s, 422 MB/s
osc.lustre-OST0000-osc-ffff88012b4f4000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30611: Wed May 21 07:14:38 2025
  write: IOPS=640, BW=2560KiB/s (2621kB/s)(10.0MiB/4000msec)
    slat (usec): min=10, max=20089, avg=66.54, stdev=398.27
    clat (usec): min=1928, max=58213, avg=24819.66, stdev=7464.22
     lat (usec): min=1971, max=58236, avg=24887.38, stdev=7468.74
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15401], 10.00th=[16909], 20.00th=[18482],
     | 30.00th=[20055], 40.00th=[21365], 50.00th=[23462], 60.00th=[25035],
     | 70.00th=[27657], 80.00th=[30540], 90.00th=[35390], 95.00th=[39584],
     | 99.00th=[46400], 99.50th=[48497], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[58459]
   bw (  KiB/s): min= 2056, max= 3488, per=25.84%, avg=2643.43, stdev=553.96, samples=7
   iops        : min=  514, max=  872, avg=660.86, stdev=138.49, samples=7
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.04%, 20=29.41%, 50=70.04%
  lat (msec)   : 100=0.39%
  cpu          : usr=1.00%, sys=5.03%, ctx=1417, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30612: Wed May 21 07:14:38 2025
  write: IOPS=640, BW=2560KiB/s (2621kB/s)(10.0MiB/4000msec)
    slat (usec): min=10, max=20192, avg=63.17, stdev=400.34
    clat (usec): min=2300, max=54686, avg=24830.32, stdev=7393.56
     lat (usec): min=2373, max=54790, avg=24894.73, stdev=7406.26
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15533], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[20055], 40.00th=[21365], 50.00th=[23462], 60.00th=[25035],
     | 70.00th=[27657], 80.00th=[30540], 90.00th=[35390], 95.00th=[39060],
     | 99.00th=[46400], 99.50th=[47449], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min= 2056, max= 3440, per=25.82%, avg=2641.14, stdev=539.54, samples=7
   iops        : min=  514, max=  860, avg=660.29, stdev=134.88, samples=7
  lat (msec)   : 4=0.08%, 10=0.08%, 20=29.57%, 50=70.00%, 100=0.27%
  cpu          : usr=0.83%, sys=4.83%, ctx=1221, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30613: Wed May 21 07:14:38 2025
  write: IOPS=643, BW=2573KiB/s (2635kB/s)(10.0MiB/3980msec)
    slat (usec): min=10, max=617, avg=55.97, stdev=43.64
    clat (usec): min=2562, max=56150, avg=24777.49, stdev=7430.19
     lat (usec): min=2604, max=56186, avg=24834.59, stdev=7434.86
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15533], 10.00th=[16712], 20.00th=[18482],
     | 30.00th=[20055], 40.00th=[21365], 50.00th=[23200], 60.00th=[25035],
     | 70.00th=[27657], 80.00th=[30540], 90.00th=[35390], 95.00th=[39060],
     | 99.00th=[46400], 99.50th=[47973], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[56361]
   bw (  KiB/s): min= 2064, max= 3472, per=26.01%, avg=2660.57, stdev=548.90, samples=7
   iops        : min=  516, max=  868, avg=665.14, stdev=137.22, samples=7
  lat (msec)   : 4=0.04%, 10=0.08%, 20=30.31%, 50=69.26%, 100=0.31%
  cpu          : usr=1.76%, sys=3.57%, ctx=1278, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30614: Wed May 21 07:14:38 2025
  write: IOPS=639, BW=2557KiB/s (2619kB/s)(10.0MiB/4004msec)
    slat (usec): min=9, max=18420, avg=64.61, stdev=367.88
    clat (usec): min=1757, max=52026, avg=24814.21, stdev=7434.04
     lat (usec): min=1815, max=52064, avg=24879.95, stdev=7440.29
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15533], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[20055], 40.00th=[21365], 50.00th=[23200], 60.00th=[25035],
     | 70.00th=[27657], 80.00th=[30540], 90.00th=[35390], 95.00th=[40109],
     | 99.00th=[45876], 99.50th=[47449], 99.90th=[52167], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min= 1856, max= 3488, per=24.88%, avg=2545.00, stdev=574.31, samples=8
   iops        : min=  464, max=  872, avg=636.25, stdev=143.58, samples=8
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.04%, 20=29.02%, 50=70.74%
  lat (msec)   : 100=0.12%
  cpu          : usr=0.82%, sys=4.67%, ctx=1192, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9.99MiB/s (10.5MB/s), 2557KiB/s-2573KiB/s (2619kB/s-2635kB/s), io=40.0MiB (41.9MB), run=3980-4004msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30627: Wed May 21 07:14:42 2025
   read: IOPS=456, BW=1826KiB/s (1869kB/s)(5048KiB/2765msec)
    slat (usec): min=513, max=3234, avg=1017.23, stdev=316.50
    clat (usec): min=2521, max=33984, avg=16449.44, stdev=3655.09
     lat (usec): min=3450, max=36240, avg=17468.77, stdev=3787.65
    clat percentiles (usec):
     |  1.00th=[10159],  5.00th=[11207], 10.00th=[12387], 20.00th=[13304],
     | 30.00th=[14353], 40.00th=[15270], 50.00th=[16057], 60.00th=[16909],
     | 70.00th=[17957], 80.00th=[19006], 90.00th=[21103], 95.00th=[22938],
     | 99.00th=[27132], 99.50th=[31327], 99.90th=[33162], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1472, max= 2104, per=24.98%, avg=1812.80, stdev=245.71, samples=5
   iops        : min=  368, max=  526, avg=453.20, stdev=61.43, samples=5
  write: IOPS=469, BW=1878KiB/s (1923kB/s)(5192KiB/2765msec)
    slat (usec): min=12, max=1394, avg=49.67, stdev=55.91
    clat (usec): min=3814, max=32348, avg=16818.87, stdev=3661.60
     lat (usec): min=3884, max=32414, avg=16869.67, stdev=3667.16
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12125], 10.00th=[13042], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15401], 50.00th=[16057], 60.00th=[16909],
     | 70.00th=[17957], 80.00th=[19530], 90.00th=[21627], 95.00th=[23987],
     | 99.00th=[28443], 99.50th=[29754], 99.90th=[32113], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1536, max= 2136, per=25.01%, avg=1889.60, stdev=225.52, samples=5
   iops        : min=  384, max=  534, avg=472.40, stdev=56.38, samples=5
  lat (msec)   : 4=0.12%, 10=0.31%, 20=83.87%, 50=15.70%
  cpu          : usr=1.30%, sys=8.21%, ctx=2413, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30628: Wed May 21 07:14:42 2025
   read: IOPS=453, BW=1815KiB/s (1858kB/s)(4976KiB/2742msec)
    slat (usec): min=14, max=3289, avg=1001.23, stdev=301.22
    clat (usec): min=1063, max=34412, avg=16312.78, stdev=3783.57
     lat (usec): min=1679, max=35821, avg=17316.19, stdev=3904.60
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[11338], 10.00th=[12256], 20.00th=[13173],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15664], 60.00th=[16450],
     | 70.00th=[17695], 80.00th=[19268], 90.00th=[21365], 95.00th=[22938],
     | 99.00th=[27395], 99.50th=[29754], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1456, max= 2112, per=24.80%, avg=1800.00, stdev=241.99, samples=5
   iops        : min=  364, max=  528, avg=450.00, stdev=60.50, samples=5
  write: IOPS=479, BW=1920KiB/s (1966kB/s)(5264KiB/2742msec)
    slat (usec): min=11, max=1033, avg=46.86, stdev=37.69
    clat (usec): min=3148, max=33169, avg=16845.31, stdev=3753.06
     lat (usec): min=3190, max=33232, avg=16893.36, stdev=3754.81
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12256], 10.00th=[12911], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15401], 50.00th=[16057], 60.00th=[17171],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[21365], 95.00th=[23987],
     | 99.00th=[29754], 99.50th=[31589], 99.90th=[32375], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1672, max= 2112, per=25.58%, avg=1932.80, stdev=172.75, samples=5
   iops        : min=  418, max=  528, avg=483.20, stdev=43.19, samples=5
  lat (msec)   : 2=0.12%, 4=0.08%, 10=0.82%, 20=83.71%, 50=15.27%
  cpu          : usr=1.17%, sys=8.21%, ctx=2399, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30629: Wed May 21 07:14:42 2025
   read: IOPS=450, BW=1803KiB/s (1846kB/s)(4984KiB/2765msec)
    slat (usec): min=514, max=3402, avg=1014.63, stdev=312.76
    clat (usec): min=1267, max=35536, avg=16511.89, stdev=3758.64
     lat (usec): min=2628, max=37195, avg=17528.66, stdev=3881.83
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[11207], 10.00th=[12256], 20.00th=[13435],
     | 30.00th=[14353], 40.00th=[15139], 50.00th=[16057], 60.00th=[16909],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[21627], 95.00th=[23462],
     | 99.00th=[27395], 99.50th=[30016], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1496, max= 1984, per=24.87%, avg=1804.80, stdev=195.60, samples=5
   iops        : min=  374, max=  496, avg=451.20, stdev=48.90, samples=5
  write: IOPS=475, BW=1901KiB/s (1947kB/s)(5256KiB/2765msec)
    slat (usec): min=12, max=432, avg=48.90, stdev=35.10
    clat (usec): min=2784, max=32632, avg=16804.41, stdev=3536.67
     lat (usec): min=2823, max=32679, avg=16854.42, stdev=3537.87
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12518], 10.00th=[12911], 20.00th=[13960],
     | 30.00th=[14615], 40.00th=[15533], 50.00th=[16188], 60.00th=[17171],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[21103], 95.00th=[23462],
     | 99.00th=[28443], 99.50th=[30016], 99.90th=[32375], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1512, max= 2224, per=25.01%, avg=1889.60, stdev=279.45, samples=5
   iops        : min=  378, max=  556, avg=472.40, stdev=69.86, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.59%, 20=84.14%, 50=15.20%
  cpu          : usr=0.90%, sys=8.50%, ctx=2399, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30630: Wed May 21 07:14:42 2025
   read: IOPS=458, BW=1834KiB/s (1878kB/s)(5060KiB/2759msec)
    slat (usec): min=477, max=3425, avg=1016.07, stdev=313.38
    clat (usec): min=2534, max=35812, avg=16334.45, stdev=3700.85
     lat (usec): min=3405, max=37297, avg=17352.50, stdev=3831.74
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[11469], 10.00th=[12125], 20.00th=[13435],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[17433], 80.00th=[19006], 90.00th=[21103], 95.00th=[23462],
     | 99.00th=[28443], 99.50th=[30016], 99.90th=[32900], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1504, max= 2160, per=25.31%, avg=1836.80, stdev=242.16, samples=5
   iops        : min=  376, max=  540, avg=459.20, stdev=60.54, samples=5
  write: IOPS=469, BW=1877KiB/s (1923kB/s)(5180KiB/2759msec)
    slat (usec): min=10, max=899, avg=48.08, stdev=37.43
    clat (usec): min=2104, max=34962, avg=16901.62, stdev=3636.13
     lat (usec): min=2158, max=35012, avg=16950.83, stdev=3637.73
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12256], 10.00th=[13042], 20.00th=[14091],
     | 30.00th=[14746], 40.00th=[15533], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[21627], 95.00th=[23462],
     | 99.00th=[28705], 99.50th=[30540], 99.90th=[32375], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1504, max= 2088, per=24.76%, avg=1870.40, stdev=225.38, samples=5
   iops        : min=  376, max=  522, avg=467.60, stdev=56.35, samples=5
  lat (msec)   : 4=0.16%, 10=0.47%, 20=83.67%, 50=15.70%
  cpu          : usr=1.05%, sys=8.30%, ctx=2486, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7258KiB/s (7432kB/s), 1803KiB/s-1834KiB/s (1846kB/s-1878kB/s), io=19.6MiB (20.5MB), run=2742-2765msec
  WRITE: bw=7556KiB/s (7737kB/s), 1877KiB/s-1920KiB/s (1923kB/s-1966kB/s), io=20.4MiB (21.4MB), run=2742-2765msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30636: Wed May 21 07:14:44 2025
   read: IOPS=11, BW=455MiB/s (477MB/s)(40.0MiB/88msec)
    slat (nsec): min=6333.9k, max=6333.9k, avg=6333927.00, stdev= 0.00
    clat (nsec): min=78097k, max=78097k, avg=78096625.00, stdev= 0.00
     lat (nsec): min=84454k, max=84454k, avg=84454052.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[78119],  5.00th=[78119], 10.00th=[78119], 20.00th=[78119],
     | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119],
     | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119],
     | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.90%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=40.0MiB (41.9MB), run=88-88msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck