== sanity test 398c: run fio to test AIO ================= 10:55:11 (1760453711)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.13144 s, 319 MB/s
osc.lustre-OST0000-osc-ffff905e4c372800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=147347: Tue Oct 14 10:55:18 2025
  write: IOPS=723, BW=2893KiB/s (2963kB/s)(10.0MiB/3539msec); 0 zone resets
    slat (usec): min=27, max=469, avg=86.19, stdev=46.86
    clat (usec): min=1939, max=36284, avg=21970.20, stdev=4349.13
     lat (usec): min=1989, max=36407, avg=22057.42, stdev=4348.17
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[15270], 10.00th=[16712], 20.00th=[18744],
     | 30.00th=[20317], 40.00th=[21103], 50.00th=[21890], 60.00th=[22676],
     | 70.00th=[23462], 80.00th=[24773], 90.00th=[27919], 95.00th=[30016],
     | 99.00th=[32900], 99.50th=[33424], 99.90th=[36439], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2632, max= 3529, per=24.91%, avg=2882.83, stdev=332.16, samples=6
   iops        : min=  658, max=  882, avg=720.67, stdev=82.94, samples=6
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.23%, 20=28.63%, 50=71.05%
  cpu          : usr=0.76%, sys=5.99%, ctx=1385, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147348: Tue Oct 14 10:55:18 2025
  write: IOPS=723, BW=2895KiB/s (2965kB/s)(10.0MiB/3537msec); 0 zone resets
    slat (usec): min=27, max=490, avg=85.94, stdev=45.11
    clat (usec): min=1850, max=38584, avg=21970.26, stdev=4393.61
     lat (usec): min=1960, max=38688, avg=22057.11, stdev=4393.69
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[15270], 10.00th=[16909], 20.00th=[18744],
     | 30.00th=[20055], 40.00th=[21103], 50.00th=[21890], 60.00th=[22676],
     | 70.00th=[23462], 80.00th=[24773], 90.00th=[27919], 95.00th=[30016],
     | 99.00th=[33817], 99.50th=[34341], 99.90th=[38536], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 2608, max= 3535, per=24.92%, avg=2883.83, stdev=343.11, samples=6
   iops        : min=  652, max=  883, avg=720.83, stdev=85.49, samples=6
  lat (msec)   : 2=0.04%, 10=0.31%, 20=28.52%, 50=71.13%
  cpu          : usr=0.65%, sys=6.19%, ctx=1375, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147349: Tue Oct 14 10:55:18 2025
  write: IOPS=723, BW=2895KiB/s (2965kB/s)(10.0MiB/3537msec); 0 zone resets
    slat (usec): min=26, max=440, avg=84.46, stdev=43.70
    clat (usec): min=2705, max=35738, avg=21962.56, stdev=4345.58
     lat (usec): min=2752, max=35843, avg=22047.86, stdev=4345.96
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[15270], 10.00th=[16712], 20.00th=[18744],
     | 30.00th=[20055], 40.00th=[21103], 50.00th=[21890], 60.00th=[22676],
     | 70.00th=[23462], 80.00th=[24773], 90.00th=[27657], 95.00th=[30016],
     | 99.00th=[32637], 99.50th=[33817], 99.90th=[35914], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 2616, max= 3509, per=24.90%, avg=2882.17, stdev=326.97, samples=6
   iops        : min=  654, max=  877, avg=720.50, stdev=81.65, samples=6
  lat (msec)   : 4=0.04%, 10=0.20%, 20=28.40%, 50=71.37%
  cpu          : usr=0.74%, sys=5.97%, ctx=1380, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147350: Tue Oct 14 10:55:18 2025
  write: IOPS=727, BW=2911KiB/s (2981kB/s)(10.0MiB/3518msec); 0 zone resets
    slat (usec): min=22, max=584, avg=83.43, stdev=46.65
    clat (usec): min=2187, max=38206, avg=21866.91, stdev=4510.96
     lat (usec): min=2394, max=38330, avg=21951.31, stdev=4509.11
    clat percentiles (usec):
     |  1.00th=[10290],  5.00th=[14877], 10.00th=[16712], 20.00th=[18482],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[21627], 60.00th=[22676],
     | 70.00th=[23462], 80.00th=[24773], 90.00th=[27657], 95.00th=[30016],
     | 99.00th=[33162], 99.50th=[33817], 99.90th=[36439], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 2624, max= 3625, per=25.07%, avg=2901.50, stdev=373.66, samples=6
   iops        : min=  656, max=  906, avg=725.33, stdev=93.32, samples=6
  lat (msec)   : 4=0.27%, 10=0.51%, 20=29.18%, 50=70.04%
  cpu          : usr=0.68%, sys=6.03%, ctx=1455, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.3MiB/s (11.9MB/s), 2893KiB/s-2911KiB/s (2963kB/s-2981kB/s), io=40.0MiB (41.9MB), run=3518-3539msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=147360: Tue Oct 14 10:55:24 2025
  read: IOPS=392, BW=1571KiB/s (1609kB/s)(5048KiB/3213msec)
    slat (usec): min=730, max=3205, avg=1302.02, stdev=326.69
    clat (usec): min=5052, max=39005, avg=19218.79, stdev=3063.97
     lat (usec): min=6395, max=40215, avg=20522.56, stdev=3106.32
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[15008], 10.00th=[15795], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19006], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21365], 90.00th=[22676], 95.00th=[23725],
     | 99.00th=[27919], 99.50th=[32375], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1448, max= 1680, per=25.11%, avg=1568.00, stdev=85.42, samples=6
   iops        : min=  362, max=  420, avg=392.00, stdev=21.35, samples=6
  write: IOPS=403, BW=1616KiB/s (1655kB/s)(5192KiB/3213msec); 0 zone resets
    slat (usec): min=22, max=677, avg=86.97, stdev=53.40
    clat (usec): min=3585, max=42683, avg=19351.04, stdev=2539.11
     lat (usec): min=3668, max=42736, avg=19439.00, stdev=2537.75
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[15926], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21103], 90.00th=[22152], 95.00th=[23200],
     | 99.00th=[26346], 99.50th=[28967], 99.90th=[41681], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1488, max= 1688, per=24.68%, avg=1604.83, stdev=80.32, samples=6
   iops        : min=  372, max=  422, avg=401.17, stdev=20.04, samples=6
  lat (msec)   : 4=0.04%, 10=0.20%, 20=64.38%, 50=35.39%
  cpu          : usr=0.87%, sys=9.62%, ctx=2311, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147361: Tue Oct 14 10:55:24 2025
  read: IOPS=390, BW=1561KiB/s (1598kB/s)(4976KiB/3188msec)
    slat (usec): min=53, max=2724, avg=1318.48, stdev=311.38
    clat (usec): min=2928, max=38845, avg=19033.73, stdev=3105.08
     lat (usec): min=2982, max=39912, avg=20353.57, stdev=3150.66
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14615], 10.00th=[15533], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18220], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20317], 80.00th=[21103], 90.00th=[22414], 95.00th=[23725],
     | 99.00th=[28705], 99.50th=[33817], 99.90th=[38011], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1504, max= 1672, per=25.01%, avg=1561.67, stdev=61.53, samples=6
   iops        : min=  376, max=  418, avg=390.33, stdev=15.46, samples=6
  write: IOPS=412, BW=1651KiB/s (1691kB/s)(5264KiB/3188msec); 0 zone resets
    slat (usec): min=24, max=1060, avg=94.74, stdev=75.41
    clat (usec): min=5958, max=41962, avg=19324.33, stdev=2773.11
     lat (usec): min=5993, max=42055, avg=19420.06, stdev=2774.36
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[15664], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[22152], 95.00th=[23462],
     | 99.00th=[28443], 99.50th=[30802], 99.90th=[38011], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1504, max= 1792, per=25.41%, avg=1652.00, stdev=105.38, samples=6
   iops        : min=  376, max=  448, avg=413.00, stdev=26.34, samples=6
  lat (msec)   : 4=0.08%, 10=0.23%, 20=66.05%, 50=33.63%
  cpu          : usr=0.63%, sys=10.01%, ctx=2379, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147362: Tue Oct 14 10:55:24 2025
  read: IOPS=388, BW=1554KiB/s (1591kB/s)(4984KiB/3208msec)
    slat (usec): min=636, max=3353, avg=1328.67, stdev=329.53
    clat (usec): min=4225, max=36880, avg=19247.39, stdev=3078.63
     lat (usec): min=5085, max=38325, avg=20577.48, stdev=3112.78
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14877], 10.00th=[15795], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19006], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21365], 90.00th=[22676], 95.00th=[23987],
     | 99.00th=[29230], 99.50th=[32900], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1408, max= 1664, per=24.94%, avg=1557.67, stdev=98.93, samples=6
   iops        : min=  352, max=  416, avg=389.33, stdev=24.71, samples=6
  write: IOPS=409, BW=1638KiB/s (1678kB/s)(5256KiB/3208msec); 0 zone resets
    slat (usec): min=23, max=1266, avg=87.87, stdev=60.43
    clat (usec): min=13693, max=33082, avg=19266.91, stdev=2393.89
     lat (usec): min=13772, max=33150, avg=19355.69, stdev=2393.27
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[16057], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[21890], 95.00th=[23200],
     | 99.00th=[27657], 99.50th=[28967], 99.90th=[32900], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1527, max= 1856, per=25.18%, avg=1637.17, stdev=114.08, samples=6
   iops        : min=  381, max=  464, avg=409.17, stdev=28.67, samples=6
  lat (msec)   : 10=0.04%, 20=66.48%, 50=33.48%
  cpu          : usr=0.65%, sys=9.95%, ctx=2347, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147363: Tue Oct 14 10:55:24 2025
  read: IOPS=395, BW=1580KiB/s (1618kB/s)(5060KiB/3202msec)
    slat (usec): min=755, max=3147, avg=1316.81, stdev=315.05
    clat (usec): min=11061, max=37921, avg=19008.97, stdev=2871.22
     lat (usec): min=12070, max=39437, avg=20327.48, stdev=2909.29
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14877], 10.00th=[15533], 20.00th=[16581],
     | 30.00th=[17433], 40.00th=[18220], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[21103], 90.00th=[22414], 95.00th=[23462],
     | 99.00th=[26870], 99.50th=[28967], 99.90th=[36963], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1464, max= 1640, per=25.34%, avg=1582.50, stdev=71.07, samples=6
   iops        : min=  366, max=  410, avg=395.50, stdev=17.89, samples=6
  write: IOPS=404, BW=1618KiB/s (1657kB/s)(5180KiB/3202msec); 0 zone resets
    slat (usec): min=24, max=609, avg=89.08, stdev=54.52
    clat (usec): min=3252, max=37237, avg=19423.03, stdev=2675.84
     lat (usec): min=3333, max=37289, avg=19513.38, stdev=2678.72
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[15926], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18482], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21103], 90.00th=[22152], 95.00th=[23725],
     | 99.00th=[29230], 99.50th=[33424], 99.90th=[36963], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1536, max= 1728, per=24.81%, avg=1613.17, stdev=78.14, samples=6
   iops        : min=  384, max=  432, avg=403.17, stdev=19.52, samples=6
  lat (msec)   : 4=0.04%, 20=66.21%, 50=33.75%
  cpu          : usr=0.78%, sys=9.87%, ctx=2361, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6246KiB/s (6396kB/s), 1554KiB/s-1580KiB/s (1591kB/s-1618kB/s), io=19.6MiB (20.5MB), run=3188-3213msec
  WRITE: bw=6502KiB/s (6658kB/s), 1616KiB/s-1651KiB/s (1655kB/s-1691kB/s), io=20.4MiB (21.4MB), run=3188-3213msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=147367: Tue Oct 14 10:55:28 2025
  read: IOPS=11, BW=449MiB/s (471MB/s)(40.0MiB/89msec)
    slat (nsec): min=12408k, max=12408k, avg=12408272.00, stdev= 0.00
    clat (nsec): min=73076k, max=73076k, avg=73075538.00, stdev= 0.00
     lat (nsec): min=85488k, max=85488k, avg=85487790.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[72877],  5.00th=[72877], 10.00th=[72877], 20.00th=[72877],
     | 30.00th=[72877], 40.00th=[72877], 50.00th=[72877], 60.00th=[72877],
     | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877],
     | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=14.77%, ctx=2, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=40.0MiB (41.9MB), run=89-89msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck