== sanity test 398c: run fio to test AIO ================= 14:20:08 (1772306408)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.112284 s, 374 MB/s
osc.lustre-OST0000-osc-ffff8fb0c95f2000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=147985: Sat Feb 28 14:20:16 2026
  write: IOPS=615, BW=2463KiB/s (2522kB/s)(10.0MiB/4158msec); 0 zone resets
    slat (usec): min=24, max=678, avg=93.74, stdev=56.61
    clat (usec): min=3154, max=47929, avg=25852.53, stdev=5683.00
     lat (usec): min=3212, max=48134, avg=25947.28, stdev=5699.52
    clat percentiles (usec):
     |  1.00th=[16057],  5.00th=[19006], 10.00th=[20055], 20.00th=[21103],
     | 30.00th=[22152], 40.00th=[23462], 50.00th=[24773], 60.00th=[26084],
     | 70.00th=[27657], 80.00th=[30016], 90.00th=[33817], 95.00th=[37487],
     | 99.00th=[42730], 99.50th=[43779], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[47973]
   bw (  KiB/s): min= 1954, max= 2928, per=25.20%, avg=2473.25, stdev=387.51, samples=8
   iops        : min=  488, max=  732, avg=618.25, stdev=96.97, samples=8
  lat (msec)   : 4=0.04%, 10=0.04%, 20=10.04%, 50=89.88%
  cpu          : usr=0.75%, sys=5.65%, ctx=1442, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147986: Sat Feb 28 14:20:16 2026
  write: IOPS=614, BW=2456KiB/s (2515kB/s)(10.0MiB/4169msec); 0 zone resets
    slat (usec): min=23, max=33340, avg=108.53, stdev=659.62
    clat (usec): min=3482, max=46655, avg=25668.42, stdev=5468.96
     lat (usec): min=3540, max=46836, avg=25778.12, stdev=5480.32
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[18744], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22152], 40.00th=[23462], 50.00th=[24511], 60.00th=[26084],
     | 70.00th=[27657], 80.00th=[29754], 90.00th=[33424], 95.00th=[36439],
     | 99.00th=[41157], 99.50th=[41681], 99.90th=[44303], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min= 1883, max= 2952, per=25.13%, avg=2466.38, stdev=409.79, samples=8
   iops        : min=  470, max=  738, avg=616.50, stdev=102.60, samples=8
  lat (msec)   : 4=0.04%, 10=0.08%, 20=9.80%, 50=90.08%
  cpu          : usr=0.89%, sys=6.36%, ctx=1398, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147987: Sat Feb 28 14:20:16 2026
  write: IOPS=613, BW=2454KiB/s (2513kB/s)(10.0MiB/4172msec); 0 zone resets
    slat (usec): min=26, max=33760, avg=107.80, stdev=667.75
    clat (usec): min=4815, max=46976, avg=25684.21, stdev=5496.29
     lat (usec): min=4888, max=47167, avg=25792.96, stdev=5518.02
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[18744], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22152], 40.00th=[23462], 50.00th=[24511], 60.00th=[26084],
     | 70.00th=[27395], 80.00th=[29754], 90.00th=[33817], 95.00th=[36963],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[45876], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1870, max= 2928, per=25.05%, avg=2457.75, stdev=410.63, samples=8
   iops        : min=  467, max=  732, avg=614.38, stdev=102.76, samples=8
  lat (msec)   : 10=0.04%, 20=10.62%, 50=89.34%
  cpu          : usr=0.84%, sys=6.19%, ctx=1449, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147988: Sat Feb 28 14:20:16 2026
  write: IOPS=613, BW=2453KiB/s (2512kB/s)(10.0MiB/4174msec); 0 zone resets
    slat (usec): min=26, max=34409, avg=109.99, stdev=680.96
    clat (usec): min=2842, max=46663, avg=25687.57, stdev=5481.15
     lat (usec): min=2914, max=48276, avg=25798.69, stdev=5513.36
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[18744], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22152], 40.00th=[23462], 50.00th=[24773], 60.00th=[26084],
     | 70.00th=[27395], 80.00th=[29754], 90.00th=[33817], 95.00th=[36963],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[45876], 99.95th=[46400],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1870, max= 2920, per=25.11%, avg=2463.75, stdev=417.63, samples=8
   iops        : min=  467, max=  730, avg=615.88, stdev=104.51, samples=8
  lat (msec)   : 4=0.04%, 20=10.08%, 50=89.88%
  cpu          : usr=0.77%, sys=6.30%, ctx=1530, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9813KiB/s (10.0MB/s), 2453KiB/s-2463KiB/s (2512kB/s-2522kB/s), io=40.0MiB (41.9MB), run=4158-4174msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=147998: Sat Feb 28 14:20:22 2026
  read: IOPS=415, BW=1662KiB/s (1701kB/s)(5048KiB/3038msec)
    slat (usec): min=633, max=4601, avg=1179.30, stdev=318.79
    clat (usec): min=3551, max=31688, avg=18091.39, stdev=2778.74
     lat (usec): min=4777, max=33865, avg=19271.86, stdev=2862.50
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[14091], 10.00th=[15008], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20055], 90.00th=[21365], 95.00th=[22414],
     | 99.00th=[26608], 99.50th=[28967], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1426, max= 1784, per=24.98%, avg=1650.00, stdev=135.34, samples=5
   iops        : min=  356, max=  446, avg=412.40, stdev=34.04, samples=5
  write: IOPS=427, BW=1709KiB/s (1750kB/s)(5192KiB/3038msec); 0 zone resets
    slat (usec): min=28, max=442, avg=71.59, stdev=38.01
    clat (usec): min=9996, max=33429, avg=18318.74, stdev=2442.28
     lat (usec): min=10087, max=33537, avg=18391.08, stdev=2448.35
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14746], 10.00th=[15533], 20.00th=[16581],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18220], 60.00th=[18744],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[20841], 95.00th=[22414],
     | 99.00th=[26346], 99.50th=[28705], 99.90th=[31851], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 1579, max= 1848, per=24.98%, avg=1717.40, stdev=101.20, samples=5
   iops        : min=  394, max=  462, avg=429.20, stdev=25.56, samples=5
  lat (msec)   : 4=0.04%, 10=0.08%, 20=81.33%, 50=18.55%
  cpu          : usr=0.99%, sys=8.10%, ctx=2341, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147999: Sat Feb 28 14:20:22 2026
  read: IOPS=411, BW=1648KiB/s (1687kB/s)(4976KiB/3020msec)
    slat (usec): min=55, max=2664, avg=1191.53, stdev=307.84
    clat (usec): min=2748, max=34209, avg=18112.03, stdev=2992.70
     lat (usec): min=3076, max=35307, avg=19304.72, stdev=3070.64
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[14091], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20317], 90.00th=[21365], 95.00th=[22676],
     | 99.00th=[27657], 99.50th=[30540], 99.90th=[32900], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1464, max= 1728, per=24.81%, avg=1638.40, stdev=117.77, samples=5
   iops        : min=  366, max=  432, avg=409.60, stdev=29.44, samples=5
  write: IOPS=435, BW=1743KiB/s (1785kB/s)(5264KiB/3020msec); 0 zone resets
    slat (usec): min=22, max=596, avg=72.55, stdev=41.38
    clat (usec): min=8411, max=31113, avg=18291.79, stdev=2470.81
     lat (usec): min=8511, max=31203, avg=18365.08, stdev=2476.39
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[15008], 10.00th=[15664], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[17957], 60.00th=[18744],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[21103], 95.00th=[22676],
     | 99.00th=[26870], 99.50th=[27919], 99.90th=[30540], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1675, max= 1928, per=25.63%, avg=1762.20, stdev=99.00, samples=5
   iops        : min=  418, max=  482, avg=440.40, stdev=24.92, samples=5
  lat (msec)   : 4=0.12%, 10=0.27%, 20=80.59%, 50=19.02%
  cpu          : usr=0.70%, sys=8.41%, ctx=2284, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148000: Sat Feb 28 14:20:22 2026
  read: IOPS=410, BW=1642KiB/s (1682kB/s)(4984KiB/3035msec)
    slat (usec): min=684, max=3109, avg=1191.50, stdev=310.48
    clat (usec): min=4306, max=34023, avg=18092.53, stdev=2883.64
     lat (usec): min=5485, max=35934, avg=19285.17, stdev=2952.64
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13829], 10.00th=[14746], 20.00th=[15795],
     | 30.00th=[16581], 40.00th=[17433], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[20317], 90.00th=[21365], 95.00th=[22676],
     | 99.00th=[26870], 99.50th=[29754], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1520, max= 1800, per=24.70%, avg=1631.40, stdev=120.16, samples=5
   iops        : min=  380, max=  450, avg=407.80, stdev=30.09, samples=5
  write: IOPS=432, BW=1732KiB/s (1773kB/s)(5256KiB/3035msec); 0 zone resets
    slat (usec): min=27, max=596, avg=71.14, stdev=38.95
    clat (usec): min=3985, max=31897, avg=18377.22, stdev=2456.44
     lat (usec): min=4110, max=32001, avg=18449.05, stdev=2463.07
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15139], 10.00th=[15926], 20.00th=[16712],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18220], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[21103], 95.00th=[22414],
     | 99.00th=[26870], 99.50th=[30540], 99.90th=[31589], 99.95th=[31851],
     | 99.99th=[31851]
   bw (  KiB/s): min= 1503, max= 1912, per=25.43%, avg=1748.60, stdev=179.01, samples=5
   iops        : min=  375, max=  478, avg=437.00, stdev=45.01, samples=5
  lat (msec)   : 4=0.04%, 10=0.12%, 20=79.84%, 50=20.00%
  cpu          : usr=0.73%, sys=8.31%, ctx=2282, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148001: Sat Feb 28 14:20:22 2026
  read: IOPS=417, BW=1668KiB/s (1708kB/s)(5060KiB/3033msec)
    slat (usec): min=657, max=4341, avg=1197.33, stdev=327.49
    clat (usec): min=2911, max=33506, avg=18010.04, stdev=2753.14
     lat (usec): min=4495, max=36670, avg=19208.53, stdev=2841.26
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14091], 10.00th=[14877], 20.00th=[15795],
     | 30.00th=[16450], 40.00th=[17171], 50.00th=[17957], 60.00th=[18482],
     | 70.00th=[19268], 80.00th=[19792], 90.00th=[21103], 95.00th=[22414],
     | 99.00th=[26084], 99.50th=[27657], 99.90th=[32375], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 1477, max= 1832, per=25.14%, avg=1660.20, stdev=126.76, samples=5
   iops        : min=  369, max=  458, avg=415.00, stdev=31.78, samples=5
  write: IOPS=426, BW=1708KiB/s (1749kB/s)(5180KiB/3033msec); 0 zone resets
    slat (usec): min=24, max=1194, avg=72.38, stdev=49.52
    clat (usec): min=10096, max=33000, avg=18362.43, stdev=2462.90
     lat (usec): min=10180, max=33065, avg=18435.52, stdev=2469.85
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15008], 10.00th=[15926], 20.00th=[16581],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18220], 60.00th=[18482],
     | 70.00th=[19006], 80.00th=[19792], 90.00th=[21103], 95.00th=[22676],
     | 99.00th=[27395], 99.50th=[30016], 99.90th=[32900], 99.95th=[32900],
     | 99.99th=[32900]
   bw (  KiB/s): min= 1547, max= 1792, per=24.91%, avg=1712.60, stdev=97.85, samples=5
   iops        : min=  386, max=  448, avg=428.00, stdev=24.78, samples=5
  lat (msec)   : 4=0.08%, 20=81.99%, 50=17.93%
  cpu          : usr=0.69%, sys=8.38%, ctx=2385, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6606KiB/s (6764kB/s), 1642KiB/s-1668KiB/s (1682kB/s-1708kB/s), io=19.6MiB (20.5MB), run=3020-3038msec
  WRITE: bw=6877KiB/s (7042kB/s), 1708KiB/s-1743KiB/s (1749kB/s-1785kB/s), io=20.4MiB (21.4MB), run=3020-3038msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=148005: Sat Feb 28 14:20:25 2026
  read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=9156.4k, max=9156.4k, avg=9156411.00, stdev= 0.00
    clat (nsec): min=80648k, max=80648k, avg=80648155.00, stdev= 0.00
     lat (nsec): min=89807k, max=89807k, avg=89806911.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[80217],  5.00th=[80217], 10.00th=[80217], 20.00th=[80217],
     | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[80217],
     | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217],
     | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.75%, ctx=2, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck