== sanity test 398c: run fio to test AIO ================= 09:37:44 (1759844264)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.147356 s, 285 MB/s
osc.lustre-OST0000-osc-ffff9ac4d04a8000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=147459: Tue Oct  7 09:37:51 2025
  write: IOPS=691, BW=2767KiB/s (2833kB/s)(10.0MiB/3701msec); 0 zone resets
    slat (usec): min=24, max=525, avg=92.15, stdev=53.95
    clat (usec): min=12445, max=50368, avg=22974.57, stdev=4722.96
     lat (usec): min=12516, max=50599, avg=23067.77, stdev=4732.43
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[17695], 10.00th=[18744], 20.00th=[19792],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23987], 80.00th=[24773], 90.00th=[27132], 95.00th=[29754],
     | 99.00th=[43779], 99.50th=[45876], 99.90th=[50594], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min= 2192, max= 2952, per=24.95%, avg=2761.00, stdev=261.62, samples=7
   iops        : min=  548, max=  738, avg=690.14, stdev=65.35, samples=7
  lat (msec)   : 20=22.34%, 50=77.54%, 100=0.12%
  cpu          : usr=0.89%, sys=6.11%, ctx=1229, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147460: Tue Oct  7 09:37:51 2025
  write: IOPS=693, BW=2774KiB/s (2841kB/s)(10.0MiB/3691msec); 0 zone resets
    slat (usec): min=25, max=587, avg=96.16, stdev=57.77
    clat (usec): min=3414, max=50214, avg=22936.88, stdev=4668.79
     lat (usec): min=3489, max=50418, avg=23034.15, stdev=4681.93
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[17433], 10.00th=[18482], 20.00th=[19792],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22414], 60.00th=[22938],
     | 70.00th=[23987], 80.00th=[24773], 90.00th=[27395], 95.00th=[30278],
     | 99.00th=[42730], 99.50th=[45876], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[50070]
   bw (  KiB/s): min= 2152, max= 2952, per=25.03%, avg=2770.14, stdev=281.06, samples=7
   iops        : min=  538, max=  738, avg=692.43, stdev=70.19, samples=7
  lat (msec)   : 4=0.04%, 10=0.12%, 20=20.66%, 50=79.14%, 100=0.04%
  cpu          : usr=0.79%, sys=6.42%, ctx=1432, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147461: Tue Oct  7 09:37:51 2025
  write: IOPS=691, BW=2767KiB/s (2833kB/s)(10.0MiB/3701msec); 0 zone resets
    slat (usec): min=25, max=26734, avg=101.72, stdev=529.34
    clat (usec): min=2390, max=46769, avg=22795.34, stdev=4542.04
     lat (usec): min=2459, max=46897, avg=22898.14, stdev=4558.87
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[17433], 10.00th=[18482], 20.00th=[19792],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22414], 60.00th=[23200],
     | 70.00th=[23725], 80.00th=[24773], 90.00th=[26608], 95.00th=[29230],
     | 99.00th=[42730], 99.50th=[45876], 99.90th=[46400], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 2184, max= 2952, per=24.95%, avg=2760.71, stdev=262.43, samples=7
   iops        : min=  546, max=  738, avg=690.14, stdev=65.60, samples=7
  lat (msec)   : 4=0.12%, 10=0.08%, 20=22.23%, 50=77.58%
  cpu          : usr=0.86%, sys=6.84%, ctx=1305, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147462: Tue Oct  7 09:37:51 2025
  write: IOPS=692, BW=2769KiB/s (2836kB/s)(10.0MiB/3698msec); 0 zone resets
    slat (usec): min=25, max=26295, avg=104.42, stdev=520.54
    clat (usec): min=3059, max=46749, avg=22778.23, stdev=4584.73
     lat (usec): min=3134, max=46944, avg=22883.67, stdev=4593.71
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17433], 10.00th=[18482], 20.00th=[19792],
     | 30.00th=[20579], 40.00th=[21365], 50.00th=[22152], 60.00th=[22938],
     | 70.00th=[23725], 80.00th=[24773], 90.00th=[26870], 95.00th=[29492],
     | 99.00th=[42730], 99.50th=[45876], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46924]
   bw (  KiB/s): min= 2168, max= 2952, per=24.96%, avg=2761.86, stdev=269.36, samples=7
   iops        : min=  542, max=  738, avg=690.43, stdev=67.32, samples=7
  lat (msec)   : 4=0.08%, 10=0.16%, 20=21.52%, 50=78.24%
  cpu          : usr=0.68%, sys=7.17%, ctx=1421, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.8MiB/s (11.3MB/s), 2767KiB/s-2774KiB/s (2833kB/s-2841kB/s), io=40.0MiB (41.9MB), run=3691-3701msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=147472: Tue Oct  7 09:37:58 2025
  read: IOPS=393, BW=1576KiB/s (1613kB/s)(5048KiB/3204msec)
    slat (usec): min=27, max=3076, avg=1271.51, stdev=317.74
    clat (usec): min=2199, max=35435, avg=19215.68, stdev=3287.21
     lat (usec): min=2254, max=36572, avg=20488.67, stdev=3362.16
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[14484], 10.00th=[15664], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19006], 60.00th=[19792],
     | 70.00th=[20579], 80.00th=[21627], 90.00th=[22676], 95.00th=[23987],
     | 99.00th=[30016], 99.50th=[32637], 99.90th=[34866], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1448, max= 1672, per=25.19%, avg=1571.17, stdev=85.13, samples=6
   iops        : min=  362, max=  418, avg=392.67, stdev=21.34, samples=6
  write: IOPS=405, BW=1620KiB/s (1659kB/s)(5192KiB/3204msec); 0 zone resets
    slat (usec): min=26, max=558, avg=84.07, stdev=47.98
    clat (usec): min=5947, max=34739, avg=19411.16, stdev=2734.33
     lat (usec): min=5978, max=34837, avg=19496.45, stdev=2730.12
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[15533], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19530], 60.00th=[20055],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22152], 95.00th=[23725],
     | 99.00th=[27132], 99.50th=[30802], 99.90th=[33817], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  KiB/s): min= 1448, max= 1768, per=24.90%, avg=1617.00, stdev=125.29, samples=6
   iops        : min=  362, max=  442, avg=404.17, stdev=31.24, samples=6
  lat (msec)   : 4=0.16%, 10=0.47%, 20=60.82%, 50=38.55%
  cpu          : usr=0.97%, sys=9.05%, ctx=2221, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147473: Tue Oct  7 09:37:58 2025
  read: IOPS=386, BW=1547KiB/s (1584kB/s)(4976KiB/3217msec)
    slat (usec): min=714, max=2579, avg=1282.36, stdev=302.82
    clat (usec): min=1496, max=33710, avg=19244.47, stdev=2985.99
     lat (usec): min=2555, max=35205, avg=20528.38, stdev=3018.97
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14746], 10.00th=[15795], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20579], 80.00th=[21627], 90.00th=[22676], 95.00th=[23987],
     | 99.00th=[27132], 99.50th=[29754], 99.90th=[32637], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1336, max= 1648, per=24.70%, avg=1540.83, stdev=111.25, samples=6
   iops        : min=  334, max=  412, avg=385.17, stdev=27.83, samples=6
  write: IOPS=409, BW=1636KiB/s (1676kB/s)(5264KiB/3217msec); 0 zone resets
    slat (usec): min=26, max=755, avg=85.59, stdev=52.30
    clat (usec): min=3801, max=30859, avg=19451.61, stdev=2596.85
     lat (usec): min=3892, max=30948, avg=19538.17, stdev=2592.90
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15795], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22414], 95.00th=[23462],
     | 99.00th=[28181], 99.50th=[28967], 99.90th=[30540], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1520, max= 1728, per=25.14%, avg=1632.33, stdev=80.52, samples=6
   iops        : min=  380, max=  432, avg=408.00, stdev=20.04, samples=6
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.04%, 20=61.64%, 50=38.20%
  cpu          : usr=0.84%, sys=9.24%, ctx=2186, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147474: Tue Oct  7 09:37:58 2025
  read: IOPS=387, BW=1549KiB/s (1586kB/s)(4984KiB/3217msec)
    slat (usec): min=710, max=3928, avg=1286.82, stdev=306.43
    clat (usec): min=4843, max=32728, avg=19377.56, stdev=3116.79
     lat (usec): min=6286, max=34693, avg=20665.91, stdev=3139.59
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14746], 10.00th=[15664], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20841], 80.00th=[21627], 90.00th=[23200], 95.00th=[24511],
     | 99.00th=[28443], 99.50th=[30802], 99.90th=[32375], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1448, max= 1623, per=24.75%, avg=1543.83, stdev=62.12, samples=6
   iops        : min=  362, max=  405, avg=385.83, stdev=15.34, samples=6
  write: IOPS=408, BW=1634KiB/s (1673kB/s)(5256KiB/3217msec); 0 zone resets
    slat (usec): min=26, max=652, avg=83.37, stdev=47.31
    clat (usec): min=2492, max=31074, avg=19334.85, stdev=2628.15
     lat (usec): min=2574, max=31244, avg=19419.31, stdev=2625.39
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[15664], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21365], 90.00th=[22414], 95.00th=[23462],
     | 99.00th=[27132], 99.50th=[28181], 99.90th=[30802], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1408, max= 1840, per=24.93%, avg=1619.17, stdev=146.85, samples=6
   iops        : min=  352, max=  460, avg=404.67, stdev=36.76, samples=6
  lat (msec)   : 4=0.04%, 10=0.35%, 20=60.98%, 50=38.63%
  cpu          : usr=1.09%, sys=8.86%, ctx=2166, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147475: Tue Oct  7 09:37:58 2025
  read: IOPS=393, BW=1575KiB/s (1613kB/s)(5060KiB/3212msec)
    slat (usec): min=693, max=2560, avg=1276.24, stdev=292.25
    clat (usec): min=2737, max=33711, avg=19220.08, stdev=2838.60
     lat (usec): min=4385, max=35318, avg=20497.80, stdev=2868.45
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14877], 10.00th=[15795], 20.00th=[16909],
     | 30.00th=[17695], 40.00th=[18482], 50.00th=[19268], 60.00th=[20055],
     | 70.00th=[20579], 80.00th=[21365], 90.00th=[22676], 95.00th=[23725],
     | 99.00th=[26870], 99.50th=[28967], 99.90th=[32113], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1432, max= 1680, per=25.23%, avg=1573.67, stdev=89.34, samples=6
   iops        : min=  358, max=  420, avg=393.33, stdev=22.37, samples=6
  write: IOPS=403, BW=1613KiB/s (1651kB/s)(5180KiB/3212msec); 0 zone resets
    slat (usec): min=28, max=425, avg=88.64, stdev=49.93
    clat (usec): min=11521, max=32646, avg=19373.41, stdev=2648.12
     lat (usec): min=11587, max=32702, avg=19462.94, stdev=2642.95
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15533], 10.00th=[16319], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[21103], 90.00th=[22414], 95.00th=[23462],
     | 99.00th=[29230], 99.50th=[31065], 99.90th=[32113], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1456, max= 1760, per=24.71%, avg=1604.83, stdev=108.81, samples=6
   iops        : min=  364, max=  440, avg=401.17, stdev=27.19, samples=6
  lat (msec)   : 4=0.04%, 20=62.27%, 50=37.70%
  cpu          : usr=0.53%, sys=9.59%, ctx=2273, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6238KiB/s (6388kB/s), 1547KiB/s-1576KiB/s (1584kB/s-1613kB/s), io=19.6MiB (20.5MB), run=3204-3217msec
  WRITE: bw=6494KiB/s (6650kB/s), 1613KiB/s-1636KiB/s (1651kB/s-1676kB/s), io=20.4MiB (21.4MB), run=3204-3217msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=147479: Tue Oct  7 09:38:01 2025
  read: IOPS=12, BW=519MiB/s (545MB/s)(40.0MiB/77msec)
    slat (nsec): min=9985.7k, max=9985.7k, avg=9985710.00, stdev= 0.00
    clat (nsec): min=63616k, max=63616k, avg=63615910.00, stdev= 0.00
     lat (nsec): min=73605k, max=73605k, avg=73604928.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[63701],  5.00th=[63701], 10.00th=[63701], 20.00th=[63701],
     | 30.00th=[63701], 40.00th=[63701], 50.00th=[63701], 60.00th=[63701],
     | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701],
     | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=13.16%, ctx=2, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=40.0MiB (41.9MB), run=77-77msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck