== sanity test 398c: run fio to test AIO ================= 04:14:36 (1780388076)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.127811 s, 328 MB/s
osc.lustre-OST0000-osc-ffff93c183390800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=184704: Tue Jun  2 04:14:45 2026
  write: IOPS=472, BW=1890KiB/s (1935kB/s)(10.0MiB/5418msec); 0 zone resets
    slat (usec): min=46, max=3168, avg=191.94, stdev=114.53
    clat (usec): min=3524, max=58075, avg=33355.52, stdev=3788.27
     lat (usec): min=3771, max=58183, avg=33548.69, stdev=3794.61
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[28443], 10.00th=[29754], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[43254], 99.50th=[46924], 99.90th=[52691], 99.95th=[55837],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1798, max= 1960, per=24.98%, avg=1886.60, stdev=58.33, samples=10
   iops        : min=  449, max=  490, avg=471.60, stdev=14.67, samples=10
  lat (msec)   : 4=0.04%, 10=0.12%, 20=0.16%, 50=99.41%, 100=0.27%
  cpu          : usr=0.55%, sys=7.24%, ctx=2709, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184705: Tue Jun  2 04:14:45 2026
  write: IOPS=472, BW=1888KiB/s (1934kB/s)(10.0MiB/5423msec); 0 zone resets
    slat (usec): min=41, max=855, avg=180.66, stdev=95.91
    clat (usec): min=8740, max=47001, avg=33351.37, stdev=3549.86
     lat (usec): min=8910, max=47192, avg=33533.26, stdev=3557.10
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[28181], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[42206], 99.50th=[44827], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1769, max= 1932, per=24.88%, avg=1878.90, stdev=58.34, samples=10
   iops        : min=  442, max=  483, avg=469.70, stdev=14.64, samples=10
  lat (msec)   : 10=0.04%, 20=0.20%, 50=99.77%
  cpu          : usr=0.63%, sys=6.31%, ctx=2584, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184706: Tue Jun  2 04:14:45 2026
  write: IOPS=475, BW=1900KiB/s (1946kB/s)(10.0MiB/5389msec); 0 zone resets
    slat (usec): min=40, max=904, avg=187.52, stdev=97.09
    clat (usec): min=7514, max=57682, avg=33437.59, stdev=3818.49
     lat (usec): min=7730, max=57793, avg=33626.48, stdev=3820.83
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[28181], 10.00th=[29754], 20.00th=[31065],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[42730], 99.50th=[44827], 99.90th=[54789], 99.95th=[57410],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1792, max= 2032, per=25.10%, avg=1895.50, stdev=67.70, samples=10
   iops        : min=  448, max=  508, avg=473.80, stdev=16.86, samples=10
  lat (msec)   : 10=0.04%, 20=0.62%, 50=99.10%, 100=0.23%
  cpu          : usr=0.56%, sys=6.76%, ctx=2596, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184707: Tue Jun  2 04:14:45 2026
  write: IOPS=473, BW=1892KiB/s (1938kB/s)(10.0MiB/5411msec); 0 zone resets
    slat (usec): min=48, max=27348, avg=203.78, stdev=544.61
    clat (usec): min=9933, max=53934, avg=33342.07, stdev=3688.92
     lat (usec): min=13536, max=54107, avg=33547.00, stdev=3668.51
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[28181], 10.00th=[29754], 20.00th=[31065],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[37487], 95.00th=[38536],
     | 99.00th=[43254], 99.50th=[44827], 99.90th=[50070], 99.95th=[50070],
     | 99.99th=[53740]
   bw (  KiB/s): min= 1800, max= 1968, per=25.02%, avg=1889.60, stdev=47.26, samples=10
   iops        : min=  450, max=  492, avg=472.40, stdev=11.82, samples=10
  lat (msec)   : 10=0.04%, 20=0.62%, 50=99.22%, 100=0.12%
  cpu          : usr=0.67%, sys=7.43%, ctx=2619, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7553KiB/s (7734kB/s), 1888KiB/s-1900KiB/s (1934kB/s-1946kB/s), io=40.0MiB (41.9MB), run=5389-5423msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=184717: Tue Jun  2 04:14:55 2026
  read: IOPS=204, BW=816KiB/s (836kB/s)(5048KiB/6185msec)
    slat (usec): min=1023, max=8271, avg=2605.54, stdev=855.44
    clat (usec): min=15308, max=64301, avg=36666.69, stdev=6875.75
     lat (usec): min=16953, max=67529, avg=39273.96, stdev=7144.20
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26870], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[54789], 99.50th=[57934], 99.90th=[62653], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  592, max= 1000, per=25.06%, avg=813.00, stdev=124.65, samples=12
   iops        : min=  148, max=  250, avg=203.25, stdev=31.16, samples=12
  write: IOPS=209, BW=839KiB/s (860kB/s)(5192KiB/6185msec); 0 zone resets
    slat (usec): min=55, max=2582, avg=176.25, stdev=125.34
    clat (usec): min=15188, max=60789, avg=37204.17, stdev=6117.61
     lat (usec): min=15300, max=60905, avg=37381.38, stdev=6121.49
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29230], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[60556], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  672, max= 1008, per=24.86%, avg=839.67, stdev=90.10, samples=12
   iops        : min=  168, max=  252, avg=209.92, stdev=22.52, samples=12
  lat (msec)   : 20=0.20%, 50=95.59%, 100=4.22%
  cpu          : usr=0.45%, sys=6.34%, ctx=4717, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184718: Tue Jun  2 04:14:55 2026
  read: IOPS=201, BW=805KiB/s (825kB/s)(4976KiB/6180msec)
    slat (usec): min=950, max=7385, avg=2643.42, stdev=870.99
    clat (usec): min=15512, max=59057, avg=36754.43, stdev=6479.45
     lat (usec): min=19011, max=61540, avg=39399.64, stdev=6726.69
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27395], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[49021],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[57934], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  592, max=  984, per=24.75%, avg=802.83, stdev=112.22, samples=12
   iops        : min=  148, max=  246, avg=200.67, stdev=28.10, samples=12
  write: IOPS=212, BW=852KiB/s (872kB/s)(5264KiB/6180msec); 0 zone resets
    slat (usec): min=55, max=2256, avg=177.48, stdev=108.54
    clat (usec): min=19837, max=61429, avg=37281.72, stdev=5863.24
     lat (usec): min=20001, max=61536, avg=37460.18, stdev=5870.17
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[58459], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  744, max=  984, per=25.23%, avg=852.00, stdev=84.90, samples=12
   iops        : min=  186, max=  246, avg=213.00, stdev=21.23, samples=12
  lat (msec)   : 20=0.12%, 50=96.29%, 100=3.59%
  cpu          : usr=0.37%, sys=6.36%, ctx=4705, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184719: Tue Jun  2 04:14:55 2026
  read: IOPS=201, BW=808KiB/s (827kB/s)(4984KiB/6170msec)
    slat (usec): min=39, max=8330, avg=2648.26, stdev=921.59
    clat (usec): min=10322, max=60532, avg=36965.58, stdev=6884.07
     lat (usec): min=12735, max=65669, avg=39615.61, stdev=7154.02
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[54264], 99.50th=[56886], 99.90th=[59507], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  720, max=  960, per=24.99%, avg=810.67, stdev=83.58, samples=12
   iops        : min=  180, max=  240, avg=202.67, stdev=20.90, samples=12
  write: IOPS=212, BW=852KiB/s (872kB/s)(5256KiB/6170msec); 0 zone resets
    slat (usec): min=40, max=1698, avg=173.90, stdev=101.86
    clat (usec): min=11471, max=58899, avg=37249.73, stdev=6077.33
     lat (usec): min=11624, max=58989, avg=37424.66, stdev=6078.99
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28967], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  616, max= 1008, per=25.20%, avg=850.92, stdev=110.80, samples=12
   iops        : min=  154, max=  252, avg=212.67, stdev=27.73, samples=12
  lat (msec)   : 20=0.23%, 50=95.98%, 100=3.79%
  cpu          : usr=0.45%, sys=6.31%, ctx=4589, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184720: Tue Jun  2 04:14:55 2026
  read: IOPS=204, BW=819KiB/s (838kB/s)(5060KiB/6181msec)
    slat (usec): min=1029, max=7970, avg=2625.44, stdev=873.51
    clat (usec): min=18740, max=65148, avg=36687.38, stdev=6449.81
     lat (usec): min=20671, max=69095, avg=39314.62, stdev=6694.43
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27395], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45876], 95.00th=[48497],
     | 99.00th=[53740], 99.50th=[54789], 99.90th=[59507], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  672, max= 1000, per=25.25%, avg=819.17, stdev=104.72, samples=12
   iops        : min=  168, max=  250, avg=204.75, stdev=26.22, samples=12
  write: IOPS=209, BW=838KiB/s (858kB/s)(5180KiB/6181msec); 0 zone resets
    slat (usec): min=53, max=1238, avg=172.97, stdev=96.15
    clat (usec): min=20676, max=63120, avg=37347.02, stdev=6206.63
     lat (usec): min=20741, max=63231, avg=37520.99, stdev=6210.92
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29230], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[46400], 95.00th=[49546],
     | 99.00th=[54264], 99.50th=[55837], 99.90th=[58459], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  648, max=  976, per=24.75%, avg=835.67, stdev=94.34, samples=12
   iops        : min=  162, max=  244, avg=208.92, stdev=23.59, samples=12
  lat (msec)   : 20=0.04%, 50=96.09%, 100=3.87%
  cpu          : usr=0.44%, sys=6.36%, ctx=4671, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3245KiB/s (3322kB/s), 805KiB/s-819KiB/s (825kB/s-838kB/s), io=19.6MiB (20.5MB), run=6170-6185msec
  WRITE: bw=3378KiB/s (3459kB/s), 838KiB/s-852KiB/s (858kB/s-872kB/s), io=20.4MiB (21.4MB), run=6170-6185msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=184725: Tue Jun  2 04:14:58 2026
  read: IOPS=10, BW=430MiB/s (451MB/s)(40.0MiB/93msec)
    slat (nsec): min=13214k, max=13214k, avg=13214085.00, stdev= 0.00
    clat (nsec): min=74304k, max=74304k, avg=74304336.00, stdev= 0.00
     lat (nsec): min=87522k, max=87522k, avg=87522485.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[73925],  5.00th=[73925], 10.00th=[73925], 20.00th=[73925],
     | 30.00th=[73925], 40.00th=[73925], 50.00th=[73925], 60.00th=[73925],
     | 70.00th=[73925], 80.00th=[73925], 90.00th=[73925], 95.00th=[73925],
     | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.96%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=430MiB/s (451MB/s), 430MiB/s-430MiB/s (451MB/s-451MB/s), io=40.0MiB (41.9MB), run=93-93msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout