== sanity test 398c: run fio to test AIO ================= 01:02:35 (1782363755)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.127485 s, 329 MB/s
osc.lustre-OST0000-osc-ffff8cd1adf77800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=188963: Thu Jun 25 01:02:44 2026
  write: IOPS=476, BW=1905KiB/s (1951kB/s)(10.0MiB/5374msec); 0 zone resets
    slat (usec): min=46, max=24252, avg=181.90, stdev=486.45
    clat (usec): min=4959, max=56504, avg=33104.56, stdev=3875.21
     lat (usec): min=13703, max=56926, avg=33287.61, stdev=3844.80
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27919], 10.00th=[28967], 20.00th=[30278],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[32637], 60.00th=[33424],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[43779], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[56361]
   bw (  KiB/s): min= 1761, max= 2048, per=25.06%, avg=1909.70, stdev=86.88, samples=10
   iops        : min=  440, max=  512, avg=477.40, stdev=21.77, samples=10
  lat (msec)   : 10=0.04%, 20=0.39%, 50=99.38%, 100=0.20%
  cpu          : usr=0.52%, sys=6.57%, ctx=2895, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=188964: Thu Jun 25 01:02:44 2026
  write: IOPS=479, BW=1919KiB/s (1965kB/s)(10.0MiB/5335msec); 0 zone resets
    slat (usec): min=43, max=811, avg=175.40, stdev=89.00
    clat (usec): min=6720, max=66140, avg=33109.47, stdev=4163.69
     lat (usec): min=6807, max=66257, avg=33286.00, stdev=4172.03
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[27657], 10.00th=[28967], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[32637], 60.00th=[33424],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[44303], 99.90th=[62653], 99.95th=[62653],
     | 99.99th=[66323]
   bw (  KiB/s): min= 1816, max= 2088, per=25.25%, avg=1924.50, stdev=80.92, samples=10
   iops        : min=  454, max=  522, avg=481.10, stdev=20.25, samples=10
  lat (msec)   : 10=0.20%, 20=0.08%, 50=99.53%, 100=0.20%
  cpu          : usr=0.62%, sys=6.37%, ctx=2896, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=188965: Thu Jun 25 01:02:44 2026
  write: IOPS=479, BW=1918KiB/s (1964kB/s)(10.0MiB/5339msec); 0 zone resets
    slat (usec): min=36, max=1422, avg=172.83, stdev=86.31
    clat (usec): min=6995, max=62597, avg=33064.99, stdev=4059.28
     lat (usec): min=7151, max=62781, avg=33238.90, stdev=4068.20
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[27657], 10.00th=[28967], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[31851], 50.00th=[32637], 60.00th=[33424],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[43779], 99.90th=[62129], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1800, max= 2072, per=25.24%, avg=1923.80, stdev=80.88, samples=10
   iops        : min=  450, max=  518, avg=480.90, stdev=20.26, samples=10
  lat (msec)   : 10=0.12%, 20=0.08%, 50=99.57%, 100=0.23%
  cpu          : usr=0.56%, sys=6.44%, ctx=2764, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=188966: Thu Jun 25 01:02:44 2026
  write: IOPS=476, BW=1905KiB/s (1951kB/s)(10.0MiB/5374msec); 0 zone resets
    slat (usec): min=45, max=640, avg=151.03, stdev=79.58
    clat (usec): min=18528, max=43747, avg=33121.53, stdev=3647.52
     lat (usec): min=18658, max=43989, avg=33273.72, stdev=3653.84
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[28181], 10.00th=[29230], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[31851], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[43254], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1754, max= 2088, per=24.98%, avg=1903.40, stdev=101.74, samples=10
   iops        : min=  438, max=  522, avg=475.80, stdev=25.52, samples=10
  lat (msec)   : 20=0.08%, 50=99.92%
  cpu          : usr=0.50%, sys=5.81%, ctx=2691, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7622KiB/s (7805kB/s), 1905KiB/s-1919KiB/s (1951kB/s-1965kB/s), io=40.0MiB (41.9MB), run=5335-5374msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=188976: Thu Jun 25 01:02:55 2026
  read: IOPS=168, BW=674KiB/s (690kB/s)(5048KiB/7490msec)
    slat (usec): min=923, max=16334, avg=3313.80, stdev=1207.98
    clat (usec): min=17300, max=69280, avg=44391.85, stdev=8071.08
     lat (usec): min=19970, max=73664, avg=47708.48, stdev=8403.90
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[30016], 10.00th=[33162], 20.00th=[36963],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[44827], 60.00th=[46924],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56361],
     | 99.00th=[62129], 99.50th=[64750], 99.90th=[68682], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  568, max=  893, per=25.04%, avg=670.64, stdev=75.98, samples=14
   iops        : min=  142, max=  223, avg=167.64, stdev=18.94, samples=14
  write: IOPS=173, BW=693KiB/s (710kB/s)(5192KiB/7490msec); 0 zone resets
    slat (usec): min=50, max=943, avg=172.15, stdev=102.94
    clat (usec): min=24179, max=67660, avg=45181.26, stdev=7690.21
     lat (usec): min=24412, max=67839, avg=45354.49, stdev=7698.19
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30540], 10.00th=[33817], 20.00th=[39060],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[51119], 90.00th=[54264], 95.00th=[57410],
     | 99.00th=[62129], 99.50th=[63701], 99.90th=[66323], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  536, max=  920, per=24.86%, avg=693.07, stdev=103.79, samples=14
   iops        : min=  134, max=  230, avg=173.21, stdev=25.86, samples=14
  lat (msec)   : 20=0.08%, 50=74.84%, 100=25.08%
  cpu          : usr=0.43%, sys=5.27%, ctx=4468, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=188977: Thu Jun 25 01:02:55 2026
  read: IOPS=166, BW=664KiB/s (680kB/s)(4976KiB/7491msec)
    slat (usec): min=1083, max=8887, avg=3316.71, stdev=1163.66
    clat (usec): min=20592, max=79715, avg=44826.90, stdev=8490.64
     lat (usec): min=22208, max=82347, avg=48145.56, stdev=8811.96
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[29754], 10.00th=[33424], 20.00th=[38011],
     | 30.00th=[40633], 40.00th=[43254], 50.00th=[44827], 60.00th=[46924],
     | 70.00th=[49021], 80.00th=[51643], 90.00th=[55313], 95.00th=[57934],
     | 99.00th=[64226], 99.50th=[69731], 99.90th=[73925], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  552, max=  805, per=24.62%, avg=659.21, stdev=75.56, samples=14
   iops        : min=  138, max=  201, avg=164.79, stdev=18.85, samples=14
  write: IOPS=175, BW=703KiB/s (720kB/s)(5264KiB/7491msec); 0 zone resets
    slat (usec): min=39, max=1027, avg=173.15, stdev=107.77
    clat (usec): min=18655, max=74657, avg=44971.53, stdev=7743.36
     lat (usec): min=18832, max=75338, avg=45146.05, stdev=7753.66
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[30278], 10.00th=[32900], 20.00th=[39060],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56361],
     | 99.00th=[62129], 99.50th=[64226], 99.90th=[67634], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  560, max=  985, per=25.19%, avg=702.36, stdev=110.52, samples=14
   iops        : min=  140, max=  246, avg=175.57, stdev=27.58, samples=14
  lat (msec)   : 20=0.04%, 50=74.22%, 100=25.74%
  cpu          : usr=0.41%, sys=5.26%, ctx=4369, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=188978: Thu Jun 25 01:02:55 2026
  read: IOPS=166, BW=667KiB/s (683kB/s)(4984KiB/7473msec)
    slat (usec): min=1001, max=17559, avg=3308.79, stdev=1241.20
    clat (usec): min=10746, max=72840, avg=44349.70, stdev=8287.39
     lat (usec): min=13754, max=76634, avg=47660.27, stdev=8609.38
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[29230], 10.00th=[32375], 20.00th=[38011],
     | 30.00th=[40633], 40.00th=[42730], 50.00th=[44827], 60.00th=[46924],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54264], 95.00th=[57410],
     | 99.00th=[61604], 99.50th=[64226], 99.90th=[70779], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  512, max=  926, per=24.96%, avg=668.43, stdev=105.51, samples=14
   iops        : min=  128, max=  231, avg=167.07, stdev=26.28, samples=14
  write: IOPS=175, BW=703KiB/s (720kB/s)(5256KiB/7473msec); 0 zone resets
    slat (usec): min=49, max=1041, avg=176.59, stdev=100.75
    clat (usec): min=21604, max=88819, avg=45210.66, stdev=7979.38
     lat (usec): min=21870, max=88924, avg=45388.32, stdev=7985.38
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30278], 10.00th=[33817], 20.00th=[39060],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56886],
     | 99.00th=[63177], 99.50th=[66847], 99.90th=[88605], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  568, max=  900, per=25.10%, avg=699.71, stdev=94.46, samples=14
   iops        : min=  142, max=  225, avg=174.93, stdev=23.61, samples=14
  lat (msec)   : 20=0.08%, 50=74.84%, 100=25.08%
  cpu          : usr=0.33%, sys=5.42%, ctx=4430, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=188979: Thu Jun 25 01:02:55 2026
  read: IOPS=169, BW=678KiB/s (695kB/s)(5060KiB/7459msec)
    slat (usec): min=53, max=17786, avg=3279.96, stdev=1252.22
    clat (usec): min=4232, max=77344, avg=44377.03, stdev=8689.39
     lat (usec): min=4396, max=82590, avg=47658.73, stdev=9079.68
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[29754], 10.00th=[32637], 20.00th=[37487],
     | 30.00th=[40633], 40.00th=[42730], 50.00th=[44827], 60.00th=[46924],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54789], 95.00th=[57410],
     | 99.00th=[63177], 99.50th=[66323], 99.90th=[72877], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  528, max=  959, per=25.22%, avg=675.36, stdev=94.98, samples=14
   iops        : min=  132, max=  239, avg=168.79, stdev=23.57, samples=14
  write: IOPS=173, BW=694KiB/s (711kB/s)(5180KiB/7459msec); 0 zone resets
    slat (usec): min=43, max=934, avg=174.48, stdev=104.86
    clat (usec): min=12801, max=91419, avg=45276.30, stdev=8075.20
     lat (usec): min=12978, max=91553, avg=45451.81, stdev=8087.03
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[31065], 10.00th=[32900], 20.00th=[39584],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[46400], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54264], 95.00th=[56886],
     | 99.00th=[62129], 99.50th=[64226], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
   bw (  KiB/s): min=  544, max=  966, per=24.90%, avg=694.14, stdev=111.41, samples=14
   iops        : min=  136, max=  241, avg=173.50, stdev=27.76, samples=14
  lat (msec)   : 10=0.20%, 20=0.23%, 50=74.18%, 100=25.39%
  cpu          : usr=0.46%, sys=5.26%, ctx=4412, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2679KiB/s (2743kB/s), 664KiB/s-678KiB/s (680kB/s-695kB/s), io=19.6MiB (20.5MB), run=7459-7491msec
  WRITE: bw=2789KiB/s (2856kB/s), 693KiB/s-703KiB/s (710kB/s-720kB/s), io=20.4MiB (21.4MB), run=7459-7491msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=188984: Thu Jun 25 01:02:59 2026
  read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=11075k, max=11075k, avg=11074532.00, stdev= 0.00
    clat (nsec): min=92780k, max=92780k, avg=92779682.00, stdev= 0.00
     lat (nsec): min=103859k, max=103859k, avg=103859199.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[92799],  5.00th=[92799], 10.00th=[92799], 20.00th=[92799],
     | 30.00th=[92799], 40.00th=[92799], 50.00th=[92799], 60.00th=[92799],
     | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799],
     | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799],
     | 99.99th=[92799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.17%, ctx=13, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout