== sanity test 398c: run fio to test AIO ================= 19:44:51 (1775864691)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.131661 s, 319 MB/s
osc.lustre-OST0000-osc-ffff9555c9014000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190651: Fri Apr 10 19:45:00 2026
  write: IOPS=474, BW=1896KiB/s (1942kB/s)(10.0MiB/5400msec); 0 zone resets
    slat (usec): min=48, max=785, avg=197.13, stdev=107.67
    clat (usec): min=6847, max=43360, avg=33218.13, stdev=3200.13
     lat (usec): min=7376, max=43481, avg=33416.58, stdev=3213.74
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27919], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[39584], 99.50th=[41681], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1816, max= 2016, per=24.92%, avg=1890.40, stdev=70.36, samples=10
   iops        : min=  454, max=  504, avg=472.60, stdev=17.59, samples=10
  lat (msec)   : 10=0.04%, 20=0.12%, 50=99.84%
  cpu          : usr=0.56%, sys=6.76%, ctx=2769, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190652: Fri Apr 10 19:45:00 2026
  write: IOPS=477, BW=1908KiB/s (1954kB/s)(10.0MiB/5366msec); 0 zone resets
    slat (usec): min=41, max=771, avg=192.66, stdev=102.36
    clat (usec): min=3923, max=65714, avg=33292.04, stdev=3909.91
     lat (usec): min=4134, max=65868, avg=33486.03, stdev=3922.93
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27919], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[44827], 99.90th=[65274], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1768, max= 2173, per=25.16%, avg=1908.50, stdev=110.58, samples=10
   iops        : min=  442, max=  543, avg=477.10, stdev=27.58, samples=10
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.66%, 50=98.83%, 100=0.43%
  cpu          : usr=0.62%, sys=6.82%, ctx=2654, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190653: Fri Apr 10 19:45:00 2026
  write: IOPS=475, BW=1902KiB/s (1947kB/s)(10.0MiB/5385msec); 0 zone resets
    slat (usec): min=42, max=26493, avg=215.03, stdev=529.66
    clat (usec): min=6077, max=67011, avg=33255.89, stdev=3745.27
     lat (usec): min=6233, max=67135, avg=33472.25, stdev=3729.76
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[27657], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[41681], 99.50th=[43254], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[66847]
   bw (  KiB/s): min= 1808, max= 2115, per=25.09%, avg=1902.70, stdev=93.73, samples=10
   iops        : min=  452, max=  528, avg=475.60, stdev=23.24, samples=10
  lat (msec)   : 10=0.04%, 20=0.43%, 50=99.14%, 100=0.39%
  cpu          : usr=0.61%, sys=7.54%, ctx=2741, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190654: Fri Apr 10 19:45:00 2026
  write: IOPS=474, BW=1897KiB/s (1943kB/s)(10.0MiB/5398msec); 0 zone resets
    slat (usec): min=46, max=755, avg=191.40, stdev=97.73
    clat (usec): min=6272, max=58779, avg=33231.27, stdev=3511.56
     lat (usec): min=6411, max=58951, avg=33423.96, stdev=3519.11
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27657], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[39584], 99.50th=[42730], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1792, max= 2057, per=24.96%, avg=1892.90, stdev=82.43, samples=10
   iops        : min=  448, max=  514, avg=473.20, stdev=20.55, samples=10
  lat (msec)   : 10=0.04%, 20=0.70%, 50=99.02%, 100=0.23%
  cpu          : usr=0.61%, sys=7.32%, ctx=2607, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7585KiB/s (7767kB/s), 1896KiB/s-1908KiB/s (1942kB/s-1954kB/s), io=40.0MiB (41.9MB), run=5366-5400msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190665: Fri Apr 10 19:45:09 2026
  read: IOPS=213, BW=854KiB/s (875kB/s)(5048KiB/5909msec)
    slat (usec): min=1136, max=5312, avg=2445.39, stdev=656.81
    clat (usec): min=7322, max=50805, avg=35231.71, stdev=4766.02
     lat (usec): min=9450, max=52941, avg=37678.65, stdev=4798.76
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[27395], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41157], 95.00th=[42730],
     | 99.00th=[45351], 99.50th=[47449], 99.90th=[50070], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  744, max=  936, per=24.97%, avg=848.00, stdev=63.80, samples=11
   iops        : min=  186, max=  234, avg=212.00, stdev=15.95, samples=11
  write: IOPS=219, BW=879KiB/s (900kB/s)(5192KiB/5909msec); 0 zone resets
    slat (usec): min=63, max=1130, avg=174.60, stdev=86.36
    clat (usec): min=9296, max=47660, avg=35410.07, stdev=3989.04
     lat (usec): min=9421, max=47829, avg=35585.74, stdev=3989.10
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[29492], 10.00th=[30802], 20.00th=[32375],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[46924], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min=  759, max=  992, per=24.71%, avg=873.36, stdev=73.81, samples=11
   iops        : min=  189, max=  248, avg=218.27, stdev=18.57, samples=11
  lat (msec)   : 10=0.08%, 20=0.27%, 50=99.61%, 100=0.04%
  cpu          : usr=0.49%, sys=6.52%, ctx=4924, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190666: Fri Apr 10 19:45:09 2026
  read: IOPS=211, BW=845KiB/s (865kB/s)(4976KiB/5892msec)
    slat (usec): min=1033, max=8843, avg=2450.04, stdev=660.02
    clat (usec): min=8914, max=62102, avg=35147.07, stdev=4592.34
     lat (usec): min=11874, max=63988, avg=37598.67, stdev=4600.19
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27657], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[46400], 99.50th=[46924], 99.90th=[53216], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  728, max=  920, per=24.79%, avg=841.82, stdev=65.71, samples=11
   iops        : min=  182, max=  230, avg=210.45, stdev=16.43, samples=11
  write: IOPS=223, BW=893KiB/s (915kB/s)(5264KiB/5892msec); 0 zone resets
    slat (usec): min=41, max=692, avg=176.58, stdev=91.33
    clat (usec): min=22473, max=57055, avg=35589.89, stdev=3651.44
     lat (usec): min=22578, max=57414, avg=35767.57, stdev=3651.10
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[30016], 10.00th=[31065], 20.00th=[32900],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[51643], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  832, max=  976, per=25.32%, avg=895.00, stdev=48.99, samples=11
   iops        : min=  208, max=  244, avg=223.73, stdev=12.25, samples=11
  lat (msec)   : 10=0.04%, 20=0.04%, 50=99.73%, 100=0.20%
  cpu          : usr=0.51%, sys=6.57%, ctx=4814, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190667: Fri Apr 10 19:45:09 2026
  read: IOPS=212, BW=849KiB/s (869kB/s)(4984KiB/5872msec)
    slat (usec): min=60, max=5288, avg=2457.78, stdev=661.11
    clat (usec): min=4855, max=57247, avg=35276.96, stdev=5059.50
     lat (usec): min=4927, max=59995, avg=37736.26, stdev=5123.43
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27657], 10.00th=[29492], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41157], 95.00th=[42730],
     | 99.00th=[45351], 99.50th=[45876], 99.90th=[51643], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  784, max=  928, per=25.04%, avg=850.45, stdev=51.88, samples=11
   iops        : min=  196, max=  232, avg=212.55, stdev=12.90, samples=11
  write: IOPS=223, BW=895KiB/s (917kB/s)(5256KiB/5872msec); 0 zone resets
    slat (usec): min=51, max=2100, avg=173.61, stdev=109.91
    clat (usec): min=19136, max=56762, avg=35413.08, stdev=3860.31
     lat (usec): min=19387, max=56959, avg=35587.77, stdev=3866.46
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29492], 10.00th=[30802], 20.00th=[32637],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[44303], 99.50th=[46924], 99.90th=[55837], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  784, max= 1064, per=25.14%, avg=888.73, stdev=76.52, samples=11
   iops        : min=  196, max=  266, avg=222.18, stdev=19.13, samples=11
  lat (msec)   : 10=0.16%, 20=0.47%, 50=99.14%, 100=0.23%
  cpu          : usr=0.63%, sys=6.32%, ctx=4860, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190668: Fri Apr 10 19:45:09 2026
  read: IOPS=214, BW=857KiB/s (877kB/s)(5060KiB/5907msec)
    slat (usec): min=1108, max=5674, avg=2438.23, stdev=664.33
    clat (usec): min=5725, max=51578, avg=35192.83, stdev=4565.29
     lat (usec): min=8367, max=54076, avg=37632.53, stdev=4605.29
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27395], 10.00th=[29492], 20.00th=[31589],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[44827], 99.50th=[46924], 99.90th=[50070], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min=  797, max=  896, per=25.05%, avg=850.64, stdev=26.47, samples=11
   iops        : min=  199, max=  224, avg=212.64, stdev= 6.67, samples=11
  write: IOPS=219, BW=877KiB/s (898kB/s)(5180KiB/5907msec); 0 zone resets
    slat (usec): min=59, max=690, avg=168.89, stdev=85.13
    clat (usec): min=22392, max=50683, avg=35575.38, stdev=3536.89
     lat (usec): min=22501, max=50901, avg=35745.42, stdev=3538.32
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30016], 10.00th=[31065], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41157],
     | 99.00th=[44303], 99.50th=[46924], 99.90th=[48497], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min=  784, max=  952, per=24.78%, avg=875.82, stdev=48.83, samples=11
   iops        : min=  196, max=  238, avg=218.91, stdev=12.28, samples=11
  lat (msec)   : 10=0.08%, 50=99.80%, 100=0.12%
  cpu          : usr=0.44%, sys=6.48%, ctx=4802, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3396KiB/s (3478kB/s), 845KiB/s-857KiB/s (865kB/s-877kB/s), io=19.6MiB (20.5MB), run=5872-5909msec
  WRITE: bw=3536KiB/s (3620kB/s), 877KiB/s-895KiB/s (898kB/s-917kB/s), io=20.4MiB (21.4MB), run=5872-5909msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190673: Fri Apr 10 19:45:13 2026
  read: IOPS=7, BW=308MiB/s (323MB/s)(40.0MiB/130msec)
    slat (nsec): min=13744k, max=13744k, avg=13743894.00, stdev= 0.00
    clat (nsec): min=111325k, max=111325k, avg=111324704.00, stdev= 0.00
     lat (nsec): min=125072k, max=125072k, avg=125071882.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  112],  5.00th=[  112], 10.00th=[  112], 20.00th=[  112],
     | 30.00th=[  112], 40.00th=[  112], 50.00th=[  112], 60.00th=[  112],
     | 70.00th=[  112], 80.00th=[  112], 90.00th=[  112], 95.00th=[  112],
     | 99.00th=[  112], 99.50th=[  112], 99.90th=[  112], 99.95th=[  112],
     | 99.99th=[  112]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=9.30%, ctx=10, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=40.0MiB (41.9MB), run=130-130msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout