== sanity test 398c: run fio to test AIO ================= 11:20:52 (1755098452)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.128634 s, 326 MB/s
osc.lustre-OST0000-osc-ffff8e75d9915800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190044: Wed Aug 13 11:21:00 2025
  write: IOPS=511, BW=2047KiB/s (2096kB/s)(10.0MiB/5003msec); 0 zone resets
    slat (usec): min=27, max=27364, avg=98.42, stdev=540.98
    clat (usec): min=14861, max=55773, avg=30780.65, stdev=3440.20
     lat (usec): min=14934, max=55862, avg=30880.07, stdev=3451.65
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25297], 10.00th=[26346], 20.00th=[28181],
     | 30.00th=[29230], 40.00th=[30016], 50.00th=[30802], 60.00th=[31589],
     | 70.00th=[32375], 80.00th=[33424], 90.00th=[34866], 95.00th=[36439],
     | 99.00th=[40109], 99.50th=[42206], 99.90th=[52691], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min= 1649, max= 2192, per=25.11%, avg=2053.44, stdev=177.43, samples=9
   iops        : min=  412, max=  548, avg=513.33, stdev=44.43, samples=9
  lat (msec)   : 20=0.08%, 50=99.80%, 100=0.12%
  cpu          : usr=0.54%, sys=4.94%, ctx=1356, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190045: Wed Aug 13 11:21:00 2025
  write: IOPS=511, BW=2044KiB/s (2093kB/s)(10.0MiB/5009msec); 0 zone resets
    slat (usec): min=27, max=372, avg=87.01, stdev=38.51
    clat (usec): min=4144, max=61256, avg=30732.49, stdev=3727.45
     lat (usec): min=4216, max=61303, avg=30820.58, stdev=3724.94
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25035], 10.00th=[26346], 20.00th=[27919],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31327],
     | 70.00th=[32113], 80.00th=[33424], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[39060], 99.50th=[42730], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min= 1630, max= 2200, per=25.09%, avg=2051.33, stdev=194.04, samples=9
   iops        : min=  407, max=  550, avg=512.78, stdev=48.65, samples=9
  lat (msec)   : 10=0.04%, 50=99.61%, 100=0.35%
  cpu          : usr=0.72%, sys=4.17%, ctx=1236, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190046: Wed Aug 13 11:21:00 2025
  write: IOPS=516, BW=2067KiB/s (2117kB/s)(10.0MiB/4953msec); 0 zone resets
    slat (usec): min=30, max=440, avg=88.80, stdev=39.35
    clat (usec): min=7988, max=65494, avg=30806.57, stdev=4059.03
     lat (usec): min=8084, max=65531, avg=30896.29, stdev=4057.71
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25035], 10.00th=[26346], 20.00th=[27919],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31327],
     | 70.00th=[32375], 80.00th=[33424], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[43254], 99.50th=[53740], 99.90th=[65274], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min= 1738, max= 2216, per=25.32%, avg=2070.44, stdev=154.11, samples=9
   iops        : min=  434, max=  554, avg=517.56, stdev=38.66, samples=9
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.14%, 100=0.74%
  cpu          : usr=0.48%, sys=4.66%, ctx=1457, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190047: Wed Aug 13 11:21:00 2025
  write: IOPS=511, BW=2046KiB/s (2095kB/s)(10.0MiB/5006msec); 0 zone resets
    slat (usec): min=28, max=396, avg=86.05, stdev=37.57
    clat (usec): min=18656, max=53068, avg=30763.88, stdev=3566.59
     lat (usec): min=18738, max=53145, avg=30850.86, stdev=3565.25
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[25297], 10.00th=[26608], 20.00th=[27919],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30802], 60.00th=[31327],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[41157], 99.50th=[41157], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1630, max= 2224, per=25.08%, avg=2050.44, stdev=196.78, samples=9
   iops        : min=  407, max=  556, avg=512.56, stdev=49.33, samples=9
  lat (msec)   : 20=0.04%, 50=99.73%, 100=0.23%
  cpu          : usr=0.64%, sys=4.66%, ctx=1228, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8177KiB/s (8374kB/s), 2044KiB/s-2067KiB/s (2093kB/s-2117kB/s), io=40.0MiB (41.9MB), run=4953-5009msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190057: Wed Aug 13 11:21:09 2025
  read: IOPS=209, BW=839KiB/s (859kB/s)(5048KiB/6016msec)
    slat (usec): min=1075, max=17983, avg=2814.27, stdev=1057.12
    clat (usec): min=12965, max=61145, avg=35628.15, stdev=6462.75
     lat (usec): min=14817, max=67555, avg=38443.74, stdev=6798.01
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26608], 10.00th=[28443], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[40633], 90.00th=[44827], 95.00th=[48497],
     | 99.00th=[53216], 99.50th=[55837], 99.90th=[60556], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  696, max=  968, per=24.75%, avg=825.45, stdev=100.61, samples=11
   iops        : min=  174, max=  242, avg=206.36, stdev=25.15, samples=11
  write: IOPS=215, BW=863KiB/s (884kB/s)(5192KiB/6016msec); 0 zone resets
    slat (usec): min=30, max=2005, avg=85.56, stdev=65.89
    clat (usec): min=11725, max=60637, avg=36098.21, stdev=6319.56
     lat (usec): min=11817, max=60708, avg=36184.73, stdev=6319.60
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28443], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[44827], 95.00th=[48497],
     | 99.00th=[55837], 99.50th=[56886], 99.90th=[60556], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  696, max= 1056, per=24.82%, avg=861.64, stdev=124.71, samples=11
   iops        : min=  174, max=  264, avg=215.36, stdev=31.20, samples=11
  lat (msec)   : 20=0.27%, 50=96.29%, 100=3.44%
  cpu          : usr=0.65%, sys=4.92%, ctx=2438, majf=0, minf=1275
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190058: Wed Aug 13 11:21:09 2025
  read: IOPS=207, BW=829KiB/s (849kB/s)(4976KiB/6005msec)
    slat (usec): min=1014, max=12609, avg=2855.86, stdev=1035.40
    clat (usec): min=10082, max=61512, avg=35847.99, stdev=6863.18
     lat (usec): min=12121, max=64307, avg=38705.38, stdev=7183.62
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26346], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38536], 80.00th=[41681], 90.00th=[45876], 95.00th=[48497],
     | 99.00th=[54789], 99.50th=[57934], 99.90th=[61080], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  696, max=  912, per=24.53%, avg=818.00, stdev=87.32, samples=11
   iops        : min=  174, max=  228, avg=204.45, stdev=21.84, samples=11
  write: IOPS=219, BW=877KiB/s (898kB/s)(5264KiB/6005msec); 0 zone resets
    slat (usec): min=28, max=582, avg=85.98, stdev=39.08
    clat (usec): min=16679, max=56426, avg=35941.03, stdev=5916.54
     lat (usec): min=16771, max=56513, avg=36027.98, stdev=5917.59
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28181], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40633], 90.00th=[44827], 95.00th=[47449],
     | 99.00th=[52167], 99.50th=[54789], 99.90th=[56361], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  712, max= 1016, per=25.29%, avg=878.18, stdev=116.61, samples=11
   iops        : min=  178, max=  254, avg=219.55, stdev=29.15, samples=11
  lat (msec)   : 20=0.20%, 50=97.07%, 100=2.73%
  cpu          : usr=0.50%, sys=5.10%, ctx=2424, majf=0, minf=1257
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190059: Wed Aug 13 11:21:09 2025
  read: IOPS=207, BW=829KiB/s (849kB/s)(4984KiB/6009msec)
    slat (usec): min=64, max=12618, avg=2837.59, stdev=1106.39
    clat (usec): min=4873, max=61614, avg=36077.06, stdev=6900.66
     lat (usec): min=4955, max=66723, avg=38916.11, stdev=7292.40
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26346], 10.00th=[28443], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38536], 80.00th=[41157], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[53740], 99.50th=[56361], 99.90th=[60031], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  608, max=  960, per=24.78%, avg=826.45, stdev=112.98, samples=11
   iops        : min=  152, max=  240, avg=206.55, stdev=28.17, samples=11
  write: IOPS=218, BW=875KiB/s (896kB/s)(5256KiB/6009msec); 0 zone resets
    slat (usec): min=29, max=1233, avg=83.79, stdev=47.24
    clat (usec): min=6460, max=59631, avg=36043.42, stdev=6304.00
     lat (usec): min=6539, max=59707, avg=36128.14, stdev=6303.60
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28181], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[41681], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[52691], 99.50th=[55313], 99.90th=[58983], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  608, max= 1056, per=24.90%, avg=864.55, stdev=136.54, samples=11
   iops        : min=  152, max=  264, avg=216.09, stdev=34.12, samples=11
  lat (msec)   : 10=0.16%, 20=0.20%, 50=96.25%, 100=3.40%
  cpu          : usr=0.40%, sys=5.06%, ctx=2315, majf=0, minf=1261
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190060: Wed Aug 13 11:21:09 2025
  read: IOPS=210, BW=842KiB/s (862kB/s)(5060KiB/6012msec)
    slat (usec): min=1047, max=17809, avg=2826.87, stdev=1050.92
    clat (usec): min=3802, max=63330, avg=35518.66, stdev=6875.24
     lat (usec): min=6602, max=71242, avg=38346.89, stdev=7243.05
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26346], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[40633], 90.00th=[45351], 95.00th=[49021],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[62129], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  704, max= 1000, per=25.02%, avg=834.45, stdev=94.01, samples=11
   iops        : min=  176, max=  250, avg=208.55, stdev=23.50, samples=11
  write: IOPS=215, BW=862KiB/s (882kB/s)(5180KiB/6012msec); 0 zone resets
    slat (usec): min=31, max=437, avg=83.30, stdev=35.47
    clat (usec): min=24018, max=60147, avg=36106.70, stdev=6076.20
     lat (usec): min=24078, max=60202, avg=36190.94, stdev=6076.96
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28443], 10.00th=[29754], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[40633], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[53740], 99.50th=[56886], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  664, max= 1008, per=24.63%, avg=855.00, stdev=127.91, samples=11
   iops        : min=  166, max=  252, avg=213.73, stdev=31.98, samples=11
  lat (msec)   : 4=0.04%, 10=0.08%, 50=96.48%, 100=3.40%
  cpu          : usr=0.63%, sys=4.89%, ctx=2397, majf=0, minf=1279
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3336KiB/s (3416kB/s), 829KiB/s-842KiB/s (849kB/s-862kB/s), io=19.6MiB (20.5MB), run=6005-6016msec
  WRITE: bw=3473KiB/s (3556kB/s), 862KiB/s-877KiB/s (882kB/s-898kB/s), io=20.4MiB (21.4MB), run=6005-6016msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190064: Wed Aug 13 11:21:13 2025
  read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec)
    slat (nsec): min=10223k, max=10223k, avg=10223411.00, stdev= 0.00
    clat (nsec): min=90660k, max=90660k, avg=90659550.00, stdev= 0.00
     lat (nsec): min=100887k, max=100887k, avg=100886862.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.95%, sys=9.52%, ctx=2, majf=0, minf=10250
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout