== sanity test 398c: run fio to test AIO ================= 22:57:24 (1772423844)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.137025 s, 306 MB/s
osc.lustre-OST0000-osc-ffff9a6a02c73000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182426: Sun Mar  1 22:57:33 2026
  write: IOPS=493, BW=1975KiB/s (2023kB/s)(10.0MiB/5184msec); 0 zone resets
    slat (usec): min=38, max=3096, avg=175.63, stdev=116.05
    clat (usec): min=15317, max=45015, avg=31855.23, stdev=3820.18
     lat (usec): min=15420, max=45162, avg=32032.05, stdev=3836.60
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[25822], 10.00th=[27132], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[40633], 99.50th=[41681], 99.90th=[42206], 99.95th=[42206],
     | 99.99th=[44827]
   bw (  KiB/s): min= 1904, max= 2151, per=24.90%, avg=1967.10, stdev=72.30, samples=10
   iops        : min=  476, max=  537, avg=491.70, stdev=17.86, samples=10
  lat (msec)   : 20=0.86%, 50=99.14%
  cpu          : usr=0.58%, sys=6.42%, ctx=2396, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182427: Sun Mar  1 22:57:33 2026
  write: IOPS=497, BW=1990KiB/s (2038kB/s)(10.0MiB/5145msec); 0 zone resets
    slat (usec): min=40, max=705, avg=188.11, stdev=94.17
    clat (usec): min=6679, max=66719, avg=31915.32, stdev=4206.00
     lat (usec): min=7164, max=67052, avg=32104.68, stdev=4205.61
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25297], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[44303], 99.90th=[59507], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min= 1856, max= 2287, per=25.08%, avg=1981.50, stdev=117.53, samples=10
   iops        : min=  464, max=  571, avg=495.30, stdev=29.17, samples=10
  lat (msec)   : 10=0.08%, 20=0.27%, 50=99.38%, 100=0.27%
  cpu          : usr=0.52%, sys=7.04%, ctx=2608, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182428: Sun Mar  1 22:57:33 2026
  write: IOPS=494, BW=1980KiB/s (2027kB/s)(10.0MiB/5172msec); 0 zone resets
    slat (usec): min=47, max=1094, avg=189.85, stdev=101.76
    clat (usec): min=7385, max=54545, avg=31793.05, stdev=4035.28
     lat (usec): min=7517, max=54672, avg=31984.12, stdev=4038.18
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[25297], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[41157], 99.50th=[41681], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1888, max= 2171, per=24.96%, avg=1972.30, stdev=81.54, samples=10
   iops        : min=  472, max=  542, avg=493.00, stdev=20.18, samples=10
  lat (msec)   : 10=0.16%, 20=0.66%, 50=99.02%, 100=0.16%
  cpu          : usr=0.33%, sys=7.68%, ctx=2668, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182429: Sun Mar  1 22:57:33 2026
  write: IOPS=495, BW=1981KiB/s (2029kB/s)(10.0MiB/5169msec); 0 zone resets
    slat (usec): min=48, max=30043, avg=202.64, stdev=598.54
    clat (usec): min=7215, max=54163, avg=31809.41, stdev=3997.01
     lat (usec): min=8752, max=54364, avg=32013.15, stdev=3969.49
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[25035], 10.00th=[26608], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[34341], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[41157], 99.50th=[41681], 99.90th=[44303], 99.95th=[47449],
     | 99.99th=[54264]
   bw (  KiB/s): min= 1896, max= 2216, per=25.01%, avg=1976.00, stdev=93.22, samples=10
   iops        : min=  474, max=  554, avg=494.00, stdev=23.30, samples=10
  lat (msec)   : 10=0.08%, 20=0.35%, 50=99.53%, 100=0.04%
  cpu          : usr=0.50%, sys=7.57%, ctx=2720, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7901KiB/s (8091kB/s), 1975KiB/s-1990KiB/s (2023kB/s-2038kB/s), io=40.0MiB (41.9MB), run=5145-5184msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182439: Sun Mar  1 22:57:42 2026
  read: IOPS=208, BW=836KiB/s (856kB/s)(5048KiB/6040msec)
    slat (usec): min=1073, max=7055, avg=2553.70, stdev=808.36
    clat (usec): min=13437, max=63146, avg=35820.25, stdev=6140.84
     lat (usec): min=15167, max=65523, avg=38375.35, stdev=6297.50
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26870], 10.00th=[28443], 20.00th=[30802],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[53740], 99.50th=[56361], 99.90th=[61604], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  728, max= 1016, per=24.81%, avg=824.18, stdev=78.73, samples=11
   iops        : min=  182, max=  254, avg=206.00, stdev=19.70, samples=11
  write: IOPS=214, BW=860KiB/s (880kB/s)(5192KiB/6040msec); 0 zone resets
    slat (usec): min=54, max=895, avg=162.55, stdev=86.58
    clat (usec): min=14533, max=58754, avg=36363.06, stdev=5194.96
     lat (usec): min=14611, max=58892, avg=36526.69, stdev=5202.28
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29230], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[43254], 95.00th=[46400],
     | 99.00th=[51643], 99.50th=[52691], 99.90th=[56886], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  736, max=  968, per=24.78%, avg=856.91, stdev=74.34, samples=11
   iops        : min=  184, max=  242, avg=214.18, stdev=18.66, samples=11
  lat (msec)   : 20=0.31%, 50=97.27%, 100=2.42%
  cpu          : usr=0.43%, sys=6.19%, ctx=4789, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182440: Sun Mar  1 22:57:42 2026
  read: IOPS=205, BW=824KiB/s (844kB/s)(4976KiB/6040msec)
    slat (usec): min=898, max=10248, avg=2519.62, stdev=817.62
    clat (usec): min=6173, max=59142, avg=35886.60, stdev=5980.59
     lat (usec): min=9103, max=62670, avg=38407.70, stdev=6129.84
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26608], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[42730], 95.00th=[45351],
     | 99.00th=[52167], 99.50th=[55313], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  707, max=  912, per=24.37%, avg=809.73, stdev=60.29, samples=11
   iops        : min=  176, max=  228, avg=202.36, stdev=15.20, samples=11
  write: IOPS=217, BW=872KiB/s (892kB/s)(5264KiB/6040msec); 0 zone resets
    slat (usec): min=55, max=1775, avg=163.14, stdev=96.86
    clat (usec): min=14994, max=61334, avg=36398.84, stdev=5276.19
     lat (usec): min=15101, max=61446, avg=36563.01, stdev=5281.68
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[42730], 95.00th=[45876],
     | 99.00th=[51643], 99.50th=[54264], 99.90th=[59507], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  712, max= 1008, per=25.20%, avg=871.45, stdev=80.90, samples=11
   iops        : min=  178, max=  252, avg=217.82, stdev=20.25, samples=11
  lat (msec)   : 10=0.04%, 20=0.20%, 50=98.20%, 100=1.56%
  cpu          : usr=0.40%, sys=6.16%, ctx=4730, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182441: Sun Mar  1 22:57:42 2026
  read: IOPS=207, BW=829KiB/s (849kB/s)(4984KiB/6012msec)
    slat (usec): min=67, max=10573, avg=2557.04, stdev=830.46
    clat (usec): min=6870, max=59764, avg=36099.82, stdev=6146.92
     lat (usec): min=7011, max=62708, avg=38658.27, stdev=6298.43
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28443], 20.00th=[30802],
     | 30.00th=[32637], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[43779], 95.00th=[46400],
     | 99.00th=[52691], 99.50th=[53216], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  712, max= 1008, per=25.00%, avg=830.55, stdev=82.89, samples=11
   iops        : min=  178, max=  252, avg=207.64, stdev=20.72, samples=11
  write: IOPS=218, BW=874KiB/s (895kB/s)(5256KiB/6012msec); 0 zone resets
    slat (usec): min=49, max=2111, avg=168.92, stdev=111.28
    clat (usec): min=17155, max=60431, avg=36231.16, stdev=5177.62
     lat (usec): min=17407, max=60580, avg=36401.19, stdev=5178.24
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[42730], 95.00th=[45876],
     | 99.00th=[51119], 99.50th=[53740], 99.90th=[56361], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  744, max= 1032, per=25.10%, avg=868.00, stdev=82.97, samples=11
   iops        : min=  186, max=  258, avg=217.00, stdev=20.74, samples=11
  lat (msec)   : 10=0.12%, 20=0.20%, 50=97.73%, 100=1.95%
  cpu          : usr=0.45%, sys=6.17%, ctx=4699, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182442: Sun Mar  1 22:57:42 2026
  read: IOPS=209, BW=839KiB/s (859kB/s)(5060KiB/6032msec)
    slat (usec): min=1015, max=10938, avg=2538.88, stdev=844.49
    clat (usec): min=19818, max=59688, avg=35784.47, stdev=5772.08
     lat (usec): min=22495, max=62633, avg=38324.72, stdev=5912.11
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27395], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[43254], 95.00th=[46400],
     | 99.00th=[51643], 99.50th=[53740], 99.90th=[58983], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  776, max=  976, per=25.14%, avg=835.00, stdev=57.95, samples=11
   iops        : min=  194, max=  244, avg=208.73, stdev=14.51, samples=11
  write: IOPS=214, BW=859KiB/s (879kB/s)(5180KiB/6032msec); 0 zone resets
    slat (usec): min=43, max=2313, avg=168.15, stdev=108.35
    clat (usec): min=23621, max=58718, avg=36387.19, stdev=5313.83
     lat (usec): min=23743, max=58886, avg=36556.50, stdev=5324.70
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28705], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[51643], 99.50th=[53216], 99.90th=[56361], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  760, max=  944, per=24.75%, avg=855.73, stdev=62.49, samples=11
   iops        : min=  190, max=  236, avg=213.91, stdev=15.65, samples=11
  lat (msec)   : 20=0.04%, 50=98.24%, 100=1.72%
  cpu          : usr=0.55%, sys=6.18%, ctx=4678, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3323KiB/s (3402kB/s), 824KiB/s-839KiB/s (844kB/s-859kB/s), io=19.6MiB (20.5MB), run=6012-6040msec
  WRITE: bw=3459KiB/s (3542kB/s), 859KiB/s-874KiB/s (879kB/s-895kB/s), io=20.4MiB (21.4MB), run=6012-6040msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182446: Sun Mar  1 22:57:46 2026
  read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=6644.7k, max=6644.7k, avg=6644678.00, stdev= 0.00
    clat (nsec): min=91299k, max=91299k, avg=91298808.00, stdev= 0.00
     lat (nsec): min=97952k, max=97952k, avg=97951976.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.86%, ctx=6, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout