== sanity test 398c: run fio to test AIO ================= 01:05:49 (1789189549)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.127098 s, 330 MB/s
osc.lustre-OST0000-osc-ffff945dd2dd2800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183296: Sat Sep 12 01:05:58 2026
  write: IOPS=498, BW=1993KiB/s (2041kB/s)(10.0MiB/5137msec); 0 zone resets
    slat (usec): min=25, max=26121, avg=109.23, stdev=522.72
    clat (usec): min=6506, max=62863, avg=31707.44, stdev=5219.26
     lat (usec): min=6641, max=62928, avg=31817.69, stdev=5202.54
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[25560], 10.00th=[26870], 20.00th=[28181],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32900], 80.00th=[34341], 90.00th=[36963], 95.00th=[40633],
     | 99.00th=[56361], 99.50th=[58983], 99.90th=[60556], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1608, max= 2192, per=24.88%, avg=1983.60, stdev=154.03, samples=10
   iops        : min=  402, max=  548, avg=495.90, stdev=38.51, samples=10
  lat (msec)   : 10=0.12%, 20=0.04%, 50=98.44%, 100=1.41%
  cpu          : usr=0.39%, sys=4.38%, ctx=2906, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183297: Sat Sep 12 01:05:58 2026
  write: IOPS=504, BW=2016KiB/s (2065kB/s)(10.0MiB/5079msec); 0 zone resets
    slat (usec): min=23, max=3428, avg=99.74, stdev=94.21
    clat (usec): min=3782, max=62734, avg=31587.06, stdev=5224.73
     lat (usec): min=3857, max=62815, avg=31688.03, stdev=5226.28
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25560], 10.00th=[26870], 20.00th=[28181],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32900], 80.00th=[34866], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[53216], 99.50th=[58983], 99.90th=[62653], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1632, max= 2216, per=25.11%, avg=2002.33, stdev=173.68, samples=9
   iops        : min=  408, max=  554, avg=500.56, stdev=43.39, samples=9
  lat (msec)   : 4=0.08%, 10=0.12%, 20=0.27%, 50=98.32%, 100=1.21%
  cpu          : usr=0.53%, sys=3.94%, ctx=2824, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183298: Sat Sep 12 01:05:58 2026
  write: IOPS=498, BW=1993KiB/s (2041kB/s)(10.0MiB/5137msec); 0 zone resets
    slat (usec): min=21, max=1305, avg=87.60, stdev=61.36
    clat (usec): min=13483, max=62488, avg=31674.73, stdev=5053.12
     lat (usec): min=13736, max=62706, avg=31763.25, stdev=5057.72
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25822], 10.00th=[26870], 20.00th=[28443],
     | 30.00th=[29230], 40.00th=[30016], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32637], 80.00th=[34866], 90.00th=[36963], 95.00th=[39584],
     | 99.00th=[53740], 99.50th=[61604], 99.90th=[62129], 99.95th=[62129],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1656, max= 2256, per=24.91%, avg=1986.40, stdev=161.51, samples=10
   iops        : min=  414, max=  564, avg=496.60, stdev=40.38, samples=10
  lat (msec)   : 20=0.16%, 50=98.63%, 100=1.21%
  cpu          : usr=0.55%, sys=3.37%, ctx=2676, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183299: Sat Sep 12 01:05:58 2026
  write: IOPS=498, BW=1994KiB/s (2042kB/s)(10.0MiB/5136msec); 0 zone resets
    slat (usec): min=25, max=1453, avg=96.64, stdev=69.43
    clat (usec): min=20230, max=62767, avg=31689.77, stdev=4974.86
     lat (usec): min=20298, max=62834, avg=31787.44, stdev=4980.40
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[25560], 10.00th=[26870], 20.00th=[28181],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31589],
     | 70.00th=[32900], 80.00th=[34866], 90.00th=[37487], 95.00th=[40109],
     | 99.00th=[51119], 99.50th=[58459], 99.90th=[62129], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1616, max= 2184, per=24.88%, avg=1983.80, stdev=156.55, samples=10
   iops        : min=  404, max=  546, avg=495.90, stdev=39.12, samples=10
  lat (msec)   : 50=98.91%, 100=1.09%
  cpu          : usr=0.47%, sys=4.28%, ctx=2818, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7974KiB/s (8165kB/s), 1993KiB/s-2016KiB/s (2041kB/s-2065kB/s), io=40.0MiB (41.9MB), run=5079-5137msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183309: Sat Sep 12 01:06:06 2026
  read: IOPS=212, BW=848KiB/s (868kB/s)(5048KiB/5952msec)
    slat (usec): min=51, max=8140, avg=2371.14, stdev=824.71
    clat (usec): min=4546, max=59478, avg=35505.35, stdev=5918.76
     lat (usec): min=5085, max=61944, avg=37878.21, stdev=6093.23
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27132], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39584], 90.00th=[42730], 95.00th=[45876],
     | 99.00th=[52167], 99.50th=[54789], 99.90th=[58983], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  720, max=  936, per=25.23%, avg=845.73, stdev=75.91, samples=11
   iops        : min=  180, max=  234, avg=211.36, stdev=18.92, samples=11
  write: IOPS=218, BW=872KiB/s (893kB/s)(5192KiB/5952msec); 0 zone resets
    slat (usec): min=28, max=534, avg=93.85, stdev=59.49
    clat (usec): min=12774, max=60485, avg=36296.22, stdev=5269.08
     lat (usec): min=13145, max=60553, avg=36391.05, stdev=5269.56
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29230], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[57410], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  744, max=  984, per=24.94%, avg=870.45, stdev=74.17, samples=11
   iops        : min=  186, max=  246, avg=217.55, stdev=18.54, samples=11
  lat (msec)   : 10=0.20%, 20=0.23%, 50=97.93%, 100=1.64%
  cpu          : usr=0.57%, sys=4.10%, ctx=4907, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183310: Sat Sep 12 01:06:06 2026
  read: IOPS=207, BW=831KiB/s (851kB/s)(4976KiB/5986msec)
    slat (usec): min=997, max=8421, avg=2397.00, stdev=809.66
    clat (usec): min=20285, max=60778, avg=35612.63, stdev=5504.41
     lat (usec): min=23531, max=63707, avg=38011.39, stdev=5662.73
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26870], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[42730], 95.00th=[44827],
     | 99.00th=[52167], 99.50th=[53216], 99.90th=[58459], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  656, max=  920, per=24.48%, avg=820.45, stdev=75.39, samples=11
   iops        : min=  164, max=  230, avg=205.09, stdev=18.88, samples=11
  write: IOPS=219, BW=879KiB/s (900kB/s)(5264KiB/5986msec); 0 zone resets
    slat (usec): min=29, max=611, avg=91.55, stdev=59.79
    clat (usec): min=20193, max=52315, avg=36215.50, stdev=4904.10
     lat (usec): min=20273, max=52434, avg=36308.24, stdev=4905.73
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28705], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[42730], 95.00th=[44827],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min=  784, max= 1000, per=25.26%, avg=881.73, stdev=63.07, samples=11
   iops        : min=  196, max=  250, avg=220.36, stdev=15.77, samples=11
  lat (msec)   : 50=99.10%, 100=0.90%
  cpu          : usr=0.45%, sys=4.19%, ctx=4877, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183311: Sat Sep 12 01:06:06 2026
  read: IOPS=208, BW=834KiB/s (854kB/s)(4984KiB/5974msec)
    slat (usec): min=1029, max=7989, avg=2415.74, stdev=800.28
    clat (usec): min=4993, max=63388, avg=35215.37, stdev=5385.39
     lat (usec): min=6837, max=65329, avg=37632.82, stdev=5537.04
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27395], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41681], 95.00th=[44827],
     | 99.00th=[50594], 99.50th=[53740], 99.90th=[58983], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  688, max= 1000, per=24.98%, avg=837.27, stdev=94.99, samples=11
   iops        : min=  172, max=  250, avg=209.27, stdev=23.75, samples=11
  write: IOPS=219, BW=880KiB/s (901kB/s)(5256KiB/5974msec); 0 zone resets
    slat (usec): min=28, max=2534, avg=95.62, stdev=105.96
    clat (usec): min=25111, max=58135, avg=36463.39, stdev=5085.79
     lat (usec): min=25178, max=58202, avg=36560.31, stdev=5085.35
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[51119], 99.50th=[53216], 99.90th=[55837], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  776, max=  984, per=24.95%, avg=870.91, stdev=68.03, samples=11
   iops        : min=  194, max=  246, avg=217.73, stdev=17.01, samples=11
  lat (msec)   : 10=0.04%, 50=98.52%, 100=1.45%
  cpu          : usr=0.37%, sys=4.30%, ctx=4830, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183312: Sat Sep 12 01:06:06 2026
  read: IOPS=211, BW=845KiB/s (866kB/s)(5060KiB/5986msec)
    slat (usec): min=938, max=6575, avg=2394.08, stdev=791.60
    clat (usec): min=14966, max=57200, avg=35543.29, stdev=5703.51
     lat (usec): min=16666, max=60571, avg=37939.69, stdev=5876.43
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27132], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42730], 95.00th=[46400],
     | 99.00th=[52167], 99.50th=[53216], 99.90th=[54789], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  792, max=  928, per=25.06%, avg=839.91, stdev=47.67, samples=11
   iops        : min=  198, max=  232, avg=209.91, stdev=11.97, samples=11
  write: IOPS=216, BW=865KiB/s (886kB/s)(5180KiB/5986msec); 0 zone resets
    slat (usec): min=29, max=752, avg=90.47, stdev=58.20
    clat (usec): min=16672, max=58058, avg=36270.87, stdev=5275.67
     lat (usec): min=16743, max=58137, avg=36362.31, stdev=5275.95
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29230], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43779], 95.00th=[46400],
     | 99.00th=[51643], 99.50th=[54264], 99.90th=[57410], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  688, max=  968, per=24.62%, avg=859.36, stdev=77.52, samples=11
   iops        : min=  172, max=  242, avg=214.82, stdev=19.39, samples=11
  lat (msec)   : 20=0.23%, 50=98.20%, 100=1.56%
  cpu          : usr=0.50%, sys=4.21%, ctx=4802, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3352KiB/s (3433kB/s), 831KiB/s-848KiB/s (851kB/s-868kB/s), io=19.6MiB (20.5MB), run=5952-5986msec
  WRITE: bw=3490KiB/s (3574kB/s), 865KiB/s-880KiB/s (886kB/s-901kB/s), io=20.4MiB (21.4MB), run=5952-5986msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183316: Sat Sep 12 01:06:10 2026
  read: IOPS=8, BW=323MiB/s (338MB/s)(40.0MiB/124msec)
    slat (nsec): min=10801k, max=10801k, avg=10800853.00, stdev= 0.00
    clat (nsec): min=106694k, max=106694k, avg=106693944.00, stdev= 0.00
     lat (nsec): min=117502k, max=117502k, avg=117501704.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  107], 10.00th=[  107], 20.00th=[  107],
     | 30.00th=[  107], 40.00th=[  107], 50.00th=[  107], 60.00th=[  107],
     | 70.00th=[  107], 80.00th=[  107], 90.00th=[  107], 95.00th=[  107],
     | 99.00th=[  107], 99.50th=[  107], 99.90th=[  107], 99.95th=[  107],
     | 99.99th=[  107]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.81%, sys=7.32%, ctx=4, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=323MiB/s (338MB/s), 323MiB/s-323MiB/s (338MB/s-338MB/s), io=40.0MiB (41.9MB), run=124-124msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout