== sanity test 398c: run fio to test AIO ================= 21:26:46 (1777426006)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.130569 s, 321 MB/s
osc.lustre-OST0000-osc-ffff8f7b42d27800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190429: Tue Apr 28 21:26:54 2026
  write: IOPS=553, BW=2212KiB/s (2265kB/s)(10.0MiB/4629msec); 0 zone resets
    slat (usec): min=33, max=762, avg=150.47, stdev=80.97
    clat (usec): min=7038, max=38994, avg=28491.94, stdev=2887.83
     lat (usec): min=7332, max=39127, avg=28643.37, stdev=2899.15
    clat percentiles (usec):
     |  1.00th=[17695],  5.00th=[24511], 10.00th=[25822], 20.00th=[26870],
     | 30.00th=[27395], 40.00th=[27919], 50.00th=[28443], 60.00th=[28967],
     | 70.00th=[29492], 80.00th=[30278], 90.00th=[31589], 95.00th=[32637],
     | 99.00th=[36963], 99.50th=[37487], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 2072, max= 2328, per=24.91%, avg=2204.44, stdev=73.33, samples=9
   iops        : min=  518, max=  582, avg=551.11, stdev=18.33, samples=9
  lat (msec)   : 10=0.04%, 20=1.76%, 50=98.20%
  cpu          : usr=0.41%, sys=6.46%, ctx=2506, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190430: Tue Apr 28 21:26:54 2026
  write: IOPS=556, BW=2225KiB/s (2278kB/s)(10.0MiB/4603msec); 0 zone resets
    slat (usec): min=33, max=597, avg=162.67, stdev=81.43
    clat (usec): min=3064, max=50607, avg=28558.50, stdev=3157.04
     lat (usec): min=3191, max=50738, avg=28722.15, stdev=3161.23
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[24773], 10.00th=[25822], 20.00th=[26870],
     | 30.00th=[27395], 40.00th=[27919], 50.00th=[28443], 60.00th=[28967],
     | 70.00th=[29754], 80.00th=[30278], 90.00th=[31589], 95.00th=[33162],
     | 99.00th=[36963], 99.50th=[39060], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[50594]
   bw (  KiB/s): min= 2112, max= 2365, per=25.10%, avg=2220.62, stdev=91.43, samples=8
   iops        : min=  528, max=  591, avg=555.12, stdev=22.80, samples=8
  lat (msec)   : 4=0.04%, 20=1.88%, 50=98.05%, 100=0.04%
  cpu          : usr=0.33%, sys=7.06%, ctx=2702, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190431: Tue Apr 28 21:26:54 2026
  write: IOPS=554, BW=2220KiB/s (2273kB/s)(10.0MiB/4613msec); 0 zone resets
    slat (usec): min=38, max=23146, avg=172.82, stdev=461.57
    clat (usec): min=6576, max=46565, avg=28482.68, stdev=3095.51
     lat (usec): min=11694, max=46691, avg=28656.60, stdev=3070.07
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[24773], 10.00th=[25822], 20.00th=[26870],
     | 30.00th=[27395], 40.00th=[27919], 50.00th=[28443], 60.00th=[28967],
     | 70.00th=[29492], 80.00th=[30278], 90.00th=[31589], 95.00th=[32900],
     | 99.00th=[36963], 99.50th=[38536], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[46400]
   bw (  KiB/s): min= 2096, max= 2332, per=25.03%, avg=2214.50, stdev=84.88, samples=8
   iops        : min=  524, max=  583, avg=553.63, stdev=21.22, samples=8
  lat (msec)   : 10=0.04%, 20=1.91%, 50=98.05%
  cpu          : usr=0.54%, sys=7.44%, ctx=2693, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190432: Tue Apr 28 21:26:54 2026
  write: IOPS=553, BW=2212KiB/s (2265kB/s)(10.0MiB/4629msec); 0 zone resets
    slat (usec): min=35, max=660, avg=150.23, stdev=79.84
    clat (usec): min=10820, max=39070, avg=28513.80, stdev=2943.37
     lat (usec): min=10975, max=39180, avg=28665.02, stdev=2944.14
    clat percentiles (usec):
     |  1.00th=[17695],  5.00th=[24511], 10.00th=[25560], 20.00th=[26608],
     | 30.00th=[27395], 40.00th=[28181], 50.00th=[28705], 60.00th=[28967],
     | 70.00th=[29754], 80.00th=[30278], 90.00th=[31589], 95.00th=[32900],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 2072, max= 2352, per=24.90%, avg=2203.22, stdev=75.45, samples=9
   iops        : min=  518, max=  588, avg=550.78, stdev=18.85, samples=9
  lat (msec)   : 20=1.84%, 50=98.16%
  cpu          : usr=0.52%, sys=6.70%, ctx=2667, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8849KiB/s (9061kB/s), 2212KiB/s-2225KiB/s (2265kB/s-2278kB/s), io=40.0MiB (41.9MB), run=4603-4629msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190442: Tue Apr 28 21:27:02 2026
  read: IOPS=223, BW=893KiB/s (915kB/s)(5048KiB/5652msec)
    slat (usec): min=971, max=5389, avg=2326.36, stdev=699.30
    clat (usec): min=3036, max=52980, avg=33724.58, stdev=5805.07
     lat (usec): min=4668, max=54582, avg=36052.82, stdev=5965.99
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[25297], 10.00th=[27132], 20.00th=[29230],
     | 30.00th=[30540], 40.00th=[31851], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[49021], 99.50th=[49546], 99.90th=[51119], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  656, max= 1072, per=25.07%, avg=889.82, stdev=103.90, samples=11
   iops        : min=  164, max=  268, avg=222.45, stdev=25.97, samples=11
  write: IOPS=229, BW=919KiB/s (941kB/s)(5192KiB/5652msec); 0 zone resets
    slat (usec): min=53, max=1118, avg=163.28, stdev=88.51
    clat (usec): min=14428, max=50967, avg=34003.70, stdev=4866.90
     lat (usec): min=14503, max=51028, avg=34167.97, stdev=4868.81
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[40633], 95.00th=[43254],
     | 99.00th=[46924], 99.50th=[49021], 99.90th=[50594], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=  800, max= 1008, per=24.87%, avg=919.09, stdev=68.42, samples=11
   iops        : min=  200, max=  252, avg=229.73, stdev=17.18, samples=11
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.27%, 50=99.30%, 100=0.35%
  cpu          : usr=0.50%, sys=6.67%, ctx=4769, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190443: Tue Apr 28 21:27:02 2026
  read: IOPS=220, BW=883KiB/s (904kB/s)(4976KiB/5635msec)
    slat (usec): min=952, max=7243, avg=2369.14, stdev=744.19
    clat (usec): min=5381, max=60710, avg=33727.01, stdev=5845.33
     lat (usec): min=7044, max=65451, avg=36097.87, stdev=6024.31
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[25297], 10.00th=[27132], 20.00th=[28967],
     | 30.00th=[30540], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[38536], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[49546], 99.50th=[51643], 99.90th=[58459], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  752, max=  992, per=24.77%, avg=879.18, stdev=72.49, samples=11
   iops        : min=  188, max=  248, avg=219.73, stdev=18.20, samples=11
  write: IOPS=233, BW=934KiB/s (957kB/s)(5264KiB/5635msec); 0 zone resets
    slat (usec): min=40, max=778, avg=169.26, stdev=86.75
    clat (usec): min=21053, max=53255, avg=33927.71, stdev=4972.13
     lat (usec): min=21144, max=53421, avg=34098.02, stdev=4978.02
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26870], 10.00th=[27919], 20.00th=[29754],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[40633], 95.00th=[43254],
     | 99.00th=[46924], 99.50th=[49021], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=  784, max= 1064, per=25.19%, avg=931.18, stdev=82.20, samples=11
   iops        : min=  196, max=  266, avg=232.73, stdev=20.50, samples=11
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.26%, 100=0.62%
  cpu          : usr=0.44%, sys=6.85%, ctx=4763, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190444: Tue Apr 28 21:27:02 2026
  read: IOPS=222, BW=888KiB/s (909kB/s)(4984KiB/5612msec)
    slat (usec): min=37, max=5669, avg=2336.62, stdev=755.88
    clat (usec): min=2895, max=51268, avg=33503.10, stdev=6055.72
     lat (usec): min=4323, max=54336, avg=35841.61, stdev=6269.00
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[25035], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30540], 40.00th=[32113], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[41157], 95.00th=[44303],
     | 99.00th=[47973], 99.50th=[49021], 99.90th=[50594], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=  776, max=  984, per=24.85%, avg=882.20, stdev=68.42, samples=10
   iops        : min=  194, max=  246, avg=220.50, stdev=17.06, samples=10
  write: IOPS=234, BW=937KiB/s (959kB/s)(5256KiB/5612msec); 0 zone resets
    slat (usec): min=37, max=934, avg=165.58, stdev=88.73
    clat (usec): min=9404, max=55225, avg=34060.58, stdev=5068.50
     lat (usec): min=9645, max=55312, avg=34227.30, stdev=5075.05
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[26870], 10.00th=[28181], 20.00th=[29754],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33424], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[38536], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[46924], 99.50th=[48497], 99.90th=[54789], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  768, max= 1048, per=25.14%, avg=929.20, stdev=85.38, samples=10
   iops        : min=  192, max=  262, avg=232.30, stdev=21.34, samples=10
  lat (msec)   : 4=0.04%, 10=0.35%, 20=0.27%, 50=99.06%, 100=0.27%
  cpu          : usr=0.46%, sys=6.70%, ctx=4740, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190445: Tue Apr 28 21:27:02 2026
  read: IOPS=223, BW=896KiB/s (917kB/s)(5060KiB/5649msec)
    slat (usec): min=948, max=5405, avg=2324.63, stdev=740.94
    clat (usec): min=4705, max=55359, avg=33469.09, stdev=5701.69
     lat (usec): min=7084, max=59267, avg=35795.22, stdev=5922.84
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[25297], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30540], 40.00th=[31589], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[41157], 95.00th=[43779],
     | 99.00th=[49021], 99.50th=[51119], 99.90th=[54789], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  736, max= 1064, per=25.21%, avg=894.91, stdev=86.16, samples=11
   iops        : min=  184, max=  266, avg=223.73, stdev=21.54, samples=11
  write: IOPS=229, BW=917KiB/s (939kB/s)(5180KiB/5649msec); 0 zone resets
    slat (usec): min=55, max=1484, avg=163.45, stdev=91.70
    clat (usec): min=18579, max=52877, avg=34196.57, stdev=5184.48
     lat (usec): min=18719, max=52933, avg=34360.96, stdev=5183.59
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[36439], 80.00th=[38536], 90.00th=[41157], 95.00th=[43779],
     | 99.00th=[49546], 99.50th=[50070], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min=  808, max= 1064, per=24.81%, avg=916.91, stdev=84.32, samples=11
   iops        : min=  202, max=  266, avg=229.18, stdev=21.12, samples=11
  lat (msec)   : 10=0.16%, 20=0.16%, 50=99.06%, 100=0.62%
  cpu          : usr=0.32%, sys=6.89%, ctx=4750, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3551KiB/s (3636kB/s), 883KiB/s-896KiB/s (904kB/s-917kB/s), io=19.6MiB (20.5MB), run=5612-5652msec
  WRITE: bw=3696KiB/s (3785kB/s), 917KiB/s-937KiB/s (939kB/s-959kB/s), io=20.4MiB (21.4MB), run=5612-5652msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190449: Tue Apr 28 21:27:06 2026
  read: IOPS=10, BW=421MiB/s (442MB/s)(40.0MiB/95msec)
    slat (nsec): min=9675.4k, max=9675.4k, avg=9675431.00, stdev= 0.00
    clat (nsec): min=80613k, max=80613k, avg=80613366.00, stdev= 0.00
     lat (nsec): min=90293k, max=90293k, avg=90293198.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[80217],  5.00th=[80217], 10.00th=[80217], 20.00th=[80217],
     | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[80217],
     | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217],
     | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.51%, ctx=11, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=421MiB/s (442MB/s), 421MiB/s-421MiB/s (442MB/s-442MB/s), io=40.0MiB (41.9MB), run=95-95msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout