== sanity test 398c: run fio to test AIO ================= 03:51:59 (1782633119)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.154721 s, 271 MB/s
osc.lustre-OST0000-osc-ffff8dfe38dfa000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=193901: Sun Jun 28 03:52:08 2026
  write: IOPS=473, BW=1892KiB/s (1938kB/s)(10.0MiB/5411msec); 0 zone resets
    slat (usec): min=40, max=1215, avg=192.40, stdev=121.09
    clat (usec): min=22934, max=58992, avg=33147.83, stdev=5299.27
     lat (usec): min=23114, max=59353, avg=33341.42, stdev=5303.98
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[27657], 10.00th=[28443], 20.00th=[29754],
     | 30.00th=[30540], 40.00th=[31065], 50.00th=[31851], 60.00th=[32900],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[38011], 95.00th=[40633],
     | 99.00th=[56886], 99.50th=[57934], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1432, max= 2160, per=24.81%, avg=1878.10, stdev=226.35, samples=10
   iops        : min=  358, max=  540, avg=469.50, stdev=56.61, samples=10
  lat (msec)   : 50=97.19%, 100=2.81%
  cpu          : usr=0.54%, sys=6.86%, ctx=2564, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193902: Sun Jun 28 03:52:08 2026
  write: IOPS=473, BW=1894KiB/s (1939kB/s)(10.0MiB/5407msec); 0 zone resets
    slat (usec): min=43, max=990, avg=211.94, stdev=129.59
    clat (usec): min=5793, max=59194, avg=33143.45, stdev=5323.34
     lat (usec): min=5909, max=59381, avg=33356.61, stdev=5334.30
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[27657], 10.00th=[28443], 20.00th=[29754],
     | 30.00th=[30540], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38011], 95.00th=[40633],
     | 99.00th=[56886], 99.50th=[57934], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1440, max= 2176, per=24.85%, avg=1881.00, stdev=217.48, samples=10
   iops        : min=  360, max=  544, avg=470.20, stdev=54.42, samples=10
  lat (msec)   : 10=0.08%, 50=97.07%, 100=2.85%
  cpu          : usr=0.35%, sys=8.03%, ctx=2842, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193903: Sun Jun 28 03:52:08 2026
  write: IOPS=477, BW=1909KiB/s (1955kB/s)(10.0MiB/5364msec); 0 zone resets
    slat (usec): min=43, max=968, avg=209.24, stdev=115.17
    clat (usec): min=4917, max=58709, avg=33244.19, stdev=5494.89
     lat (usec): min=5067, max=58993, avg=33454.60, stdev=5504.13
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[27395], 10.00th=[28443], 20.00th=[29492],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38536], 95.00th=[44303],
     | 99.00th=[56886], 99.50th=[57410], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1472, max= 2144, per=25.10%, avg=1900.10, stdev=200.75, samples=10
   iops        : min=  368, max=  536, avg=475.00, stdev=50.19, samples=10
  lat (msec)   : 10=0.04%, 20=0.08%, 50=97.03%, 100=2.85%
  cpu          : usr=0.48%, sys=7.61%, ctx=2651, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=193904: Sun Jun 28 03:52:08 2026
  write: IOPS=474, BW=1897KiB/s (1943kB/s)(10.0MiB/5397msec); 0 zone resets
    slat (usec): min=46, max=27991, avg=223.13, stdev=563.00
    clat (usec): min=21898, max=58909, avg=33165.97, stdev=5311.00
     lat (usec): min=23421, max=59177, avg=33390.32, stdev=5324.54
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[27657], 10.00th=[28705], 20.00th=[29492],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[38536], 95.00th=[42730],
     | 99.00th=[56886], 99.50th=[57410], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1464, max= 2160, per=24.94%, avg=1887.80, stdev=211.87, samples=10
   iops        : min=  366, max=  540, avg=471.90, stdev=53.00, samples=10
  lat (msec)   : 50=97.34%, 100=2.66%
  cpu          : usr=0.61%, sys=7.89%, ctx=2726, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7570KiB/s (7751kB/s), 1892KiB/s-1909KiB/s (1938kB/s-1955kB/s), io=40.0MiB (41.9MB), run=5364-5411msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=193915: Sun Jun 28 03:52:17 2026
  read: IOPS=195, BW=781KiB/s (800kB/s)(5048KiB/6462msec)
    slat (usec): min=1067, max=11533, avg=2921.29, stdev=1020.24
    clat (usec): min=12190, max=65279, avg=38518.50, stdev=7199.67
     lat (usec): min=14107, max=69039, avg=41441.36, stdev=7485.12
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[41681], 80.00th=[44827], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[58983], 99.50th=[60031], 99.90th=[64226], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  664, max=  912, per=25.21%, avg=782.92, stdev=94.15, samples=12
   iops        : min=  166, max=  228, avg=195.67, stdev=23.52, samples=12
  write: IOPS=200, BW=803KiB/s (823kB/s)(5192KiB/6462msec); 0 zone resets
    slat (usec): min=65, max=851, avg=198.75, stdev=105.80
    clat (usec): min=21722, max=63806, avg=38561.72, stdev=6659.71
     lat (usec): min=21880, max=63950, avg=38761.75, stdev=6648.88
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29754], 10.00th=[31327], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[41681], 80.00th=[44303], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[61080], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  592, max= 1032, per=25.01%, avg=808.58, stdev=140.16, samples=12
   iops        : min=  148, max=  258, avg=202.08, stdev=35.06, samples=12
  lat (msec)   : 20=0.12%, 50=92.66%, 100=7.23%
  cpu          : usr=0.46%, sys=7.07%, ctx=4591, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193916: Sun Jun 28 03:52:17 2026
  read: IOPS=194, BW=779KiB/s (798kB/s)(4976KiB/6389msec)
    slat (usec): min=39, max=9564, avg=2914.06, stdev=995.73
    clat (usec): min=21268, max=75712, avg=38362.68, stdev=7186.76
     lat (usec): min=24312, max=79690, avg=41278.41, stdev=7474.22
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28705], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[41157], 80.00th=[44303], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[72877], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  632, max=  896, per=25.18%, avg=781.92, stdev=91.18, samples=12
   iops        : min=  158, max=  224, avg=195.42, stdev=22.81, samples=12
  write: IOPS=205, BW=824KiB/s (844kB/s)(5264KiB/6389msec); 0 zone resets
    slat (usec): min=42, max=1404, avg=195.82, stdev=111.70
    clat (usec): min=10535, max=75547, avg=38308.57, stdev=6711.64
     lat (usec): min=10615, max=75704, avg=38505.72, stdev=6701.91
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30016], 10.00th=[31327], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[44303], 90.00th=[47449], 95.00th=[50594],
     | 99.00th=[55837], 99.50th=[57934], 99.90th=[71828], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  648, max=  968, per=25.69%, avg=830.42, stdev=133.05, samples=12
   iops        : min=  162, max=  242, avg=207.58, stdev=33.25, samples=12
  lat (msec)   : 20=0.20%, 50=93.05%, 100=6.76%
  cpu          : usr=0.49%, sys=7.03%, ctx=4581, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193917: Sun Jun 28 03:52:17 2026
  read: IOPS=192, BW=772KiB/s (790kB/s)(4984KiB/6459msec)
    slat (usec): min=1134, max=8592, avg=2894.70, stdev=979.43
    clat (usec): min=22507, max=61860, avg=38258.45, stdev=6878.88
     lat (usec): min=25469, max=65258, avg=41154.67, stdev=7195.36
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28705], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[56361], 99.50th=[57410], 99.90th=[60556], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  568, max=  968, per=25.02%, avg=776.83, stdev=133.28, samples=12
   iops        : min=  142, max=  242, avg=194.17, stdev=33.33, samples=12
  write: IOPS=203, BW=814KiB/s (833kB/s)(5256KiB/6459msec); 0 zone resets
    slat (usec): min=42, max=1892, avg=197.60, stdev=122.90
    clat (usec): min=22837, max=61077, avg=38825.22, stdev=6511.12
     lat (usec): min=22967, max=61145, avg=39024.02, stdev=6503.92
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30016], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[42206], 80.00th=[45351], 90.00th=[48497], 95.00th=[50594],
     | 99.00th=[55313], 99.50th=[58459], 99.90th=[60556], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  640, max= 1016, per=25.13%, avg=812.58, stdev=106.50, samples=12
   iops        : min=  160, max=  254, avg=203.08, stdev=26.64, samples=12
  lat (msec)   : 50=93.16%, 100=6.84%
  cpu          : usr=0.65%, sys=6.83%, ctx=4623, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=193918: Sun Jun 28 03:52:17 2026
  read: IOPS=195, BW=783KiB/s (802kB/s)(5060KiB/6459msec)
    slat (usec): min=1118, max=9136, avg=2905.18, stdev=996.28
    clat (usec): min=20490, max=59194, avg=38431.86, stdev=6986.60
     lat (usec): min=23027, max=62778, avg=41338.62, stdev=7295.05
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[41681], 80.00th=[44827], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  600, max=  920, per=25.18%, avg=781.75, stdev=106.78, samples=12
   iops        : min=  150, max=  230, avg=195.42, stdev=26.70, samples=12
  write: IOPS=200, BW=802KiB/s (821kB/s)(5180KiB/6459msec); 0 zone resets
    slat (usec): min=62, max=942, avg=197.22, stdev=111.90
    clat (usec): min=23602, max=60278, avg=38600.06, stdev=6199.45
     lat (usec): min=23714, max=60457, avg=38799.23, stdev=6188.73
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30278], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[44827], 90.00th=[47973], 95.00th=[50070],
     | 99.00th=[53740], 99.50th=[54789], 99.90th=[56886], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  576, max=  952, per=25.04%, avg=809.50, stdev=113.54, samples=12
   iops        : min=  144, max=  238, avg=202.33, stdev=28.40, samples=12
  lat (msec)   : 50=93.63%, 100=6.37%
  cpu          : usr=0.51%, sys=7.01%, ctx=4643, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3106KiB/s (3180kB/s), 772KiB/s-783KiB/s (790kB/s-802kB/s), io=19.6MiB (20.5MB), run=6389-6462msec
  WRITE: bw=3233KiB/s (3311kB/s), 802KiB/s-824KiB/s (821kB/s-844kB/s), io=20.4MiB (21.4MB), run=6389-6462msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=193923: Sun Jun 28 03:52:21 2026
  read: IOPS=10, BW=430MiB/s (451MB/s)(40.0MiB/93msec)
    slat (nsec): min=9573.6k, max=9573.6k, avg=9573627.00, stdev= 0.00
    clat (nsec): min=78142k, max=78142k, avg=78141776.00, stdev= 0.00
     lat (nsec): min=87718k, max=87718k, avg=87718475.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[78119],  5.00th=[78119], 10.00th=[78119], 20.00th=[78119],
     | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119],
     | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119],
     | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.70%, ctx=10, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=430MiB/s (451MB/s), 430MiB/s-430MiB/s (451MB/s-451MB/s), io=40.0MiB (41.9MB), run=93-93msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout