== sanity test 398c: run fio to test AIO ================= 12:28:19 (1776875299)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.147256 s, 285 MB/s
osc.lustre-OST0000-osc-ffff949a39118000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=186975: Wed Apr 22 12:28:28 2026
  write: IOPS=470, BW=1880KiB/s (1925kB/s)(10.0MiB/5446msec); 0 zone resets
    slat (usec): min=41, max=1117, avg=200.82, stdev=105.60
    clat (usec): min=15784, max=71950, avg=33782.27, stdev=4512.49
     lat (usec): min=15909, max=72019, avg=33984.37, stdev=4517.60
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[26084], 10.00th=[28443], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[46924], 99.50th=[53216], 99.90th=[64750], 99.95th=[69731],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1792, max= 2157, per=25.26%, avg=1884.50, stdev=106.42, samples=10
   iops        : min=  448, max=  539, avg=471.10, stdev=26.53, samples=10
  lat (msec)   : 20=1.21%, 50=98.05%, 100=0.74%
  cpu          : usr=0.55%, sys=7.02%, ctx=2649, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186976: Wed Apr 22 12:28:28 2026
  write: IOPS=466, BW=1865KiB/s (1910kB/s)(10.0MiB/5491msec); 0 zone resets
    slat (usec): min=48, max=709, avg=194.57, stdev=99.27
    clat (usec): min=13639, max=58952, avg=33784.36, stdev=3987.07
     lat (usec): min=13921, max=59107, avg=33980.20, stdev=3989.80
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26608], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[43779], 99.50th=[48497], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1784, max= 1983, per=25.06%, avg=1869.50, stdev=61.88, samples=10
   iops        : min=  446, max=  495, avg=467.30, stdev=15.32, samples=10
  lat (msec)   : 20=0.31%, 50=99.30%, 100=0.39%
  cpu          : usr=0.51%, sys=7.43%, ctx=2724, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186977: Wed Apr 22 12:28:28 2026
  write: IOPS=468, BW=1873KiB/s (1918kB/s)(10.0MiB/5466msec); 0 zone resets
    slat (usec): min=47, max=27335, avg=210.69, stdev=546.10
    clat (usec): min=6725, max=66662, avg=33656.33, stdev=4473.18
     lat (usec): min=6866, max=66758, avg=33868.26, stdev=4451.56
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[26084], 10.00th=[28443], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[45351], 99.50th=[52167], 99.90th=[54789], 99.95th=[60031],
     | 99.99th=[66847]
   bw (  KiB/s): min= 1792, max= 2067, per=25.20%, avg=1879.50, stdev=80.99, samples=10
   iops        : min=  448, max=  516, avg=469.80, stdev=20.05, samples=10
  lat (msec)   : 10=0.16%, 20=1.17%, 50=98.12%, 100=0.55%
  cpu          : usr=0.48%, sys=7.56%, ctx=2670, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186978: Wed Apr 22 12:28:28 2026
  write: IOPS=466, BW=1865KiB/s (1910kB/s)(10.0MiB/5491msec); 0 zone resets
    slat (usec): min=49, max=697, avg=184.43, stdev=98.32
    clat (usec): min=10602, max=54671, avg=33769.13, stdev=3893.61
     lat (usec): min=10734, max=54869, avg=33954.84, stdev=3903.25
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26608], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36439], 90.00th=[38011], 95.00th=[38536],
     | 99.00th=[43254], 99.50th=[50594], 99.90th=[54264], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1792, max= 1951, per=24.95%, avg=1860.70, stdev=58.40, samples=10
   iops        : min=  448, max=  487, avg=465.10, stdev=14.47, samples=10
  lat (msec)   : 20=0.39%, 50=99.06%, 100=0.55%
  cpu          : usr=0.71%, sys=6.25%, ctx=2682, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7459KiB/s (7639kB/s), 1865KiB/s-1880KiB/s (1910kB/s-1925kB/s), io=40.0MiB (41.9MB), run=5446-5491msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=186988: Wed Apr 22 12:28:38 2026
  read: IOPS=185, BW=743KiB/s (761kB/s)(5048KiB/6795msec)
    slat (usec): min=75, max=8149, avg=2940.72, stdev=950.98
    clat (usec): min=16428, max=60406, avg=40612.17, stdev=6656.21
     lat (usec): min=17067, max=64440, avg=43554.52, stdev=6922.64
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[29754], 10.00th=[32113], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49546], 95.00th=[51643],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[58983], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  584, max=  952, per=25.10%, avg=739.54, stdev=85.40, samples=13
   iops        : min=  146, max=  238, avg=184.85, stdev=21.38, samples=13
  write: IOPS=191, BW=764KiB/s (782kB/s)(5192KiB/6795msec); 0 zone resets
    slat (usec): min=46, max=735, avg=178.97, stdev=92.84
    clat (usec): min=11137, max=62369, avg=40911.53, stdev=6331.85
     lat (usec): min=11259, max=62499, avg=41091.63, stdev=6331.89
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[31065], 10.00th=[33162], 20.00th=[35390],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40633], 60.00th=[42730],
     | 70.00th=[44827], 80.00th=[46400], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[61080], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  680, max=  920, per=24.80%, avg=760.69, stdev=68.34, samples=13
   iops        : min=  170, max=  230, avg=190.08, stdev=17.15, samples=13
  lat (msec)   : 20=0.35%, 50=91.48%, 100=8.16%
  cpu          : usr=0.46%, sys=6.14%, ctx=4798, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186989: Wed Apr 22 12:28:38 2026
  read: IOPS=184, BW=736KiB/s (754kB/s)(4976KiB/6758msec)
    slat (usec): min=41, max=7090, avg=2935.17, stdev=978.57
    clat (usec): min=15143, max=62543, avg=40653.84, stdev=6617.08
     lat (usec): min=15195, max=64718, avg=43590.83, stdev=6890.08
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30540], 10.00th=[32375], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[38536], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[59507], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  552, max=  888, per=24.94%, avg=734.77, stdev=80.89, samples=13
   iops        : min=  138, max=  222, avg=183.62, stdev=20.25, samples=13
  write: IOPS=194, BW=779KiB/s (798kB/s)(5264KiB/6758msec); 0 zone resets
    slat (usec): min=42, max=714, avg=180.66, stdev=93.40
    clat (usec): min=11569, max=65872, avg=40639.30, stdev=6724.89
     lat (usec): min=11736, max=66070, avg=40821.05, stdev=6729.34
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[31065], 10.00th=[32900], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49021], 95.00th=[50594],
     | 99.00th=[54789], 99.50th=[56886], 99.90th=[63701], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  654, max=  976, per=25.31%, avg=776.23, stdev=83.66, samples=13
   iops        : min=  163, max=  244, avg=194.00, stdev=20.96, samples=13
  lat (msec)   : 20=0.94%, 50=91.72%, 100=7.34%
  cpu          : usr=0.47%, sys=6.14%, ctx=4798, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186990: Wed Apr 22 12:28:38 2026
  read: IOPS=183, BW=733KiB/s (751kB/s)(4984KiB/6800msec)
    slat (usec): min=63, max=6464, avg=2916.83, stdev=987.99
    clat (usec): min=17255, max=61918, avg=40716.75, stdev=6693.83
     lat (usec): min=17348, max=64704, avg=43635.17, stdev=6980.25
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[30278], 10.00th=[32375], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49546], 95.00th=[51643],
     | 99.00th=[57934], 99.50th=[60556], 99.90th=[61604], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  664, max=  904, per=24.84%, avg=731.69, stdev=82.84, samples=13
   iops        : min=  166, max=  226, avg=182.85, stdev=20.74, samples=13
  write: IOPS=193, BW=773KiB/s (791kB/s)(5256KiB/6800msec); 0 zone resets
    slat (usec): min=59, max=2353, avg=185.38, stdev=114.15
    clat (usec): min=18977, max=58999, avg=40900.05, stdev=6062.60
     lat (usec): min=19088, max=59092, avg=41086.45, stdev=6063.39
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[31065], 10.00th=[33424], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[46400], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  600, max=  896, per=24.93%, avg=764.69, stdev=85.24, samples=13
   iops        : min=  150, max=  224, avg=191.08, stdev=21.41, samples=13
  lat (msec)   : 20=0.12%, 50=92.38%, 100=7.50%
  cpu          : usr=0.34%, sys=6.16%, ctx=4780, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186991: Wed Apr 22 12:28:38 2026
  read: IOPS=185, BW=743KiB/s (761kB/s)(5060KiB/6810msec)
    slat (usec): min=1200, max=8126, avg=2965.59, stdev=950.14
    clat (usec): min=4765, max=63642, avg=40654.24, stdev=7058.27
     lat (usec): min=7092, max=67499, avg=43621.56, stdev=7285.71
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[30016], 10.00th=[32113], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[38536], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44827], 80.00th=[46924], 90.00th=[49546], 95.00th=[51643],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[60556], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  590, max=  864, per=24.92%, avg=734.15, stdev=72.94, samples=13
   iops        : min=  147, max=  216, avg=183.46, stdev=18.35, samples=13
  write: IOPS=190, BW=761KiB/s (779kB/s)(5180KiB/6810msec); 0 zone resets
    slat (usec): min=53, max=693, avg=184.97, stdev=94.03
    clat (usec): min=7100, max=61763, avg=40848.88, stdev=6390.50
     lat (usec): min=7346, max=62184, avg=41034.85, stdev=6385.07
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[30802], 10.00th=[33162], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49021], 95.00th=[50594],
     | 99.00th=[54264], 99.50th=[56886], 99.90th=[61080], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  640, max=  888, per=24.52%, avg=752.00, stdev=75.07, samples=13
   iops        : min=  160, max=  222, avg=187.92, stdev=18.77, samples=13
  lat (msec)   : 10=0.20%, 20=0.31%, 50=91.80%, 100=7.70%
  cpu          : usr=0.51%, sys=6.11%, ctx=4822, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2947KiB/s (3018kB/s), 733KiB/s-743KiB/s (751kB/s-761kB/s), io=19.6MiB (20.5MB), run=6758-6810msec
  WRITE: bw=3068KiB/s (3141kB/s), 761KiB/s-779KiB/s (779kB/s-798kB/s), io=20.4MiB (21.4MB), run=6758-6810msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=186995: Wed Apr 22 12:28:42 2026
  read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec)
    slat (nsec): min=8353.9k, max=8353.9k, avg=8353925.00, stdev= 0.00
    clat (nsec): min=92302k, max=92302k, avg=92301560.00, stdev= 0.00
     lat (nsec): min=100659k, max=100659k, avg=100659115.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[92799],  5.00th=[92799], 10.00th=[92799], 20.00th=[92799],
     | 30.00th=[92799], 40.00th=[92799], 50.00th=[92799], 60.00th=[92799],
     | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799],
     | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799],
     | 99.99th=[92799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.62%, ctx=6, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout