== sanity test 398c: run fio to test AIO ================= 09:57:57 (1789307877)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.150407 s, 279 MB/s
osc.lustre-OST0000-osc-ffff979d48304800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182152: Sun Sep 13 09:58:07 2026
  write: IOPS=411, BW=1646KiB/s (1686kB/s)(10.0MiB/6221msec); 0 zone resets
    slat (usec): min=22, max=2837, avg=110.68, stdev=100.26
    clat (usec): min=20062, max=67472, avg=38397.76, stdev=7282.43
     lat (usec): min=20306, max=67662, avg=38509.57, stdev=7289.92
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[28443], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[47973], 95.00th=[52691],
     | 99.00th=[59507], 99.50th=[62653], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1488, max= 1852, per=24.89%, avg=1638.67, stdev=113.86, samples=12
   iops        : min=  372, max=  463, avg=409.67, stdev=28.46, samples=12
  lat (msec)   : 50=92.85%, 100=7.15%
  cpu          : usr=0.68%, sys=3.63%, ctx=2484, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182153: Sun Sep 13 09:58:07 2026
  write: IOPS=414, BW=1657KiB/s (1696kB/s)(10.0MiB/6181msec); 0 zone resets
    slat (usec): min=22, max=3053, avg=119.02, stdev=119.26
    clat (usec): min=6131, max=79945, avg=38453.34, stdev=7499.63
     lat (usec): min=6257, max=80069, avg=38573.57, stdev=7509.63
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[28705], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[47973], 95.00th=[53216],
     | 99.00th=[62129], 99.50th=[64750], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1456, max= 1980, per=25.11%, avg=1653.33, stdev=143.72, samples=12
   iops        : min=  364, max=  495, avg=413.33, stdev=35.93, samples=12
  lat (msec)   : 10=0.08%, 20=0.12%, 50=92.62%, 100=7.19%
  cpu          : usr=0.71%, sys=3.79%, ctx=2607, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182154: Sun Sep 13 09:58:07 2026
  write: IOPS=412, BW=1648KiB/s (1688kB/s)(10.0MiB/6213msec); 0 zone resets
    slat (usec): min=28, max=2713, avg=122.28, stdev=119.56
    clat (usec): min=3911, max=65093, avg=38380.54, stdev=7329.53
     lat (usec): min=4202, max=65369, avg=38503.98, stdev=7336.71
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[28443], 10.00th=[30802], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[47973], 95.00th=[52691],
     | 99.00th=[60031], 99.50th=[61080], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[65274]
   bw (  KiB/s): min= 1432, max= 1871, per=24.92%, avg=1640.92, stdev=125.45, samples=12
   iops        : min=  358, max=  467, avg=410.17, stdev=31.24, samples=12
  lat (msec)   : 4=0.04%, 10=0.12%, 20=0.20%, 50=92.58%, 100=7.07%
  cpu          : usr=0.66%, sys=4.31%, ctx=2624, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182155: Sun Sep 13 09:58:07 2026
  write: IOPS=412, BW=1652KiB/s (1692kB/s)(10.0MiB/6199msec); 0 zone resets
    slat (usec): min=27, max=2733, avg=112.70, stdev=103.16
    clat (usec): min=17852, max=82920, avg=38502.35, stdev=7627.96
     lat (usec): min=17982, max=83121, avg=38616.21, stdev=7635.01
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[28443], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[43254], 90.00th=[48497], 95.00th=[53216],
     | 99.00th=[60556], 99.50th=[65274], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1440, max= 2006, per=25.04%, avg=1648.83, stdev=151.11, samples=12
   iops        : min=  360, max=  501, avg=412.17, stdev=37.67, samples=12
  lat (msec)   : 20=0.35%, 50=92.38%, 100=7.27%
  cpu          : usr=0.53%, sys=3.81%, ctx=2598, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6584KiB/s (6742kB/s), 1646KiB/s-1657KiB/s (1686kB/s-1696kB/s), io=40.0MiB (41.9MB), run=6181-6221msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182165: Sun Sep 13 09:58:18 2026
  read: IOPS=164, BW=657KiB/s (673kB/s)(5048KiB/7682msec)
    slat (usec): min=1172, max=11847, avg=3168.34, stdev=1310.57
    clat (usec): min=24494, max=91194, avg=45852.64, stdev=9288.64
     lat (usec): min=26717, max=95400, avg=49023.90, stdev=9605.04
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[32900], 10.00th=[34866], 20.00th=[37487],
     | 30.00th=[40109], 40.00th=[42730], 50.00th=[44827], 60.00th=[47449],
     | 70.00th=[50070], 80.00th=[52691], 90.00th=[57934], 95.00th=[61604],
     | 99.00th=[72877], 99.50th=[79168], 99.90th=[88605], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min=  509, max=  800, per=25.13%, avg=656.07, stdev=80.97, samples=15
   iops        : min=  127, max=  200, avg=163.93, stdev=20.25, samples=15
  write: IOPS=168, BW=676KiB/s (692kB/s)(5192KiB/7682msec); 0 zone resets
    slat (usec): min=25, max=2206, avg=112.98, stdev=115.80
    clat (usec): min=24523, max=95232, avg=46221.29, stdev=8973.42
     lat (usec): min=24577, max=95292, avg=46335.41, stdev=8974.62
    clat percentiles (usec):
     |  1.00th=[31851],  5.00th=[34341], 10.00th=[35914], 20.00th=[38536],
     | 30.00th=[41157], 40.00th=[42730], 50.00th=[44827], 60.00th=[46924],
     | 70.00th=[49546], 80.00th=[52691], 90.00th=[57934], 95.00th=[61604],
     | 99.00th=[74974], 99.50th=[78119], 99.90th=[89654], 99.95th=[94897],
     | 99.99th=[94897]
   bw (  KiB/s): min=  488, max=  814, per=24.89%, avg=676.40, stdev=90.90, samples=15
   iops        : min=  122, max=  203, avg=168.93, stdev=22.69, samples=15
  lat (msec)   : 50=70.86%, 100=29.14%
  cpu          : usr=0.43%, sys=4.15%, ctx=4714, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182166: Sun Sep 13 09:58:18 2026
  read: IOPS=162, BW=649KiB/s (664kB/s)(4976KiB/7669msec)
    slat (usec): min=1166, max=10963, avg=3206.27, stdev=1267.33
    clat (usec): min=10142, max=87055, avg=45749.00, stdev=9465.82
     lat (usec): min=12788, max=90722, avg=48958.19, stdev=9790.07
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[32637], 10.00th=[34866], 20.00th=[37487],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44827], 60.00th=[46924],
     | 70.00th=[50070], 80.00th=[53216], 90.00th=[57934], 95.00th=[62129],
     | 99.00th=[73925], 99.50th=[78119], 99.90th=[80217], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min=  533, max=  800, per=24.82%, avg=647.93, stdev=83.33, samples=15
   iops        : min=  133, max=  200, avg=161.87, stdev=20.85, samples=15
  write: IOPS=171, BW=686KiB/s (703kB/s)(5264KiB/7669msec); 0 zone resets
    slat (usec): min=24, max=1654, avg=106.99, stdev=85.63
    clat (usec): min=24758, max=94489, avg=46400.45, stdev=8811.68
     lat (usec): min=24823, max=94571, avg=46508.64, stdev=8812.32
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[34866], 10.00th=[36439], 20.00th=[39060],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[52691], 90.00th=[57934], 95.00th=[62129],
     | 99.00th=[74974], 99.50th=[81265], 99.90th=[90702], 99.95th=[94897],
     | 99.99th=[94897]
   bw (  KiB/s): min=  560, max=  816, per=25.31%, avg=687.93, stdev=85.04, samples=15
   iops        : min=  140, max=  204, avg=171.80, stdev=21.24, samples=15
  lat (msec)   : 20=0.08%, 50=70.82%, 100=29.10%
  cpu          : usr=0.59%, sys=4.02%, ctx=4524, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182167: Sun Sep 13 09:58:18 2026
  read: IOPS=162, BW=649KiB/s (664kB/s)(4984KiB/7684msec)
    slat (usec): min=1217, max=10027, avg=3197.61, stdev=1299.67
    clat (usec): min=20352, max=94156, avg=45831.59, stdev=9573.77
     lat (usec): min=23294, max=97445, avg=49032.25, stdev=9946.02
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[33424], 10.00th=[35390], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44303], 60.00th=[46924],
     | 70.00th=[49546], 80.00th=[53216], 90.00th=[57934], 95.00th=[62653],
     | 99.00th=[73925], 99.50th=[84411], 99.90th=[90702], 99.95th=[93848],
     | 99.99th=[93848]
   bw (  KiB/s): min=  480, max=  816, per=24.87%, avg=649.27, stdev=102.97, samples=15
   iops        : min=  120, max=  204, avg=162.20, stdev=25.75, samples=15
  write: IOPS=171, BW=684KiB/s (700kB/s)(5256KiB/7684msec); 0 zone resets
    slat (usec): min=33, max=944, avg=105.14, stdev=70.17
    clat (usec): min=18891, max=91858, avg=46224.34, stdev=8593.47
     lat (usec): min=18962, max=91933, avg=46330.55, stdev=8597.22
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[34341], 10.00th=[35914], 20.00th=[39060],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[53216], 90.00th=[56886], 95.00th=[61080],
     | 99.00th=[70779], 99.50th=[76022], 99.90th=[90702], 99.95th=[91751],
     | 99.99th=[91751]
   bw (  KiB/s): min=  560, max=  800, per=25.09%, avg=681.93, stdev=67.33, samples=15
   iops        : min=  140, max=  200, avg=170.33, stdev=16.79, samples=15
  lat (msec)   : 20=0.04%, 50=71.37%, 100=28.59%
  cpu          : usr=0.44%, sys=4.18%, ctx=4777, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182168: Sun Sep 13 09:58:18 2026
  read: IOPS=165, BW=662KiB/s (677kB/s)(5060KiB/7648msec)
    slat (usec): min=36, max=11061, avg=3184.00, stdev=1330.76
    clat (usec): min=6880, max=80519, avg=45826.72, stdev=9518.66
     lat (usec): min=6928, max=84871, avg=49013.43, stdev=9871.93
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[32375], 10.00th=[34866], 20.00th=[37487],
     | 30.00th=[40109], 40.00th=[42730], 50.00th=[44827], 60.00th=[47449],
     | 70.00th=[50070], 80.00th=[53216], 90.00th=[58459], 95.00th=[62653],
     | 99.00th=[70779], 99.50th=[76022], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  488, max=  760, per=25.27%, avg=659.80, stdev=78.20, samples=15
   iops        : min=  122, max=  190, avg=164.87, stdev=19.54, samples=15
  write: IOPS=169, BW=677KiB/s (694kB/s)(5180KiB/7648msec); 0 zone resets
    slat (usec): min=29, max=2199, avg=109.92, stdev=112.71
    clat (msec): min=5, max=102, avg=46.35, stdev= 9.71
     lat (msec): min=5, max=102, avg=46.46, stdev= 9.70
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   34], 10.00th=[   37], 20.00th=[   39],
     | 30.00th=[   42], 40.00th=[   43], 50.00th=[   46], 60.00th=[   47],
     | 70.00th=[   50], 80.00th=[   53], 90.00th=[   58], 95.00th=[   64],
     | 99.00th=[   79], 99.50th=[   90], 99.90th=[  102], 99.95th=[  103],
     | 99.99th=[  103]
   bw (  KiB/s): min=  549, max=  814, per=24.86%, avg=675.67, stdev=76.31, samples=15
   iops        : min=  137, max=  203, avg=168.80, stdev=18.99, samples=15
  lat (msec)   : 10=0.16%, 20=0.23%, 50=70.35%, 100=29.18%, 250=0.08%
  cpu          : usr=0.54%, sys=3.99%, ctx=4735, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2612KiB/s (2674kB/s), 649KiB/s-662KiB/s (664kB/s-677kB/s), io=19.6MiB (20.5MB), run=7648-7684msec
  WRITE: bw=2719KiB/s (2784kB/s), 676KiB/s-686KiB/s (692kB/s-703kB/s), io=20.4MiB (21.4MB), run=7648-7684msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182173: Sun Sep 13 09:58:22 2026
  read: IOPS=8, BW=328MiB/s (344MB/s)(40.0MiB/122msec)
    slat (nsec): min=15061k, max=15061k, avg=15061371.00, stdev= 0.00
    clat (nsec): min=98990k, max=98990k, avg=98989742.00, stdev= 0.00
     lat (nsec): min=114056k, max=114056k, avg=114056291.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[99091],  5.00th=[99091], 10.00th=[99091], 20.00th=[99091],
     | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091],
     | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091],
     | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091],
     | 99.99th=[99091]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.09%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=40.0MiB (41.9MB), run=122-122msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout