== sanity test 398c: run fio to test AIO ================= 17:53:21 (1786485201)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.129158 s, 325 MB/s
osc.lustre-OST0000-osc-ffff89f5030a5800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=179474: Tue Aug 11 17:53:30 2026
  write: IOPS=459, BW=1839KiB/s (1883kB/s)(10.0MiB/5569msec); 0 zone resets
    slat (usec): min=41, max=869, avg=189.77, stdev=108.16
    clat (usec): min=8123, max=63006, avg=34560.98, stdev=4748.05
     lat (usec): min=8313, max=63132, avg=34752.08, stdev=4757.09
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[25297], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[40633], 95.00th=[41681],
     | 99.00th=[43779], 99.50th=[44827], 99.90th=[56886], 99.95th=[60031],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1656, max= 2184, per=24.93%, avg=1822.40, stdev=155.99, samples=10
   iops        : min=  414, max=  546, avg=455.60, stdev=39.00, samples=10
  lat (msec)   : 10=0.08%, 20=0.59%, 50=99.10%, 100=0.23%
  cpu          : usr=0.54%, sys=6.48%, ctx=2682, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=179475: Tue Aug 11 17:53:30 2026
  write: IOPS=456, BW=1828KiB/s (1871kB/s)(10.0MiB/5603msec); 0 zone resets
    slat (usec): min=46, max=1463, avg=198.42, stdev=112.97
    clat (usec): min=16748, max=55146, avg=34505.93, stdev=4435.39
     lat (usec): min=16971, max=55533, avg=34705.58, stdev=4447.45
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25560], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[43779], 99.50th=[44827], 99.90th=[48497], 99.95th=[52167],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1648, max= 2067, per=24.74%, avg=1808.30, stdev=133.98, samples=10
   iops        : min=  412, max=  516, avg=452.00, stdev=33.33, samples=10
  lat (msec)   : 20=0.31%, 50=99.61%, 100=0.08%
  cpu          : usr=0.50%, sys=7.28%, ctx=2714, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=179476: Tue Aug 11 17:53:30 2026
  write: IOPS=456, BW=1828KiB/s (1871kB/s)(10.0MiB/5603msec); 0 zone resets
    slat (usec): min=39, max=982, avg=189.13, stdev=109.96
    clat (usec): min=6345, max=48576, avg=34489.91, stdev=4415.78
     lat (usec): min=6511, max=48770, avg=34680.21, stdev=4442.90
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[24773], 10.00th=[29492], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[43779], 99.50th=[44827], 99.90th=[45876], 99.95th=[48497],
     | 99.99th=[48497]
   bw (  KiB/s): min= 1656, max= 2022, per=24.74%, avg=1808.60, stdev=120.84, samples=10
   iops        : min=  414, max=  505, avg=452.10, stdev=30.11, samples=10
  lat (msec)   : 10=0.04%, 50=99.96%
  cpu          : usr=0.50%, sys=6.62%, ctx=2580, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=179477: Tue Aug 11 17:53:30 2026
  write: IOPS=457, BW=1832KiB/s (1876kB/s)(10.0MiB/5590msec); 0 zone resets
    slat (usec): min=46, max=25822, avg=211.17, stdev=517.77
    clat (usec): min=9389, max=58424, avg=34444.65, stdev=4702.61
     lat (usec): min=12623, max=58512, avg=34657.15, stdev=4689.06
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[25822], 10.00th=[29492], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1656, max= 2087, per=24.84%, avg=1815.90, stdev=137.06, samples=10
   iops        : min=  414, max=  521, avg=453.90, stdev=34.10, samples=10
  lat (msec)   : 10=0.04%, 20=0.66%, 50=99.06%, 100=0.23%
  cpu          : usr=0.55%, sys=7.55%, ctx=2633, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7310KiB/s (7486kB/s), 1828KiB/s-1839KiB/s (1871kB/s-1883kB/s), io=40.0MiB (41.9MB), run=5569-5603msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=179487: Tue Aug 11 17:53:41 2026
  read: IOPS=163, BW=652KiB/s (668kB/s)(5048KiB/7737msec)
    slat (usec): min=1230, max=22658, avg=3382.57, stdev=1264.61
    clat (usec): min=28274, max=77327, avg=45631.79, stdev=7026.75
     lat (usec): min=30970, max=80956, avg=49015.90, stdev=7325.95
    clat percentiles (usec):
     |  1.00th=[30802],  5.00th=[34341], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54789], 95.00th=[56886],
     | 99.00th=[64226], 99.50th=[66847], 99.90th=[76022], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  542, max=  792, per=25.21%, avg=653.20, stdev=70.53, samples=15
   iops        : min=  135, max=  198, avg=163.27, stdev=17.69, samples=15
  write: IOPS=167, BW=671KiB/s (687kB/s)(5192KiB/7737msec); 0 zone resets
    slat (usec): min=42, max=2915, avg=186.04, stdev=126.00
    clat (usec): min=7818, max=83463, avg=46559.69, stdev=6754.61
     lat (usec): min=7929, max=83604, avg=46747.02, stdev=6760.95
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[36439], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[52167], 90.00th=[54789], 95.00th=[57934],
     | 99.00th=[65274], 99.50th=[67634], 99.90th=[79168], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  535, max=  808, per=25.00%, avg=674.60, stdev=83.46, samples=15
   iops        : min=  133, max=  202, avg=168.60, stdev=20.96, samples=15
  lat (msec)   : 10=0.04%, 50=73.67%, 100=26.29%
  cpu          : usr=0.40%, sys=5.44%, ctx=4684, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=179488: Tue Aug 11 17:53:41 2026
  read: IOPS=160, BW=643KiB/s (658kB/s)(4976KiB/7743msec)
    slat (usec): min=1159, max=12621, avg=3395.54, stdev=1160.10
    clat (usec): min=5774, max=78104, avg=46087.97, stdev=7774.20
     lat (usec): min=9824, max=82066, avg=49485.34, stdev=8036.62
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[34341], 10.00th=[36439], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[52167], 90.00th=[55837], 95.00th=[58983],
     | 99.00th=[65799], 99.50th=[71828], 99.90th=[76022], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  528, max=  840, per=24.76%, avg=641.47, stdev=89.15, samples=15
   iops        : min=  132, max=  210, avg=160.33, stdev=22.33, samples=15
  write: IOPS=169, BW=680KiB/s (696kB/s)(5264KiB/7743msec); 0 zone resets
    slat (usec): min=45, max=715, avg=176.26, stdev=91.71
    clat (usec): min=27578, max=73543, avg=46530.35, stdev=6889.93
     lat (usec): min=27667, max=73734, avg=46707.83, stdev=6889.63
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[35914], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[46400], 60.00th=[48497],
     | 70.00th=[50070], 80.00th=[52167], 90.00th=[55313], 95.00th=[57410],
     | 99.00th=[65274], 99.50th=[68682], 99.90th=[71828], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  561, max=  808, per=25.32%, avg=683.27, stdev=74.18, samples=15
   iops        : min=  140, max=  202, avg=170.80, stdev=18.57, samples=15
  lat (msec)   : 10=0.04%, 20=0.04%, 50=70.08%, 100=29.84%
  cpu          : usr=0.41%, sys=5.37%, ctx=4726, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=179489: Tue Aug 11 17:53:41 2026
  read: IOPS=160, BW=644KiB/s (659kB/s)(4984KiB/7743msec)
    slat (usec): min=1267, max=22470, avg=3416.70, stdev=1307.12
    clat (usec): min=2122, max=74731, avg=46049.85, stdev=7698.36
     lat (usec): min=5003, max=80564, avg=49468.16, stdev=7979.15
    clat percentiles (usec):
     |  1.00th=[30802],  5.00th=[34866], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[52167], 90.00th=[55837], 95.00th=[58459],
     | 99.00th=[65274], 99.50th=[68682], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  491, max=  784, per=24.94%, avg=646.07, stdev=70.79, samples=15
   iops        : min=  122, max=  196, avg=161.47, stdev=17.82, samples=15
  write: IOPS=169, BW=679KiB/s (695kB/s)(5256KiB/7743msec); 0 zone resets
    slat (usec): min=46, max=1013, avg=182.39, stdev=99.49
    clat (usec): min=15677, max=71088, avg=46189.25, stdev=6692.38
     lat (usec): min=15845, max=71354, avg=46372.74, stdev=6690.57
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[35914], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[55313], 95.00th=[57410],
     | 99.00th=[61604], 99.50th=[63177], 99.90th=[67634], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  536, max=  936, per=25.06%, avg=676.20, stdev=97.59, samples=15
   iops        : min=  134, max=  234, avg=169.00, stdev=24.46, samples=15
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.08%, 50=72.66%, 100=27.15%
  cpu          : usr=0.26%, sys=5.57%, ctx=4722, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=179490: Tue Aug 11 17:53:41 2026
  read: IOPS=164, BW=660KiB/s (676kB/s)(5060KiB/7669msec)
    slat (usec): min=69, max=12667, avg=3407.14, stdev=1184.89
    clat (usec): min=8199, max=78595, avg=45804.25, stdev=7773.33
     lat (usec): min=8453, max=83007, avg=49213.10, stdev=8069.10
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[34341], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[52167], 90.00th=[55313], 95.00th=[58459],
     | 99.00th=[65799], 99.50th=[67634], 99.90th=[77071], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  584, max=  776, per=25.48%, avg=660.20, stdev=66.99, samples=15
   iops        : min=  146, max=  194, avg=165.00, stdev=16.80, samples=15
  write: IOPS=168, BW=675KiB/s (692kB/s)(5180KiB/7669msec); 0 zone resets
    slat (usec): min=63, max=4188, avg=183.92, stdev=144.90
    clat (usec): min=27723, max=89007, avg=46292.07, stdev=6743.02
     lat (usec): min=27872, max=89247, avg=46477.12, stdev=6747.26
    clat percentiles (usec):
     |  1.00th=[32637],  5.00th=[35914], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54789], 95.00th=[57410],
     | 99.00th=[65274], 99.50th=[71828], 99.90th=[88605], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  568, max=  816, per=25.07%, avg=676.33, stdev=72.70, samples=15
   iops        : min=  142, max=  204, avg=169.07, stdev=18.19, samples=15
  lat (msec)   : 10=0.04%, 20=0.16%, 50=73.09%, 100=26.72%
  cpu          : usr=0.34%, sys=5.56%, ctx=4751, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2592KiB/s (2654kB/s), 643KiB/s-660KiB/s (658kB/s-676kB/s), io=19.6MiB (20.5MB), run=7669-7743msec
  WRITE: bw=2698KiB/s (2763kB/s), 671KiB/s-680KiB/s (687kB/s-696kB/s), io=20.4MiB (21.4MB), run=7669-7743msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=179495: Tue Aug 11 17:53:45 2026
  read: IOPS=8, BW=323MiB/s (338MB/s)(40.0MiB/124msec)
    slat (nsec): min=9678.5k, max=9678.5k, avg=9678488.00, stdev= 0.00
    clat (nsec): min=108144k, max=108144k, avg=108143760.00, stdev= 0.00
     lat (nsec): min=117826k, max=117826k, avg=117826308.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  109],  5.00th=[  109], 10.00th=[  109], 20.00th=[  109],
     | 30.00th=[  109], 40.00th=[  109], 50.00th=[  109], 60.00th=[  109],
     | 70.00th=[  109], 80.00th=[  109], 90.00th=[  109], 95.00th=[  109],
     | 99.00th=[  109], 99.50th=[  109], 99.90th=[  109], 99.95th=[  109],
     | 99.99th=[  109]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.32%, ctx=8, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=323MiB/s (338MB/s), 323MiB/s-323MiB/s (338MB/s-338MB/s), io=40.0MiB (41.9MB), run=124-124msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout