== sanity test 398c: run fio to test AIO ================= 10:53:37 (1784127217)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.138684 s, 302 MB/s
osc.lustre-OST0000-osc-ffff9d70588a3800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=186796: Wed Jul 15 10:53:46 2026
  write: IOPS=474, BW=1897KiB/s (1943kB/s)(10.0MiB/5397msec); 0 zone resets
    slat (usec): min=46, max=912, avg=198.34, stdev=111.25
    clat (usec): min=20577, max=63159, avg=33138.10, stdev=4059.56
     lat (usec): min=20721, max=63292, avg=33337.68, stdev=4067.00
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26346], 10.00th=[28443], 20.00th=[30540],
     | 30.00th=[31589], 40.00th=[32113], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[42730], 99.50th=[45351], 99.90th=[60031], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1784, max= 1988, per=25.07%, avg=1902.50, stdev=65.90, samples=10
   iops        : min=  446, max=  497, avg=475.60, stdev=16.48, samples=10
  lat (msec)   : 50=99.69%, 100=0.31%
  cpu          : usr=0.78%, sys=7.39%, ctx=2708, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186797: Wed Jul 15 10:53:46 2026
  write: IOPS=476, BW=1907KiB/s (1953kB/s)(10.0MiB/5369msec); 0 zone resets
    slat (usec): min=46, max=30967, avg=214.44, stdev=617.44
    clat (usec): min=4357, max=62417, avg=32993.07, stdev=4119.93
     lat (usec): min=4619, max=62605, avg=33208.86, stdev=4101.25
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26084], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32113], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[41681], 99.50th=[43779], 99.90th=[62129], 99.95th=[62129],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1792, max= 2008, per=25.17%, avg=1910.40, stdev=65.98, samples=10
   iops        : min=  448, max=  502, avg=477.60, stdev=16.49, samples=10
  lat (msec)   : 10=0.12%, 20=0.59%, 50=99.14%, 100=0.16%
  cpu          : usr=0.37%, sys=8.08%, ctx=2681, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186798: Wed Jul 15 10:53:46 2026
  write: IOPS=474, BW=1897KiB/s (1943kB/s)(10.0MiB/5397msec); 0 zone resets
    slat (usec): min=47, max=972, avg=188.11, stdev=106.40
    clat (usec): min=20369, max=56594, avg=33119.31, stdev=3774.64
     lat (usec): min=20582, max=56784, avg=33308.55, stdev=3774.54
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26870], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[31589], 40.00th=[32113], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[56361]
   bw (  KiB/s): min= 1784, max= 1976, per=25.01%, avg=1898.30, stdev=60.90, samples=10
   iops        : min=  446, max=  494, avg=474.50, stdev=15.27, samples=10
  lat (msec)   : 50=99.88%, 100=0.12%
  cpu          : usr=0.61%, sys=6.63%, ctx=2631, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186799: Wed Jul 15 10:53:46 2026
  write: IOPS=477, BW=1910KiB/s (1956kB/s)(10.0MiB/5362msec); 0 zone resets
    slat (usec): min=48, max=1470, avg=207.63, stdev=112.51
    clat (usec): min=18219, max=69425, avg=33234.57, stdev=4377.98
     lat (usec): min=18675, max=69546, avg=33443.41, stdev=4373.51
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[26608], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32113], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[47449], 99.50th=[56361], 99.90th=[66323], 99.95th=[66323],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1800, max= 2009, per=25.22%, avg=1914.10, stdev=68.43, samples=10
   iops        : min=  450, max=  502, avg=478.50, stdev=17.07, samples=10
  lat (msec)   : 20=0.27%, 50=99.10%, 100=0.62%
  cpu          : usr=0.58%, sys=7.44%, ctx=2647, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7589KiB/s (7772kB/s), 1897KiB/s-1910KiB/s (1943kB/s-1956kB/s), io=40.0MiB (41.9MB), run=5362-5397msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=186809: Wed Jul 15 10:53:57 2026
  read: IOPS=159, BW=639KiB/s (654kB/s)(5048KiB/7903msec)
    slat (usec): min=42, max=10741, avg=3487.03, stdev=1166.75
    clat (usec): min=26068, max=93820, avg=47341.16, stdev=7708.09
     lat (usec): min=28643, max=95889, avg=50830.47, stdev=7961.46
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[34866], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[53740], 90.00th=[56361], 95.00th=[58983],
     | 99.00th=[64750], 99.50th=[67634], 99.90th=[90702], 99.95th=[93848],
     | 99.99th=[93848]
   bw (  KiB/s): min=  528, max=  744, per=25.46%, avg=642.40, stdev=62.30, samples=15
   iops        : min=  132, max=  186, avg=160.60, stdev=15.57, samples=15
  write: IOPS=164, BW=657KiB/s (673kB/s)(5192KiB/7903msec); 0 zone resets
    slat (usec): min=44, max=1446, avg=191.71, stdev=106.53
    clat (usec): min=15407, max=90135, avg=47630.78, stdev=6988.54
     lat (usec): min=15501, max=90305, avg=47823.94, stdev=6985.97
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[35914], 10.00th=[38011], 20.00th=[41681],
     | 30.00th=[44303], 40.00th=[46400], 50.00th=[48497], 60.00th=[50070],
     | 70.00th=[51643], 80.00th=[53216], 90.00th=[55837], 95.00th=[57934],
     | 99.00th=[61080], 99.50th=[62129], 99.90th=[80217], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min=  552, max=  768, per=25.04%, avg=657.73, stdev=55.95, samples=15
   iops        : min=  138, max=  192, avg=164.40, stdev=14.00, samples=15
  lat (msec)   : 20=0.12%, 50=61.05%, 100=38.83%
  cpu          : usr=0.35%, sys=5.86%, ctx=4369, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186810: Wed Jul 15 10:53:57 2026
  read: IOPS=156, BW=627KiB/s (642kB/s)(4976KiB/7933msec)
    slat (usec): min=1267, max=12965, avg=3477.35, stdev=1179.36
    clat (usec): min=7396, max=69414, avg=47195.86, stdev=7476.05
     lat (usec): min=10039, max=74569, avg=50675.48, stdev=7691.23
    clat percentiles (usec):
     |  1.00th=[30540],  5.00th=[34866], 10.00th=[36963], 20.00th=[40633],
     | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[53740], 90.00th=[56361], 95.00th=[58459],
     | 99.00th=[63177], 99.50th=[65274], 99.90th=[66847], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  536, max=  744, per=24.86%, avg=627.33, stdev=68.35, samples=15
   iops        : min=  134, max=  186, avg=156.80, stdev=17.13, samples=15
  write: IOPS=165, BW=664KiB/s (679kB/s)(5264KiB/7933msec); 0 zone resets
    slat (usec): min=51, max=911, avg=192.02, stdev=103.30
    clat (usec): min=24584, max=65236, avg=47819.82, stdev=6465.56
     lat (usec): min=24726, max=65331, avg=48013.44, stdev=6461.82
    clat percentiles (usec):
     |  1.00th=[32900],  5.00th=[36439], 10.00th=[39060], 20.00th=[42206],
     | 30.00th=[44827], 40.00th=[46924], 50.00th=[47973], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[53216], 90.00th=[55837], 95.00th=[57410],
     | 99.00th=[61604], 99.50th=[64226], 99.90th=[65274], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  560, max=  776, per=25.33%, avg=665.53, stdev=68.65, samples=15
   iops        : min=  140, max=  194, avg=166.33, stdev=17.23, samples=15
  lat (msec)   : 10=0.04%, 50=61.60%, 100=38.36%
  cpu          : usr=0.40%, sys=5.67%, ctx=4437, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186811: Wed Jul 15 10:53:57 2026
  read: IOPS=157, BW=630KiB/s (645kB/s)(4984KiB/7917msec)
    slat (usec): min=1175, max=17736, avg=3523.45, stdev=1223.66
    clat (usec): min=27288, max=83031, avg=47416.32, stdev=7578.65
     lat (usec): min=30271, max=86007, avg=50942.17, stdev=7759.23
    clat percentiles (usec):
     |  1.00th=[30802],  5.00th=[34866], 10.00th=[36963], 20.00th=[40633],
     | 30.00th=[43254], 40.00th=[45876], 50.00th=[47449], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[53740], 90.00th=[56886], 95.00th=[59507],
     | 99.00th=[64226], 99.50th=[65799], 99.90th=[79168], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  525, max=  712, per=25.17%, avg=635.00, stdev=49.40, samples=15
   iops        : min=  131, max=  178, avg=158.73, stdev=12.39, samples=15
  write: IOPS=165, BW=664KiB/s (680kB/s)(5256KiB/7917msec); 0 zone resets
    slat (usec): min=60, max=5724, avg=196.28, stdev=225.02
    clat (usec): min=30016, max=78846, avg=47390.96, stdev=6558.38
     lat (usec): min=30190, max=78961, avg=47588.66, stdev=6552.29
    clat percentiles (usec):
     |  1.00th=[32900],  5.00th=[35914], 10.00th=[38011], 20.00th=[41681],
     | 30.00th=[44303], 40.00th=[45876], 50.00th=[47449], 60.00th=[49546],
     | 70.00th=[51119], 80.00th=[52691], 90.00th=[55313], 95.00th=[57410],
     | 99.00th=[62653], 99.50th=[64226], 99.90th=[77071], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  584, max=  832, per=25.22%, avg=662.40, stdev=77.46, samples=15
   iops        : min=  146, max=  208, avg=165.60, stdev=19.36, samples=15
  lat (msec)   : 50=62.03%, 100=37.97%
  cpu          : usr=0.44%, sys=5.65%, ctx=4517, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186812: Wed Jul 15 10:53:57 2026
  read: IOPS=159, BW=636KiB/s (652kB/s)(5060KiB/7952msec)
    slat (usec): min=1154, max=12061, avg=3480.49, stdev=1186.37
    clat (usec): min=2849, max=69603, avg=47050.22, stdev=7584.99
     lat (usec): min=5590, max=74800, avg=50532.90, stdev=7793.64
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[34866], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[43254], 40.00th=[45351], 50.00th=[46924], 60.00th=[49546],
     | 70.00th=[51119], 80.00th=[53216], 90.00th=[55837], 95.00th=[58459],
     | 99.00th=[63701], 99.50th=[66323], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  506, max=  776, per=25.12%, avg=633.73, stdev=78.22, samples=15
   iops        : min=  126, max=  194, avg=158.40, stdev=19.61, samples=15
  write: IOPS=162, BW=651KiB/s (667kB/s)(5180KiB/7952msec); 0 zone resets
    slat (usec): min=66, max=878, avg=189.53, stdev=102.60
    clat (usec): min=11743, max=70933, avg=47836.67, stdev=7135.73
     lat (usec): min=11827, max=71034, avg=48027.64, stdev=7133.39
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[35390], 10.00th=[38536], 20.00th=[42206],
     | 30.00th=[44827], 40.00th=[46400], 50.00th=[47973], 60.00th=[50070],
     | 70.00th=[51643], 80.00th=[53216], 90.00th=[56886], 95.00th=[58983],
     | 99.00th=[64750], 99.50th=[66323], 99.90th=[69731], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  538, max=  768, per=24.85%, avg=652.93, stdev=63.08, samples=15
   iops        : min=  134, max=  192, avg=163.20, stdev=15.83, samples=15
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.23%, 50=61.09%, 100=38.55%
  cpu          : usr=0.47%, sys=5.63%, ctx=4481, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2524KiB/s (2584kB/s), 627KiB/s-639KiB/s (642kB/s-654kB/s), io=19.6MiB (20.5MB), run=7903-7952msec
  WRITE: bw=2627KiB/s (2690kB/s), 651KiB/s-664KiB/s (667kB/s-680kB/s), io=20.4MiB (21.4MB), run=7903-7952msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=186817: Wed Jul 15 10:54:01 2026
  read: IOPS=7, BW=303MiB/s (318MB/s)(40.0MiB/132msec)
    slat (nsec): min=11570k, max=11570k, avg=11570161.00, stdev= 0.00
    clat (nsec): min=115407k, max=115407k, avg=115406927.00, stdev= 0.00
     lat (nsec): min=126981k, max=126981k, avg=126981316.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  116],  5.00th=[  116], 10.00th=[  116], 20.00th=[  116],
     | 30.00th=[  116], 40.00th=[  116], 50.00th=[  116], 60.00th=[  116],
     | 70.00th=[  116], 80.00th=[  116], 90.00th=[  116], 95.00th=[  116],
     | 99.00th=[  116], 99.50th=[  116], 99.90th=[  116], 99.95th=[  116],
     | 99.99th=[  116]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=8.40%, ctx=9, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=303MiB/s (318MB/s), 303MiB/s-303MiB/s (318MB/s-318MB/s), io=40.0MiB (41.9MB), run=132-132msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout