== sanity test 398c: run fio to test AIO ================= 21:13:06 (1782868386)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.11505 s, 365 MB/s
osc.lustre-OST0000-osc-ffff967a910ab000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=194883: Tue Jun 30 21:13:17 2026
  write: IOPS=372, BW=1489KiB/s (1524kB/s)(10.0MiB/6879msec); 0 zone resets
    slat (usec): min=49, max=1175, avg=266.20, stdev=170.02
    clat (usec): min=26235, max=78340, avg=42055.43, stdev=7165.71
     lat (usec): min=26344, max=78852, avg=42323.21, stdev=7205.18
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[32900], 10.00th=[33817], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47973], 90.00th=[51643], 95.00th=[54264],
     | 99.00th=[63701], 99.50th=[68682], 99.90th=[74974], 99.95th=[76022],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1027, max= 1784, per=24.95%, avg=1485.77, stdev=218.96, samples=13
   iops        : min=  256, max=  446, avg=371.38, stdev=54.87, samples=13
  lat (msec)   : 50=84.06%, 100=15.94%
  cpu          : usr=0.76%, sys=7.23%, ctx=2636, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194884: Tue Jun 30 21:13:17 2026
  write: IOPS=372, BW=1490KiB/s (1525kB/s)(10.0MiB/6874msec); 0 zone resets
    slat (usec): min=48, max=1866, avg=273.01, stdev=174.43
    clat (usec): min=11032, max=79452, avg=42097.49, stdev=7205.91
     lat (usec): min=11243, max=79627, avg=42372.26, stdev=7258.46
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[32900], 10.00th=[33817], 20.00th=[35390],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45876], 80.00th=[47973], 90.00th=[51643], 95.00th=[54264],
     | 99.00th=[63177], 99.50th=[66847], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1116, max= 1752, per=25.00%, avg=1488.31, stdev=196.79, samples=13
   iops        : min=  279, max=  438, avg=372.08, stdev=49.20, samples=13
  lat (msec)   : 20=0.04%, 50=85.12%, 100=14.84%
  cpu          : usr=0.41%, sys=8.29%, ctx=2786, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194885: Tue Jun 30 21:13:17 2026
  write: IOPS=375, BW=1501KiB/s (1538kB/s)(10.0MiB/6820msec); 0 zone resets
    slat (usec): min=41, max=1195, avg=298.17, stdev=170.99
    clat (usec): min=18461, max=77864, avg=42246.38, stdev=7622.42
     lat (usec): min=18712, max=78503, avg=42546.19, stdev=7670.83
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[32900], 10.00th=[33817], 20.00th=[35390],
     | 30.00th=[36963], 40.00th=[39584], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47973], 90.00th=[52167], 95.00th=[54789],
     | 99.00th=[68682], 99.50th=[72877], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1231, max= 1776, per=25.20%, avg=1500.23, stdev=188.04, samples=13
   iops        : min=  307, max=  444, avg=375.00, stdev=47.10, samples=13
  lat (msec)   : 20=0.04%, 50=84.77%, 100=15.20%
  cpu          : usr=0.59%, sys=8.42%, ctx=2750, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194886: Tue Jun 30 21:13:17 2026
  write: IOPS=374, BW=1497KiB/s (1533kB/s)(10.0MiB/6839msec); 0 zone resets
    slat (usec): min=59, max=47616, avg=316.64, stdev=950.16
    clat (usec): min=11626, max=78468, avg=41989.07, stdev=7239.95
     lat (usec): min=12091, max=78982, avg=42307.38, stdev=7297.45
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[32637], 10.00th=[33817], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47973], 90.00th=[51643], 95.00th=[53740],
     | 99.00th=[63177], 99.50th=[71828], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1148, max= 1784, per=25.11%, avg=1495.08, stdev=199.63, samples=13
   iops        : min=  287, max=  446, avg=373.77, stdev=49.91, samples=13
  lat (msec)   : 20=0.08%, 50=85.47%, 100=14.45%
  cpu          : usr=0.69%, sys=8.92%, ctx=2770, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5954KiB/s (6097kB/s), 1489KiB/s-1501KiB/s (1524kB/s-1538kB/s), io=40.0MiB (41.9MB), run=6820-6879msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=194897: Tue Jun 30 21:13:27 2026
  read: IOPS=162, BW=649KiB/s (664kB/s)(5048KiB/7780msec)
    slat (usec): min=38, max=10621, avg=3547.73, stdev=1263.61
    clat (usec): min=9223, max=79314, avg=46674.29, stdev=6575.03
     lat (usec): min=13000, max=82415, avg=50223.33, stdev=6672.18
    clat percentiles (usec):
     |  1.00th=[33162],  5.00th=[36439], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[46400], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[54789], 95.00th=[57934],
     | 99.00th=[64226], 99.50th=[67634], 99.90th=[74974], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  528, max=  744, per=25.21%, avg=647.93, stdev=52.00, samples=15
   iops        : min=  132, max=  186, avg=161.93, stdev=13.05, samples=15
  write: IOPS=166, BW=667KiB/s (683kB/s)(5192KiB/7780msec); 0 zone resets
    slat (usec): min=41, max=738, avg=161.96, stdev=88.55
    clat (usec): min=17837, max=72359, avg=46718.29, stdev=5817.12
     lat (usec): min=18106, max=72468, avg=46881.18, stdev=5824.46
    clat percentiles (usec):
     |  1.00th=[34341],  5.00th=[38536], 10.00th=[40109], 20.00th=[42206],
     | 30.00th=[43779], 40.00th=[45351], 50.00th=[46400], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56361],
     | 99.00th=[64750], 99.50th=[66847], 99.90th=[68682], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  605, max=  744, per=25.00%, avg=669.13, stdev=49.65, samples=15
   iops        : min=  151, max=  186, avg=167.27, stdev=12.44, samples=15
  lat (msec)   : 10=0.04%, 20=0.12%, 50=73.79%, 100=26.05%
  cpu          : usr=0.31%, sys=4.86%, ctx=4701, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194898: Tue Jun 30 21:13:27 2026
  read: IOPS=159, BW=637KiB/s (653kB/s)(4976KiB/7806msec)
    slat (usec): min=1036, max=11127, avg=3575.29, stdev=1230.66
    clat (usec): min=9709, max=75029, avg=46399.24, stdev=6785.33
     lat (usec): min=11900, max=78671, avg=49975.89, stdev=6929.46
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[35914], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[46400], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[54264], 95.00th=[56886],
     | 99.00th=[64750], 99.50th=[67634], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  496, max=  720, per=24.68%, avg=634.40, stdev=66.35, samples=15
   iops        : min=  124, max=  180, avg=158.60, stdev=16.59, samples=15
  write: IOPS=168, BW=674KiB/s (691kB/s)(5264KiB/7806msec); 0 zone resets
    slat (usec): min=46, max=3827, avg=163.42, stdev=131.48
    clat (usec): min=9596, max=77565, avg=46582.07, stdev=5909.09
     lat (usec): min=9700, max=77743, avg=46746.41, stdev=5916.81
    clat percentiles (usec):
     |  1.00th=[35390],  5.00th=[38011], 10.00th=[39584], 20.00th=[41681],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54264], 95.00th=[56886],
     | 99.00th=[62653], 99.50th=[64750], 99.90th=[68682], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  528, max=  784, per=25.15%, avg=673.07, stdev=63.63, samples=15
   iops        : min=  132, max=  196, avg=168.27, stdev=15.91, samples=15
  lat (msec)   : 10=0.08%, 20=0.16%, 50=73.20%, 100=26.56%
  cpu          : usr=0.29%, sys=4.91%, ctx=4535, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194899: Tue Jun 30 21:13:27 2026
  read: IOPS=159, BW=639KiB/s (654kB/s)(4984KiB/7803msec)
    slat (usec): min=1094, max=12351, avg=3607.58, stdev=1208.65
    clat (usec): min=14752, max=72508, avg=46322.43, stdev=6826.31
     lat (usec): min=17259, max=75564, avg=49931.28, stdev=6948.63
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[35914], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[55313], 95.00th=[57934],
     | 99.00th=[64226], 99.50th=[67634], 99.90th=[71828], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  470, max=  704, per=24.98%, avg=642.00, stdev=61.21, samples=15
   iops        : min=  117, max=  176, avg=160.47, stdev=15.40, samples=15
  write: IOPS=168, BW=674KiB/s (690kB/s)(5256KiB/7803msec); 0 zone resets
    slat (usec): min=53, max=1833, avg=170.34, stdev=109.46
    clat (usec): min=30013, max=71256, avg=46628.32, stdev=5836.66
     lat (usec): min=30209, max=71430, avg=46799.70, stdev=5841.59
    clat percentiles (usec):
     |  1.00th=[34866],  5.00th=[38011], 10.00th=[39584], 20.00th=[42206],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[46400], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56886],
     | 99.00th=[64750], 99.50th=[66847], 99.90th=[69731], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  552, max=  800, per=25.07%, avg=671.00, stdev=77.45, samples=15
   iops        : min=  138, max=  200, avg=167.73, stdev=19.39, samples=15
  lat (msec)   : 20=0.04%, 50=74.49%, 100=25.47%
  cpu          : usr=0.27%, sys=4.93%, ctx=4724, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194900: Tue Jun 30 21:13:27 2026
  read: IOPS=162, BW=650KiB/s (665kB/s)(5060KiB/7788msec)
    slat (usec): min=1099, max=12126, avg=3583.13, stdev=1210.16
    clat (usec): min=29040, max=70882, avg=46298.53, stdev=6243.97
     lat (usec): min=32043, max=73565, avg=49883.05, stdev=6363.56
    clat percentiles (usec):
     |  1.00th=[32900],  5.00th=[36439], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[42730], 40.00th=[44303], 50.00th=[46400], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[54264], 95.00th=[56886],
     | 99.00th=[62129], 99.50th=[63701], 99.90th=[66847], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  541, max=  728, per=25.33%, avg=651.00, stdev=52.10, samples=15
   iops        : min=  135, max=  182, avg=162.73, stdev=13.06, samples=15
  write: IOPS=166, BW=665KiB/s (681kB/s)(5180KiB/7788msec); 0 zone resets
    slat (usec): min=40, max=800, avg=162.75, stdev=85.09
    clat (usec): min=12397, max=82671, avg=46634.63, stdev=5667.09
     lat (usec): min=12616, max=82792, avg=46798.36, stdev=5668.30
    clat percentiles (usec):
     |  1.00th=[34866],  5.00th=[38011], 10.00th=[40109], 20.00th=[42206],
     | 30.00th=[43779], 40.00th=[44827], 50.00th=[46400], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[53740], 95.00th=[56361],
     | 99.00th=[61604], 99.50th=[63177], 99.90th=[72877], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min=  541, max=  800, per=24.81%, avg=663.80, stdev=59.00, samples=15
   iops        : min=  135, max=  200, avg=165.93, stdev=14.79, samples=15
  lat (msec)   : 20=0.04%, 50=74.10%, 100=25.86%
  cpu          : usr=0.24%, sys=5.07%, ctx=4692, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2571KiB/s (2633kB/s), 637KiB/s-650KiB/s (653kB/s-665kB/s), io=19.6MiB (20.5MB), run=7780-7806msec
  WRITE: bw=2676KiB/s (2741kB/s), 665KiB/s-674KiB/s (681kB/s-691kB/s), io=20.4MiB (21.4MB), run=7780-7806msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=194905: Tue Jun 30 21:13:31 2026
  read: IOPS=5, BW=240MiB/s (251MB/s)(40.0MiB/167msec)
    slat (nsec): min=8920.5k, max=8920.5k, avg=8920549.00, stdev= 0.00
    clat (nsec): min=151987k, max=151987k, avg=151986656.00, stdev= 0.00
     lat (nsec): min=160910k, max=160910k, avg=160909675.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  153],  5.00th=[  153], 10.00th=[  153], 20.00th=[  153],
     | 30.00th=[  153], 40.00th=[  153], 50.00th=[  153], 60.00th=[  153],
     | 70.00th=[  153], 80.00th=[  153], 90.00th=[  153], 95.00th=[  153],
     | 99.00th=[  153], 99.50th=[  153], 99.90th=[  153], 99.95th=[  153],
     | 99.99th=[  153]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.82%, ctx=11, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=240MiB/s (251MB/s), 240MiB/s-240MiB/s (251MB/s-251MB/s), io=40.0MiB (41.9MB), run=167-167msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout