== sanity test 398c: run fio to test AIO ================= 21:43:33 (1784943813)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.150104 s, 279 MB/s
osc.lustre-OST0000-osc-ffff8bceca165800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=181813: Fri Jul 24 21:43:44 2026
  write: IOPS=355, BW=1421KiB/s (1455kB/s)(10.0MiB/7208msec); 0 zone resets
    slat (usec): min=39, max=30023, avg=215.91, stdev=602.80
    clat (usec): min=7330, max=82260, avg=44543.02, stdev=9381.00
     lat (usec): min=7550, max=82616, avg=44760.12, stdev=9367.91
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[32637], 10.00th=[34866], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[43254], 60.00th=[45876],
     | 70.00th=[48497], 80.00th=[52167], 90.00th=[57410], 95.00th=[61604],
     | 99.00th=[71828], 99.50th=[72877], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1120, max= 1728, per=25.06%, avg=1417.43, stdev=195.56, samples=14
   iops        : min=  280, max=  432, avg=354.29, stdev=48.85, samples=14
  lat (msec)   : 10=0.20%, 20=0.43%, 50=73.55%, 100=25.82%
  cpu          : usr=0.46%, sys=5.94%, ctx=2770, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181814: Fri Jul 24 21:43:44 2026
  write: IOPS=355, BW=1423KiB/s (1457kB/s)(10.0MiB/7198msec); 0 zone resets
    slat (usec): min=35, max=1349, avg=199.93, stdev=121.64
    clat (usec): min=6601, max=98146, avg=44716.34, stdev=9456.81
     lat (usec): min=6773, max=98242, avg=44917.79, stdev=9473.63
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[32637], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[43254], 60.00th=[45876],
     | 70.00th=[48497], 80.00th=[52167], 90.00th=[57410], 95.00th=[61604],
     | 99.00th=[72877], 99.50th=[76022], 99.90th=[85459], 99.95th=[98042],
     | 99.99th=[98042]
   bw (  KiB/s): min= 1120, max= 1720, per=25.12%, avg=1421.29, stdev=195.22, samples=14
   iops        : min=  280, max=  430, avg=355.29, stdev=48.79, samples=14
  lat (msec)   : 10=0.04%, 20=0.62%, 50=73.59%, 100=25.74%
  cpu          : usr=0.57%, sys=5.31%, ctx=2704, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181815: Fri Jul 24 21:43:44 2026
  write: IOPS=353, BW=1415KiB/s (1449kB/s)(10.0MiB/7237msec); 0 zone resets
    slat (usec): min=48, max=1440, avg=198.25, stdev=116.65
    clat (usec): min=12186, max=77076, avg=44666.80, stdev=9186.87
     lat (usec): min=12570, max=77166, avg=44866.26, stdev=9193.49
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[33162], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[43254], 60.00th=[45876],
     | 70.00th=[48497], 80.00th=[52167], 90.00th=[57410], 95.00th=[62129],
     | 99.00th=[71828], 99.50th=[74974], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1144, max= 1680, per=24.96%, avg=1411.86, stdev=185.81, samples=14
   iops        : min=  286, max=  420, avg=352.93, stdev=46.43, samples=14
  lat (msec)   : 20=0.16%, 50=74.26%, 100=25.59%
  cpu          : usr=0.32%, sys=5.85%, ctx=2671, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181816: Fri Jul 24 21:43:44 2026
  write: IOPS=353, BW=1414KiB/s (1448kB/s)(10.0MiB/7240msec); 0 zone resets
    slat (usec): min=35, max=1557, avg=194.95, stdev=115.06
    clat (usec): min=5281, max=78089, avg=44671.59, stdev=9136.02
     lat (usec): min=5490, max=78236, avg=44868.00, stdev=9148.17
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[33162], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[43254], 60.00th=[45876],
     | 70.00th=[48497], 80.00th=[52167], 90.00th=[57410], 95.00th=[61080],
     | 99.00th=[70779], 99.50th=[73925], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1152, max= 1696, per=24.95%, avg=1411.36, stdev=181.87, samples=14
   iops        : min=  288, max=  424, avg=352.79, stdev=45.44, samples=14
  lat (msec)   : 10=0.12%, 20=0.04%, 50=74.45%, 100=25.39%
  cpu          : usr=0.52%, sys=5.14%, ctx=2754, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5657KiB/s (5793kB/s), 1414KiB/s-1423KiB/s (1448kB/s-1457kB/s), io=40.0MiB (41.9MB), run=7198-7240msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=181826: Fri Jul 24 21:43:54 2026
  read: IOPS=180, BW=723KiB/s (741kB/s)(5048KiB/6980msec)
    slat (usec): min=54, max=7913, avg=2836.68, stdev=988.28
    clat (usec): min=17481, max=70138, avg=41885.44, stdev=7725.96
     lat (usec): min=21111, max=73459, avg=44724.79, stdev=8064.89
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[30278], 10.00th=[32900], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47449], 90.00th=[51119], 95.00th=[56361],
     | 99.00th=[64226], 99.50th=[67634], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  624, max=  888, per=25.03%, avg=717.23, stdev=68.85, samples=13
   iops        : min=  156, max=  222, avg=179.31, stdev=17.21, samples=13
  write: IOPS=185, BW=744KiB/s (762kB/s)(5192KiB/6980msec); 0 zone resets
    slat (usec): min=48, max=1160, avg=173.14, stdev=101.85
    clat (usec): min=11061, max=75258, avg=42223.75, stdev=7240.62
     lat (usec): min=11124, max=75422, avg=42398.06, stdev=7244.22
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[32637], 10.00th=[34341], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[51119], 95.00th=[56361],
     | 99.00th=[62653], 99.50th=[66323], 99.90th=[69731], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  560, max=  824, per=24.77%, avg=738.69, stdev=70.96, samples=13
   iops        : min=  140, max=  206, avg=184.62, stdev=17.78, samples=13
  lat (msec)   : 20=0.27%, 50=87.27%, 100=12.46%
  cpu          : usr=0.60%, sys=5.63%, ctx=4388, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181827: Fri Jul 24 21:43:54 2026
  read: IOPS=177, BW=710KiB/s (728kB/s)(4976KiB/7004msec)
    slat (usec): min=1014, max=9172, avg=2834.34, stdev=936.26
    clat (usec): min=12713, max=77681, avg=42072.74, stdev=7716.47
     lat (usec): min=14599, max=82154, avg=44909.34, stdev=8006.79
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30802], 10.00th=[33162], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47973], 90.00th=[51643], 95.00th=[55313],
     | 99.00th=[64750], 99.50th=[68682], 99.90th=[73925], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  600, max=  840, per=24.34%, avg=697.31, stdev=66.56, samples=13
   iops        : min=  150, max=  210, avg=174.31, stdev=16.67, samples=13
  write: IOPS=187, BW=752KiB/s (770kB/s)(5264KiB/7004msec); 0 zone resets
    slat (usec): min=42, max=3199, avg=168.99, stdev=122.10
    clat (usec): min=7720, max=73148, avg=42108.29, stdev=6972.98
     lat (usec): min=7862, max=73523, avg=42278.48, stdev=6968.95
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[32375], 10.00th=[34341], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[46924], 90.00th=[50594], 95.00th=[54789],
     | 99.00th=[61604], 99.50th=[64750], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  552, max=  872, per=25.13%, avg=749.38, stdev=89.83, samples=13
   iops        : min=  138, max=  218, avg=187.31, stdev=22.49, samples=13
  lat (msec)   : 10=0.04%, 20=0.27%, 50=87.15%, 100=12.54%
  cpu          : usr=0.56%, sys=5.60%, ctx=4406, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181828: Fri Jul 24 21:43:54 2026
  read: IOPS=178, BW=713KiB/s (730kB/s)(4984KiB/6989msec)
    slat (usec): min=1068, max=9159, avg=2830.75, stdev=956.04
    clat (usec): min=4131, max=86859, avg=41565.07, stdev=7506.51
     lat (usec): min=7970, max=91685, avg=44398.18, stdev=7792.29
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30540], 10.00th=[32900], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[51119], 95.00th=[54264],
     | 99.00th=[62653], 99.50th=[67634], 99.90th=[84411], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  595, max=  832, per=24.88%, avg=712.85, stdev=70.19, samples=13
   iops        : min=  148, max=  208, avg=178.15, stdev=17.65, samples=13
  write: IOPS=188, BW=752KiB/s (770kB/s)(5256KiB/6989msec); 0 zone resets
    slat (usec): min=50, max=2151, avg=171.80, stdev=114.27
    clat (usec): min=11394, max=76652, avg=42399.50, stdev=6970.05
     lat (usec): min=11575, max=76820, avg=42572.66, stdev=6977.95
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[32375], 10.00th=[34341], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[45351], 80.00th=[47449], 90.00th=[51643], 95.00th=[54789],
     | 99.00th=[61604], 99.50th=[63701], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  616, max=  992, per=24.79%, avg=739.31, stdev=105.11, samples=13
   iops        : min=  154, max=  248, avg=184.77, stdev=26.33, samples=13
  lat (msec)   : 10=0.04%, 20=0.04%, 50=87.62%, 100=12.30%
  cpu          : usr=0.73%, sys=5.41%, ctx=4456, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181829: Fri Jul 24 21:43:54 2026
  read: IOPS=180, BW=722KiB/s (740kB/s)(5060KiB/7004msec)
    slat (usec): min=1108, max=9453, avg=2865.15, stdev=934.90
    clat (usec): min=7343, max=74747, avg=41517.44, stdev=7328.59
     lat (usec): min=9384, max=78949, avg=44385.03, stdev=7647.45
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[30278], 10.00th=[32637], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44827], 80.00th=[46924], 90.00th=[50070], 95.00th=[54264],
     | 99.00th=[61080], 99.50th=[64750], 99.90th=[72877], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  588, max=  856, per=24.95%, avg=714.77, stdev=76.62, samples=13
   iops        : min=  147, max=  214, avg=178.69, stdev=19.15, samples=13
  write: IOPS=184, BW=740KiB/s (757kB/s)(5180KiB/7004msec); 0 zone resets
    slat (usec): min=48, max=3319, avg=172.77, stdev=132.23
    clat (usec): min=11267, max=68477, avg=42396.93, stdev=7105.00
     lat (usec): min=11412, max=68571, avg=42570.98, stdev=7108.90
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[32900], 10.00th=[34866], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[52167], 95.00th=[55837],
     | 99.00th=[62129], 99.50th=[64226], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  600, max=  880, per=24.51%, avg=730.92, stdev=82.18, samples=13
   iops        : min=  150, max=  220, avg=182.69, stdev=20.60, samples=13
  lat (msec)   : 10=0.04%, 20=0.35%, 50=87.34%, 100=12.27%
  cpu          : usr=0.40%, sys=5.81%, ctx=4482, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2865KiB/s (2934kB/s), 710KiB/s-723KiB/s (728kB/s-741kB/s), io=19.6MiB (20.5MB), run=6980-7004msec
  WRITE: bw=2983KiB/s (3054kB/s), 740KiB/s-752KiB/s (757kB/s-770kB/s), io=20.4MiB (21.4MB), run=6980-7004msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=181834: Fri Jul 24 21:43:58 2026
  read: IOPS=8, BW=333MiB/s (350MB/s)(40.0MiB/120msec)
    slat (nsec): min=9856.2k, max=9856.2k, avg=9856226.00, stdev= 0.00
    clat (nsec): min=104495k, max=104495k, avg=104495349.00, stdev= 0.00
     lat (nsec): min=114355k, max=114355k, avg=114355195.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  105],  5.00th=[  105], 10.00th=[  105], 20.00th=[  105],
     | 30.00th=[  105], 40.00th=[  105], 50.00th=[  105], 60.00th=[  105],
     | 70.00th=[  105], 80.00th=[  105], 90.00th=[  105], 95.00th=[  105],
     | 99.00th=[  105], 99.50th=[  105], 99.90th=[  105], 99.95th=[  105],
     | 99.99th=[  105]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.56%, ctx=10, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=40.0MiB (41.9MB), run=120-120msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout