== sanity test 398c: run fio to test AIO ================= 10:32:35 (1758810755)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.178924 s, 234 MB/s
osc.lustre-OST0000-osc-ffff8a4844c89000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190006: Thu Sep 25 10:32:49 2025
  write: IOPS=261, BW=1048KiB/s (1073kB/s)(10.0MiB/9771msec); 0 zone resets
    slat (usec): min=33, max=2506, avg=134.58, stdev=118.98
    clat (msec): min=9, max=131, avg=60.85, stdev=12.86
     lat (msec): min=9, max=131, avg=60.99, stdev=12.87
    clat percentiles (msec):
     |  1.00th=[   38],  5.00th=[   44], 10.00th=[   46], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   59], 60.00th=[   63],
     | 70.00th=[   67], 80.00th=[   72], 90.00th=[   80], 95.00th=[   83],
     | 99.00th=[   96], 99.50th=[  106], 99.90th=[  118], 99.95th=[  132],
     | 99.99th=[  132]
   bw (  KiB/s): min=  792, max= 1296, per=25.19%, avg=1049.00, stdev=158.00, samples=19
   iops        : min=  198, max=  324, avg=262.05, stdev=39.48, samples=19
  lat (msec)   : 10=0.08%, 20=0.08%, 50=18.36%, 100=80.59%, 250=0.90%
  cpu          : usr=0.56%, sys=2.87%, ctx=2602, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190007: Thu Sep 25 10:32:49 2025
  write: IOPS=260, BW=1041KiB/s (1066kB/s)(10.0MiB/9836msec); 0 zone resets
    slat (usec): min=34, max=1940, avg=115.43, stdev=100.59
    clat (msec): min=7, max=111, avg=60.94, stdev=12.60
     lat (msec): min=8, max=112, avg=61.06, stdev=12.61
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   44], 10.00th=[   46], 20.00th=[   52],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   59], 60.00th=[   62],
     | 70.00th=[   67], 80.00th=[   73], 90.00th=[   79], 95.00th=[   83],
     | 99.00th=[   96], 99.50th=[  102], 99.90th=[  112], 99.95th=[  112],
     | 99.99th=[  112]
   bw (  KiB/s): min=  792, max= 1280, per=25.01%, avg=1041.53, stdev=154.68, samples=19
   iops        : min=  198, max=  320, avg=260.21, stdev=38.69, samples=19
  lat (msec)   : 10=0.04%, 20=0.04%, 50=16.29%, 100=82.97%, 250=0.66%
  cpu          : usr=0.45%, sys=2.63%, ctx=2763, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190008: Thu Sep 25 10:32:49 2025
  write: IOPS=260, BW=1042KiB/s (1067kB/s)(10.0MiB/9828msec); 0 zone resets
    slat (usec): min=33, max=26581, avg=143.16, stdev=536.09
    clat (msec): min=9, max=136, avg=60.90, stdev=12.96
     lat (msec): min=9, max=136, avg=61.05, stdev=12.94
    clat percentiles (msec):
     |  1.00th=[   38],  5.00th=[   44], 10.00th=[   46], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   59], 60.00th=[   63],
     | 70.00th=[   67], 80.00th=[   72], 90.00th=[   80], 95.00th=[   83],
     | 99.00th=[   96], 99.50th=[  106], 99.90th=[  124], 99.95th=[  124],
     | 99.99th=[  138]
   bw (  KiB/s): min=  776, max= 1320, per=25.07%, avg=1043.95, stdev=159.02, samples=19
   iops        : min=  194, max=  330, avg=260.79, stdev=39.75, samples=19
  lat (msec)   : 10=0.08%, 20=0.20%, 50=17.81%, 100=81.05%, 250=0.86%
  cpu          : usr=0.47%, sys=3.14%, ctx=2651, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190009: Thu Sep 25 10:32:49 2025
  write: IOPS=260, BW=1044KiB/s (1069kB/s)(10.0MiB/9812msec); 0 zone resets
    slat (usec): min=33, max=2970, avg=133.69, stdev=118.02
    clat (msec): min=11, max=137, avg=61.02, stdev=13.16
     lat (msec): min=12, max=137, avg=61.16, stdev=13.17
    clat percentiles (msec):
     |  1.00th=[   38],  5.00th=[   44], 10.00th=[   46], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   59], 60.00th=[   63],
     | 70.00th=[   67], 80.00th=[   72], 90.00th=[   80], 95.00th=[   83],
     | 99.00th=[  103], 99.50th=[  111], 99.90th=[  138], 99.95th=[  138],
     | 99.99th=[  138]
   bw (  KiB/s): min=  792, max= 1336, per=25.14%, avg=1046.68, stdev=159.06, samples=19
   iops        : min=  198, max=  334, avg=261.47, stdev=39.76, samples=19
  lat (msec)   : 20=0.04%, 50=18.28%, 100=80.47%, 250=1.21%
  cpu          : usr=0.55%, sys=2.83%, ctx=2680, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=4164KiB/s (4264kB/s), 1041KiB/s-1048KiB/s (1066kB/s-1073kB/s), io=40.0MiB (41.9MB), run=9771-9836msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190020: Thu Sep 25 10:33:02 2025
  read: IOPS=142, BW=569KiB/s (583kB/s)(5048KiB/8868msec)
    slat (usec): min=1552, max=11626, avg=3833.44, stdev=1178.48
    clat (usec): min=11285, max=90459, avg=52707.65, stdev=9333.03
     lat (usec): min=14563, max=95637, avg=56545.88, stdev=9588.56
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[38011], 10.00th=[41681], 20.00th=[45351],
     | 30.00th=[47449], 40.00th=[50070], 50.00th=[52167], 60.00th=[54789],
     | 70.00th=[56886], 80.00th=[60031], 90.00th=[64750], 95.00th=[68682],
     | 99.00th=[78119], 99.50th=[80217], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min=  456, max=  670, per=25.12%, avg=568.12, stdev=60.34, samples=17
   iops        : min=  114, max=  167, avg=141.94, stdev=15.03, samples=17
  write: IOPS=146, BW=585KiB/s (600kB/s)(5192KiB/8868msec); 0 zone resets
    slat (usec): min=36, max=1143, avg=135.02, stdev=90.49
    clat (usec): min=28909, max=88818, avg=53439.49, stdev=8530.57
     lat (usec): min=28983, max=89000, avg=53575.90, stdev=8530.41
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[40109], 10.00th=[43779], 20.00th=[46924],
     | 30.00th=[49021], 40.00th=[51119], 50.00th=[53216], 60.00th=[54789],
     | 70.00th=[57410], 80.00th=[60031], 90.00th=[64226], 95.00th=[67634],
     | 99.00th=[77071], 99.50th=[79168], 99.90th=[85459], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  464, max=  736, per=24.80%, avg=584.12, stdev=70.63, samples=17
   iops        : min=  116, max=  184, avg=145.94, stdev=17.70, samples=17
  lat (msec)   : 20=0.04%, 50=37.50%, 100=62.46%
  cpu          : usr=0.56%, sys=4.91%, ctx=4780, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190021: Thu Sep 25 10:33:02 2025
  read: IOPS=140, BW=563KiB/s (576kB/s)(4976KiB/8846msec)
    slat (usec): min=1369, max=9163, avg=3806.47, stdev=1176.00
    clat (usec): min=10777, max=87888, avg=52887.16, stdev=9139.73
     lat (usec): min=14739, max=93090, avg=56699.21, stdev=9421.53
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[38536], 10.00th=[41157], 20.00th=[45351],
     | 30.00th=[47973], 40.00th=[50594], 50.00th=[52167], 60.00th=[54789],
     | 70.00th=[57410], 80.00th=[60031], 90.00th=[64750], 95.00th=[68682],
     | 99.00th=[74974], 99.50th=[78119], 99.90th=[83362], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min=  432, max=  656, per=24.82%, avg=561.41, stdev=64.77, samples=17
   iops        : min=  108, max=  164, avg=140.24, stdev=16.15, samples=17
  write: IOPS=148, BW=595KiB/s (609kB/s)(5264KiB/8846msec); 0 zone resets
    slat (usec): min=36, max=2777, avg=135.49, stdev=114.80
    clat (usec): min=18943, max=80739, avg=53382.63, stdev=8236.98
     lat (usec): min=19067, max=80913, avg=53519.50, stdev=8244.26
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[40633], 10.00th=[43779], 20.00th=[46924],
     | 30.00th=[49021], 40.00th=[51119], 50.00th=[53216], 60.00th=[55313],
     | 70.00th=[56886], 80.00th=[60031], 90.00th=[64226], 95.00th=[67634],
     | 99.00th=[73925], 99.50th=[74974], 99.90th=[79168], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  487, max=  768, per=25.21%, avg=593.76, stdev=73.48, samples=17
   iops        : min=  121, max=  192, avg=148.29, stdev=18.48, samples=17
  lat (msec)   : 20=0.16%, 50=35.82%, 100=64.02%
  cpu          : usr=0.78%, sys=4.60%, ctx=4803, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190022: Thu Sep 25 10:33:02 2025
  read: IOPS=141, BW=566KiB/s (580kB/s)(4984KiB/8805msec)
    slat (usec): min=23, max=10499, avg=3869.80, stdev=1228.33
    clat (usec): min=6038, max=86756, avg=52597.04, stdev=9864.57
     lat (usec): min=6076, max=90457, avg=56471.51, stdev=10220.97
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[38011], 10.00th=[41157], 20.00th=[44827],
     | 30.00th=[47973], 40.00th=[50070], 50.00th=[52691], 60.00th=[54789],
     | 70.00th=[57410], 80.00th=[60031], 90.00th=[65274], 95.00th=[68682],
     | 99.00th=[76022], 99.50th=[79168], 99.90th=[83362], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  424, max=  688, per=25.06%, avg=566.82, stdev=78.53, samples=17
   iops        : min=  106, max=  172, avg=141.59, stdev=19.64, samples=17
  write: IOPS=149, BW=597KiB/s (611kB/s)(5256KiB/8805msec); 0 zone resets
    slat (usec): min=27, max=2770, avg=136.98, stdev=130.80
    clat (usec): min=8939, max=90961, avg=53359.42, stdev=8649.50
     lat (usec): min=9225, max=91033, avg=53498.27, stdev=8653.52
    clat percentiles (usec):
     |  1.00th=[33424],  5.00th=[40109], 10.00th=[42730], 20.00th=[46400],
     | 30.00th=[49021], 40.00th=[51119], 50.00th=[53216], 60.00th=[54789],
     | 70.00th=[56886], 80.00th=[60556], 90.00th=[64226], 95.00th=[67634],
     | 99.00th=[76022], 99.50th=[79168], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min=  504, max=  680, per=25.24%, avg=594.47, stdev=46.11, samples=17
   iops        : min=  126, max=  170, avg=148.53, stdev=11.57, samples=17
  lat (msec)   : 10=0.16%, 20=0.16%, 50=36.37%, 100=63.32%
  cpu          : usr=0.69%, sys=4.69%, ctx=4759, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190023: Thu Sep 25 10:33:02 2025
  read: IOPS=142, BW=571KiB/s (585kB/s)(5060KiB/8864msec)
    slat (usec): min=1391, max=11250, avg=3859.78, stdev=1175.02
    clat (usec): min=26610, max=82963, avg=52629.67, stdev=9083.92
     lat (usec): min=31125, max=88473, avg=56494.23, stdev=9361.57
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[38011], 10.00th=[41157], 20.00th=[44827],
     | 30.00th=[47973], 40.00th=[50070], 50.00th=[52691], 60.00th=[54789],
     | 70.00th=[56886], 80.00th=[60031], 90.00th=[64226], 95.00th=[68682],
     | 99.00th=[74974], 99.50th=[77071], 99.90th=[82314], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  495, max=  688, per=25.17%, avg=569.29, stdev=52.36, samples=17
   iops        : min=  123, max=  172, avg=142.18, stdev=13.14, samples=17
  write: IOPS=146, BW=584KiB/s (598kB/s)(5180KiB/8864msec); 0 zone resets
    slat (usec): min=35, max=807, avg=134.06, stdev=84.82
    clat (usec): min=29492, max=82241, avg=53640.01, stdev=8369.04
     lat (usec): min=29567, max=82301, avg=53775.35, stdev=8368.09
    clat percentiles (usec):
     |  1.00th=[34341],  5.00th=[40633], 10.00th=[43779], 20.00th=[46924],
     | 30.00th=[49021], 40.00th=[51119], 50.00th=[53740], 60.00th=[55313],
     | 70.00th=[57410], 80.00th=[60556], 90.00th=[64226], 95.00th=[67634],
     | 99.00th=[76022], 99.50th=[78119], 99.90th=[80217], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min=  512, max=  744, per=24.72%, avg=582.24, stdev=64.18, samples=17
   iops        : min=  128, max=  186, avg=145.47, stdev=16.07, samples=17
  lat (msec)   : 50=37.42%, 100=62.58%
  cpu          : usr=0.60%, sys=4.81%, ctx=4774, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2263KiB/s (2317kB/s), 563KiB/s-571KiB/s (576kB/s-585kB/s), io=19.6MiB (20.5MB), run=8805-8868msec
  WRITE: bw=2356KiB/s (2412kB/s), 584KiB/s-597KiB/s (598kB/s-611kB/s), io=20.4MiB (21.4MB), run=8805-8868msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190028: Thu Sep 25 10:33:06 2025
  read: IOPS=8, BW=339MiB/s (355MB/s)(40.0MiB/118msec)
    slat (nsec): min=13728k, max=13728k, avg=13727899.00, stdev= 0.00
    clat (nsec): min=97322k, max=97322k, avg=97321915.00, stdev= 0.00
     lat (nsec): min=111054k, max=111054k, avg=111054331.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[96994],  5.00th=[96994], 10.00th=[96994], 20.00th=[96994],
     | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994],
     | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994],
     | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.26%, ctx=11, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=40.0MiB (41.9MB), run=118-118msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout