== sanity test 398c: run fio to test AIO ================= 10:00:50 (1747404050)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0929342 s, 451 MB/s
osc.lustre-OST0000-osc-ffff88009ee49000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=24839: Fri May 16 10:01:14 2025
  write: IOPS=112, BW=450KiB/s (461kB/s)(10.0MiB/22742msec)
    slat (usec): min=12, max=323, avg=61.98, stdev=35.75
    clat (msec): min=12, max=243, avg=141.99, stdev=27.15
     lat (msec): min=12, max=243, avg=142.05, stdev=27.15
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   95], 10.00th=[  109], 20.00th=[  125],
     | 30.00th=[  132], 40.00th=[  140], 50.00th=[  144], 60.00th=[  150],
     | 70.00th=[  157], 80.00th=[  163], 90.00th=[  174], 95.00th=[  184],
     | 99.00th=[  201], 99.50th=[  211], 99.90th=[  232], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  360, max=  800, per=25.05%, avg=449.82, stdev=66.18, samples=45
   iops        : min=   90, max=  200, avg=112.38, stdev=16.54, samples=45
  lat (msec)   : 20=0.12%, 50=0.55%, 100=6.05%, 250=93.28%
  cpu          : usr=0.18%, sys=0.80%, ctx=2301, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24840: Fri May 16 10:01:14 2025
  write: IOPS=112, BW=449KiB/s (460kB/s)(10.0MiB/22799msec)
    slat (usec): min=12, max=312, avg=58.90, stdev=34.39
    clat (msec): min=42, max=242, avg=142.34, stdev=25.90
     lat (msec): min=42, max=242, avg=142.40, stdev=25.91
    clat percentiles (msec):
     |  1.00th=[   66],  5.00th=[  101], 10.00th=[  111], 20.00th=[  123],
     | 30.00th=[  131], 40.00th=[  138], 50.00th=[  144], 60.00th=[  150],
     | 70.00th=[  157], 80.00th=[  163], 90.00th=[  174], 95.00th=[  184],
     | 99.00th=[  205], 99.50th=[  213], 99.90th=[  234], 99.95th=[  234],
     | 99.99th=[  243]
   bw (  KiB/s): min=  368, max=  768, per=24.97%, avg=448.53, stdev=63.64, samples=45
   iops        : min=   92, max=  192, avg=112.13, stdev=15.91, samples=45
  lat (msec)   : 50=0.08%, 100=5.08%, 250=94.84%
  cpu          : usr=0.19%, sys=0.77%, ctx=2100, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24841: Fri May 16 10:01:14 2025
  write: IOPS=112, BW=449KiB/s (460kB/s)(10.0MiB/22799msec)
    slat (usec): min=12, max=264, avg=59.57, stdev=33.97
    clat (msec): min=42, max=240, avg=142.35, stdev=26.18
     lat (msec): min=42, max=240, avg=142.41, stdev=26.18
    clat percentiles (msec):
     |  1.00th=[   66],  5.00th=[   96], 10.00th=[  110], 20.00th=[  123],
     | 30.00th=[  131], 40.00th=[  138], 50.00th=[  144], 60.00th=[  150],
     | 70.00th=[  157], 80.00th=[  163], 90.00th=[  174], 95.00th=[  184],
     | 99.00th=[  203], 99.50th=[  215], 99.90th=[  234], 99.95th=[  234],
     | 99.99th=[  241]
   bw (  KiB/s): min=  352, max=  768, per=24.94%, avg=447.96, stdev=63.68, samples=45
   iops        : min=   88, max=  192, avg=111.96, stdev=15.94, samples=45
  lat (msec)   : 50=0.08%, 100=5.94%, 250=93.98%
  cpu          : usr=0.21%, sys=0.73%, ctx=2108, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24842: Fri May 16 10:01:14 2025
  write: IOPS=112, BW=449KiB/s (460kB/s)(10.0MiB/22800msec)
    slat (usec): min=12, max=310, avg=59.95, stdev=36.01
    clat (msec): min=45, max=228, avg=142.35, stdev=26.21
     lat (msec): min=45, max=228, avg=142.41, stdev=26.21
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   95], 10.00th=[  109], 20.00th=[  124],
     | 30.00th=[  132], 40.00th=[  138], 50.00th=[  144], 60.00th=[  150],
     | 70.00th=[  157], 80.00th=[  163], 90.00th=[  174], 95.00th=[  182],
     | 99.00th=[  203], 99.50th=[  207], 99.90th=[  226], 99.95th=[  228],
     | 99.99th=[  230]
   bw (  KiB/s): min=  360, max=  768, per=24.92%, avg=447.53, stdev=63.81, samples=45
   iops        : min=   90, max=  192, avg=111.82, stdev=15.96, samples=45
  lat (msec)   : 50=0.31%, 100=5.94%, 250=93.75%
  cpu          : usr=0.16%, sys=0.77%, ctx=2222, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1796KiB/s (1840kB/s), 449KiB/s-450KiB/s (460kB/s-461kB/s), io=40.0MiB (41.9MB), run=22742-22800msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=24870: Fri May 16 10:01:29 2025
   read: IOPS=91, BW=366KiB/s (375kB/s)(5048KiB/13780msec)
    slat (usec): min=461, max=3851, avg=792.84, stdev=230.50
    clat (usec): min=941, max=142455, avg=57765.95, stdev=12518.77
     lat (usec): min=1967, max=143826, avg=58560.59, stdev=12567.08
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   43], 10.00th=[   45], 20.00th=[   49],
     | 30.00th=[   53], 40.00th=[   55], 50.00th=[   57], 60.00th=[   61],
     | 70.00th=[   63], 80.00th=[   66], 90.00th=[   71], 95.00th=[   75],
     | 99.00th=[  104], 99.50th=[  118], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  240, max=  456, per=25.12%, avg=365.56, stdev=58.72, samples=27
   iops        : min=   60, max=  114, avg=91.33, stdev=14.70, samples=27
  write: IOPS=94, BW=377KiB/s (386kB/s)(5192KiB/13780msec)
    slat (usec): min=12, max=238, avg=48.62, stdev=23.54
    clat (msec): min=53, max=206, avg=112.58, stdev=23.94
     lat (msec): min=53, max=206, avg=112.63, stdev=23.94
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   77], 10.00th=[   84], 20.00th=[   93],
     | 30.00th=[  100], 40.00th=[  106], 50.00th=[  111], 60.00th=[  117],
     | 70.00th=[  125], 80.00th=[  133], 90.00th=[  146], 95.00th=[  157],
     | 99.00th=[  176], 99.50th=[  182], 99.90th=[  205], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=  240, max=  448, per=24.74%, avg=374.74, stdev=44.87, samples=27
   iops        : min=   60, max=  112, avg=93.63, stdev=11.15, samples=27
  lat (usec)   : 1000=0.04%
  lat (msec)   : 10=0.12%, 20=0.16%, 50=11.52%, 100=53.20%, 250=34.96%
  cpu          : usr=0.20%, sys=1.77%, ctx=2809, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24871: Fri May 16 10:01:29 2025
   read: IOPS=90, BW=361KiB/s (370kB/s)(4976KiB/13780msec)
    slat (usec): min=61, max=4189, avg=792.22, stdev=246.20
    clat (usec): min=1303, max=140021, avg=58275.21, stdev=13333.38
     lat (usec): min=1717, max=141325, avg=59069.24, stdev=13400.30
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   42], 10.00th=[   45], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   55], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   68], 90.00th=[   72], 95.00th=[   77],
     | 99.00th=[  108], 99.50th=[  118], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  232, max=  536, per=24.82%, avg=361.15, stdev=73.67, samples=27
   iops        : min=   58, max=  134, avg=90.26, stdev=18.38, samples=27
  write: IOPS=95, BW=382KiB/s (391kB/s)(5264KiB/13780msec)
    slat (usec): min=12, max=271, avg=49.68, stdev=24.37
    clat (msec): min=23, max=208, avg=111.52, stdev=25.27
     lat (msec): min=23, max=208, avg=111.57, stdev=25.27
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   73], 10.00th=[   81], 20.00th=[   91],
     | 30.00th=[   99], 40.00th=[  106], 50.00th=[  111], 60.00th=[  117],
     | 70.00th=[  125], 80.00th=[  133], 90.00th=[  144], 95.00th=[  155],
     | 99.00th=[  171], 99.50th=[  184], 99.90th=[  207], 99.95th=[  209],
     | 99.99th=[  209]
   bw (  KiB/s): min=  288, max=  488, per=25.09%, avg=380.11, stdev=40.85, samples=27
   iops        : min=   72, max=  122, avg=95.00, stdev=10.19, samples=27
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.12%, 20=0.23%, 50=10.94%
  lat (msec)   : 100=53.83%, 250=34.77%
  cpu          : usr=0.23%, sys=1.76%, ctx=2748, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24872: Fri May 16 10:01:29 2025
   read: IOPS=90, BW=361KiB/s (370kB/s)(4984KiB/13788msec)
    slat (usec): min=450, max=3812, avg=790.87, stdev=230.42
    clat (usec): min=818, max=139617, avg=57808.57, stdev=13279.08
     lat (usec): min=1477, max=141014, avg=58601.23, stdev=13342.52
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   42], 10.00th=[   45], 20.00th=[   49],
     | 30.00th=[   53], 40.00th=[   55], 50.00th=[   57], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   67], 90.00th=[   71], 95.00th=[   75],
     | 99.00th=[  105], 99.50th=[  133], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  208, max=  512, per=24.90%, avg=362.26, stdev=72.79, samples=27
   iops        : min=   52, max=  128, avg=90.52, stdev=18.16, samples=27
  write: IOPS=95, BW=381KiB/s (390kB/s)(5256KiB/13788msec)
    slat (usec): min=13, max=210, avg=48.96, stdev=23.19
    clat (msec): min=34, max=216, avg=112.04, stdev=24.35
     lat (msec): min=34, max=216, avg=112.09, stdev=24.35
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   74], 10.00th=[   82], 20.00th=[   93],
     | 30.00th=[  100], 40.00th=[  105], 50.00th=[  111], 60.00th=[  117],
     | 70.00th=[  125], 80.00th=[  134], 90.00th=[  144], 95.00th=[  157],
     | 99.00th=[  171], 99.50th=[  176], 99.90th=[  211], 99.95th=[  215],
     | 99.99th=[  215]
   bw (  KiB/s): min=  272, max=  432, per=24.87%, avg=376.81, stdev=37.70, samples=27
   iops        : min=   68, max=  108, avg=94.15, stdev= 9.44, samples=27
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.04%, 10=0.16%, 20=0.31%, 50=10.86%, 100=53.01%
  lat (msec)   : 250=35.59%
  cpu          : usr=0.25%, sys=1.70%, ctx=2852, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24873: Fri May 16 10:01:29 2025
   read: IOPS=91, BW=368KiB/s (377kB/s)(5060KiB/13753msec)
    slat (usec): min=474, max=3709, avg=793.22, stdev=233.54
    clat (usec): min=1634, max=142436, avg=58237.57, stdev=13255.19
     lat (msec): min=2, max=145, avg=59.03, stdev=13.32
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   42], 10.00th=[   45], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   55], 50.00th=[   57], 60.00th=[   60],
     | 70.00th=[   64], 80.00th=[   67], 90.00th=[   72], 95.00th=[   75],
     | 99.00th=[  110], 99.50th=[  131], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  232, max=  520, per=25.35%, avg=368.85, stdev=76.54, samples=27
   iops        : min=   58, max=  130, avg=92.19, stdev=19.10, samples=27
  write: IOPS=94, BW=377KiB/s (386kB/s)(5180KiB/13753msec)
    slat (usec): min=12, max=263, avg=48.82, stdev=24.75
    clat (msec): min=52, max=216, avg=111.93, stdev=24.64
     lat (msec): min=52, max=216, avg=111.98, stdev=24.64
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   74], 10.00th=[   82], 20.00th=[   92],
     | 30.00th=[   99], 40.00th=[  105], 50.00th=[  110], 60.00th=[  117],
     | 70.00th=[  124], 80.00th=[  133], 90.00th=[  146], 95.00th=[  157],
     | 99.00th=[  171], 99.50th=[  176], 99.90th=[  205], 99.95th=[  218],
     | 99.99th=[  218]
   bw (  KiB/s): min=  248, max=  440, per=24.74%, avg=374.78, stdev=39.35, samples=27
   iops        : min=   62, max=  110, avg=93.67, stdev= 9.83, samples=27
  lat (msec)   : 2=0.04%, 20=0.20%, 50=11.21%, 100=53.83%, 250=34.73%
  cpu          : usr=0.33%, sys=1.67%, ctx=2794, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1455KiB/s (1490kB/s), 361KiB/s-368KiB/s (370kB/s-377kB/s), io=19.6MiB (20.5MB), run=13753-13788msec
  WRITE: bw=1515KiB/s (1552kB/s), 377KiB/s-382KiB/s (386kB/s-391kB/s), io=20.4MiB (21.4MB), run=13753-13788msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=24888: Fri May 16 10:01:31 2025
   read: IOPS=11, BW=471MiB/s (493MB/s)(40.0MiB/85msec)
    slat (nsec): min=6084.4k, max=6084.4k, avg=6084413.00, stdev= 0.00
    clat (nsec): min=74668k, max=74668k, avg=74668329.00, stdev= 0.00
     lat (nsec): min=80762k, max=80762k, avg=80762202.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[74974],  5.00th=[74974], 10.00th=[74974], 20.00th=[74974],
     | 30.00th=[74974], 40.00th=[74974], 50.00th=[74974], 60.00th=[74974],
     | 70.00th=[74974], 80.00th=[74974], 90.00th=[74974], 95.00th=[74974],
     | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.33%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=471MiB/s (493MB/s), 471MiB/s-471MiB/s (493MB/s-493MB/s), io=40.0MiB (41.9MB), run=85-85msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck