== sanity test 398c: run fio to test AIO ================= 19:57:46 (1718323066)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.108435 s, 387 MB/s
osc.lustre-OST0000-osc-ffff8800b5cf5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=19789: Thu Jun 13 19:58:05 2024
  write: IOPS=148, BW=595KiB/s (610kB/s)(10.0MiB/17201msec)
    slat (usec): min=12, max=424, avg=51.72, stdev=32.78
    clat (msec): min=31, max=208, avg=107.36, stdev=15.42
     lat (msec): min=31, max=208, avg=107.42, stdev=15.43
    clat percentiles (msec):
     |  1.00th=[   61],  5.00th=[   86], 10.00th=[   94], 20.00th=[   99],
     | 30.00th=[  102], 40.00th=[  105], 50.00th=[  108], 60.00th=[  110],
     | 70.00th=[  113], 80.00th=[  117], 90.00th=[  125], 95.00th=[  130],
     | 99.00th=[  155], 99.50th=[  184], 99.90th=[  192], 99.95th=[  197],
     | 99.99th=[  209]
   bw (  KiB/s): min=  528, max=  792, per=25.06%, avg=595.24, stdev=53.13, samples=34
   iops        : min=  132, max=  198, avg=148.79, stdev=13.27, samples=34
  lat (msec)   : 50=0.08%, 100=26.29%, 250=73.63%
  cpu          : usr=0.17%, sys=0.84%, ctx=1625, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19790: Thu Jun 13 19:58:05 2024
  write: IOPS=149, BW=596KiB/s (611kB/s)(10.0MiB/17170msec)
    slat (usec): min=12, max=436, avg=51.19, stdev=30.64
    clat (msec): min=14, max=211, avg=107.19, stdev=16.61
     lat (msec): min=14, max=211, avg=107.24, stdev=16.61
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   84], 10.00th=[   94], 20.00th=[   99],
     | 30.00th=[  102], 40.00th=[  105], 50.00th=[  108], 60.00th=[  110],
     | 70.00th=[  113], 80.00th=[  116], 90.00th=[  126], 95.00th=[  131],
     | 99.00th=[  159], 99.50th=[  178], 99.90th=[  201], 99.95th=[  211],
     | 99.99th=[  211]
   bw (  KiB/s): min=  512, max=  920, per=25.11%, avg=596.29, stdev=70.70, samples=34
   iops        : min=  128, max=  230, avg=149.03, stdev=17.63, samples=34
  lat (msec)   : 20=0.23%, 50=0.47%, 100=24.69%, 250=74.61%
  cpu          : usr=0.15%, sys=0.87%, ctx=1641, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19791: Thu Jun 13 19:58:05 2024
  write: IOPS=148, BW=594KiB/s (608kB/s)(10.0MiB/17234msec)
    slat (usec): min=12, max=421, avg=53.17, stdev=32.55
    clat (msec): min=30, max=217, avg=107.56, stdev=15.67
     lat (msec): min=30, max=217, avg=107.62, stdev=15.67
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   86], 10.00th=[   94], 20.00th=[   99],
     | 30.00th=[  102], 40.00th=[  105], 50.00th=[  107], 60.00th=[  110],
     | 70.00th=[  113], 80.00th=[  118], 90.00th=[  125], 95.00th=[  131],
     | 99.00th=[  146], 99.50th=[  176], 99.90th=[  207], 99.95th=[  218],
     | 99.99th=[  218]
   bw (  KiB/s): min=  512, max=  808, per=25.00%, avg=593.82, stdev=57.53, samples=34
   iops        : min=  128, max=  202, avg=148.44, stdev=14.37, samples=34
  lat (msec)   : 50=0.27%, 100=25.59%, 250=74.14%
  cpu          : usr=0.20%, sys=0.78%, ctx=1851, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19792: Thu Jun 13 19:58:05 2024
  write: IOPS=148, BW=594KiB/s (608kB/s)(10.0MiB/17244msec)
    slat (usec): min=12, max=615, avg=54.52, stdev=35.15
    clat (msec): min=21, max=209, avg=107.65, stdev=15.78
     lat (msec): min=21, max=209, avg=107.71, stdev=15.78
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   86], 10.00th=[   94], 20.00th=[   99],
     | 30.00th=[  102], 40.00th=[  105], 50.00th=[  107], 60.00th=[  110],
     | 70.00th=[  113], 80.00th=[  117], 90.00th=[  125], 95.00th=[  130],
     | 99.00th=[  157], 99.50th=[  186], 99.90th=[  209], 99.95th=[  209],
     | 99.99th=[  209]
   bw (  KiB/s): min=  472, max=  792, per=24.99%, avg=593.41, stdev=56.46, samples=34
   iops        : min=  118, max=  198, avg=148.29, stdev=14.07, samples=34
  lat (msec)   : 50=0.20%, 100=25.39%, 250=74.41%
  cpu          : usr=0.16%, sys=0.84%, ctx=1913, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2375KiB/s (2432kB/s), 594KiB/s-596KiB/s (608kB/s-611kB/s), io=40.0MiB (41.9MB), run=17170-17244msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19815: Thu Jun 13 19:58:17 2024
   read: IOPS=112, BW=450KiB/s (460kB/s)(5048KiB/11228msec)
    slat (usec): min=697, max=2673, avg=1197.19, stdev=252.21
    clat (msec): min=36, max=120, avg=65.10, stdev= 9.08
     lat (msec): min=38, max=121, avg=66.30, stdev= 9.10
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   52], 10.00th=[   55], 20.00th=[   57],
     | 30.00th=[   60], 40.00th=[   63], 50.00th=[   66], 60.00th=[   68],
     | 70.00th=[   70], 80.00th=[   72], 90.00th=[   77], 95.00th=[   80],
     | 99.00th=[   90], 99.50th=[   93], 99.90th=[  110], 99.95th=[  121],
     | 99.99th=[  121]
   bw (  KiB/s): min=  352, max=  528, per=25.20%, avg=449.36, stdev=49.81, samples=22
   iops        : min=   88, max=  132, avg=112.27, stdev=12.48, samples=22
  write: IOPS=115, BW=462KiB/s (474kB/s)(5192KiB/11228msec)
    slat (usec): min=14, max=1097, avg=51.56, stdev=38.05
    clat (msec): min=23, max=147, avg=73.64, stdev=11.91
     lat (msec): min=23, max=147, avg=73.69, stdev=11.91
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   58], 10.00th=[   62], 20.00th=[   66],
     | 30.00th=[   68], 40.00th=[   70], 50.00th=[   72], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   88], 95.00th=[   95],
     | 99.00th=[  120], 99.50th=[  128], 99.90th=[  148], 99.95th=[  148],
     | 99.99th=[  148]
   bw (  KiB/s): min=  384, max=  536, per=24.95%, avg=463.14, stdev=44.82, samples=22
   iops        : min=   96, max=  134, avg=115.73, stdev=11.20, samples=22
  lat (msec)   : 50=2.11%, 100=96.21%, 250=1.68%
  cpu          : usr=0.21%, sys=1.92%, ctx=3435, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19816: Thu Jun 13 19:58:17 2024
   read: IOPS=110, BW=442KiB/s (453kB/s)(4976KiB/11251msec)
    slat (usec): min=641, max=2963, avg=1208.08, stdev=263.63
    clat (usec): min=1170, max=120868, avg=65280.75, stdev=9438.88
     lat (msec): min=2, max=121, avg=66.49, stdev= 9.46
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   53], 10.00th=[   55], 20.00th=[   57],
     | 30.00th=[   61], 40.00th=[   63], 50.00th=[   66], 60.00th=[   69],
     | 70.00th=[   70], 80.00th=[   73], 90.00th=[   77], 95.00th=[   80],
     | 99.00th=[   88], 99.50th=[   90], 99.90th=[   99], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  352, max=  552, per=24.82%, avg=442.45, stdev=54.44, samples=22
   iops        : min=   88, max=  138, avg=110.55, stdev=13.65, samples=22
  write: IOPS=116, BW=468KiB/s (479kB/s)(5264KiB/11251msec)
    slat (usec): min=13, max=307, avg=51.72, stdev=26.95
    clat (msec): min=21, max=133, avg=73.70, stdev=11.32
     lat (msec): min=21, max=133, avg=73.76, stdev=11.32
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   58], 10.00th=[   62], 20.00th=[   66],
     | 30.00th=[   68], 40.00th=[   70], 50.00th=[   72], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   88], 95.00th=[   95],
     | 99.00th=[  109], 99.50th=[  123], 99.90th=[  130], 99.95th=[  134],
     | 99.99th=[  134]
   bw (  KiB/s): min=  400, max=  576, per=25.17%, avg=467.18, stdev=51.01, samples=22
   iops        : min=  100, max=  144, avg=116.73, stdev=12.72, samples=22
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.04%, 50=1.88%, 100=96.72%
  lat (msec)   : 250=1.29%
  cpu          : usr=0.28%, sys=1.89%, ctx=3159, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19817: Thu Jun 13 19:58:17 2024
   read: IOPS=111, BW=446KiB/s (457kB/s)(4984KiB/11173msec)
    slat (usec): min=17, max=2614, avg=1194.89, stdev=267.09
    clat (msec): min=2, max=140, avg=64.67, stdev=10.54
     lat (msec): min=2, max=141, avg=65.87, stdev=10.60
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   51], 10.00th=[   55], 20.00th=[   57],
     | 30.00th=[   60], 40.00th=[   63], 50.00th=[   65], 60.00th=[   68],
     | 70.00th=[   70], 80.00th=[   72], 90.00th=[   77], 95.00th=[   79],
     | 99.00th=[   89], 99.50th=[   91], 99.90th=[  129], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  296, max=  568, per=25.16%, avg=448.64, stdev=64.70, samples=22
   iops        : min=   74, max=  142, avg=112.09, stdev=16.17, samples=22
  write: IOPS=117, BW=470KiB/s (482kB/s)(5256KiB/11173msec)
    slat (usec): min=13, max=324, avg=53.96, stdev=28.68
    clat (msec): min=16, max=164, avg=73.40, stdev=12.95
     lat (msec): min=16, max=164, avg=73.46, stdev=12.95
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   57], 10.00th=[   61], 20.00th=[   65],
     | 30.00th=[   68], 40.00th=[   70], 50.00th=[   72], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   88], 95.00th=[   95],
     | 99.00th=[  118], 99.50th=[  126], 99.90th=[  155], 99.95th=[  165],
     | 99.99th=[  165]
   bw (  KiB/s): min=  408, max=  560, per=25.33%, avg=470.09, stdev=40.03, samples=22
   iops        : min=  102, max=  140, avg=117.45, stdev=10.01, samples=22
  lat (msec)   : 4=0.08%, 10=0.08%, 20=0.23%, 50=2.42%, 100=95.66%
  lat (msec)   : 250=1.52%
  cpu          : usr=0.30%, sys=1.90%, ctx=3293, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19818: Thu Jun 13 19:58:17 2024
   read: IOPS=112, BW=450KiB/s (461kB/s)(5060KiB/11251msec)
    slat (usec): min=729, max=2577, avg=1205.56, stdev=255.83
    clat (msec): min=2, max=122, avg=65.62, stdev= 9.48
     lat (msec): min=3, max=123, avg=66.83, stdev= 9.49
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   53], 10.00th=[   55], 20.00th=[   58],
     | 30.00th=[   60], 40.00th=[   64], 50.00th=[   66], 60.00th=[   68],
     | 70.00th=[   71], 80.00th=[   73], 90.00th=[   77], 95.00th=[   81],
     | 99.00th=[   89], 99.50th=[   91], 99.90th=[  122], 99.95th=[  123],
     | 99.99th=[  123]
   bw (  KiB/s): min=  352, max=  536, per=25.28%, avg=450.82, stdev=47.47, samples=22
   iops        : min=   88, max=  134, avg=112.68, stdev=11.84, samples=22
  write: IOPS=115, BW=460KiB/s (471kB/s)(5180KiB/11251msec)
    slat (usec): min=14, max=1014, avg=54.51, stdev=40.09
    clat (msec): min=41, max=133, avg=73.46, stdev=11.25
     lat (msec): min=42, max=133, avg=73.52, stdev=11.25
    clat percentiles (msec):
     |  1.00th=[   54],  5.00th=[   58], 10.00th=[   62], 20.00th=[   66],
     | 30.00th=[   68], 40.00th=[   70], 50.00th=[   72], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   88], 95.00th=[   94],
     | 99.00th=[  109], 99.50th=[  117], 99.90th=[  134], 99.95th=[  134],
     | 99.99th=[  134]
   bw (  KiB/s): min=  384, max=  528, per=24.74%, avg=459.23, stdev=36.55, samples=22
   iops        : min=   96, max=  132, avg=114.77, stdev= 9.18, samples=22
  lat (msec)   : 4=0.04%, 50=1.80%, 100=96.91%, 250=1.25%
  cpu          : usr=0.31%, sys=1.88%, ctx=3306, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1784KiB/s (1826kB/s), 442KiB/s-450KiB/s (453kB/s-461kB/s), io=19.6MiB (20.5MB), run=11173-11251msec
  WRITE: bw=1857KiB/s (1901kB/s), 460KiB/s-470KiB/s (471kB/s-482kB/s), io=20.4MiB (21.4MB), run=11173-11251msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=19831: Thu Jun 13 19:58:18 2024
   read: IOPS=11, BW=449MiB/s (471MB/s)(40.0MiB/89msec)
    slat (nsec): min=5227.6k, max=5227.6k, avg=5227635.00, stdev= 0.00
    clat (nsec): min=78690k, max=78690k, avg=78690419.00, stdev= 0.00
     lat (nsec): min=83927k, max=83927k, avg=83926848.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[79168],  5.00th=[79168], 10.00th=[79168], 20.00th=[79168],
     | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168],
     | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168],
     | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.82%, ctx=4, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=40.0MiB (41.9MB), run=89-89msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout