== sanity test 398c: run fio to test AIO ================= 20:57:05 (1718931425)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0926663 s, 453 MB/s
osc.lustre-OST0000-osc-ffff8801376ce000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=20001: Thu Jun 20 20:57:33 2024
  write: IOPS=94, BW=378KiB/s (387kB/s)(10.0MiB/27092msec)
    slat (usec): min=12, max=516, avg=64.04, stdev=38.66
    clat (msec): min=19, max=279, avg=169.18, stdev=30.65
     lat (msec): min=19, max=279, avg=169.24, stdev=30.65
    clat percentiles (msec):
     |  1.00th=[   63],  5.00th=[  116], 10.00th=[  133], 20.00th=[  148],
     | 30.00th=[  157], 40.00th=[  165], 50.00th=[  174], 60.00th=[  178],
     | 70.00th=[  184], 80.00th=[  194], 90.00th=[  205], 95.00th=[  215],
     | 99.00th=[  230], 99.50th=[  236], 99.90th=[  268], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  296, max=  670, per=25.03%, avg=376.96, stdev=57.58, samples=54
   iops        : min=   74, max=  167, avg=94.20, stdev=14.36, samples=54
  lat (msec)   : 20=0.04%, 50=0.27%, 100=1.60%, 250=97.89%, 500=0.20%
  cpu          : usr=0.15%, sys=0.72%, ctx=2057, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20002: Thu Jun 20 20:57:33 2024
  write: IOPS=94, BW=377KiB/s (386kB/s)(10.0MiB/27176msec)
    slat (usec): min=13, max=349, avg=63.49, stdev=37.38
    clat (msec): min=64, max=254, avg=169.71, stdev=27.97
     lat (msec): min=64, max=254, avg=169.77, stdev=27.97
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  122], 10.00th=[  136], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  165], 50.00th=[  171], 60.00th=[  178],
     | 70.00th=[  186], 80.00th=[  192], 90.00th=[  203], 95.00th=[  213],
     | 99.00th=[  232], 99.50th=[  236], 99.90th=[  249], 99.95th=[  253],
     | 99.99th=[  255]
   bw (  KiB/s): min=  288, max=  512, per=24.92%, avg=375.26, stdev=42.54, samples=54
   iops        : min=   72, max=  128, avg=93.81, stdev=10.63, samples=54
  lat (msec)   : 100=1.13%, 250=98.79%, 500=0.08%
  cpu          : usr=0.18%, sys=0.65%, ctx=2112, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20003: Thu Jun 20 20:57:33 2024
  write: IOPS=94, BW=377KiB/s (386kB/s)(10.0MiB/27186msec)
    slat (usec): min=12, max=405, avg=64.66, stdev=38.28
    clat (msec): min=36, max=247, avg=169.76, stdev=28.09
     lat (msec): min=36, max=247, avg=169.83, stdev=28.09
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  121], 10.00th=[  136], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  165], 50.00th=[  171], 60.00th=[  178],
     | 70.00th=[  184], 80.00th=[  192], 90.00th=[  205], 95.00th=[  215],
     | 99.00th=[  230], 99.50th=[  236], 99.90th=[  247], 99.95th=[  249],
     | 99.99th=[  249]
   bw (  KiB/s): min=  296, max=  536, per=24.91%, avg=375.11, stdev=43.11, samples=54
   iops        : min=   74, max=  134, avg=93.78, stdev=10.78, samples=54
  lat (msec)   : 50=0.04%, 100=1.17%, 250=98.79%
  cpu          : usr=0.14%, sys=0.67%, ctx=2401, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20004: Thu Jun 20 20:57:33 2024
  write: IOPS=94, BW=377KiB/s (386kB/s)(10.0MiB/27176msec)
    slat (usec): min=12, max=349, avg=65.04, stdev=36.92
    clat (msec): min=51, max=264, avg=169.69, stdev=28.25
     lat (msec): min=51, max=264, avg=169.75, stdev=28.26
    clat percentiles (msec):
     |  1.00th=[   97],  5.00th=[  121], 10.00th=[  136], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  165], 50.00th=[  171], 60.00th=[  178],
     | 70.00th=[  184], 80.00th=[  192], 90.00th=[  205], 95.00th=[  215],
     | 99.00th=[  232], 99.50th=[  236], 99.90th=[  255], 99.95th=[  255],
     | 99.99th=[  266]
   bw (  KiB/s): min=  287, max=  534, per=24.96%, avg=375.85, stdev=41.63, samples=54
   iops        : min=   71, max=  133, avg=93.87, stdev=10.41, samples=54
  lat (msec)   : 100=1.02%, 250=98.79%, 500=0.20%
  cpu          : usr=0.16%, sys=0.68%, ctx=2373, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1507KiB/s (1543kB/s), 377KiB/s-378KiB/s (386kB/s-387kB/s), io=40.0MiB (41.9MB), run=27092-27186msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=20035: Thu Jun 20 20:57:50 2024
   read: IOPS=78, BW=313KiB/s (320kB/s)(5048KiB/16134msec)
    slat (usec): min=447, max=2087, avg=812.59, stdev=204.36
    clat (usec): min=655, max=109555, avg=67767.23, stdev=13055.61
     lat (usec): min=1323, max=110232, avg=68581.63, stdev=13071.71
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   47], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   68], 60.00th=[   71],
     | 70.00th=[   74], 80.00th=[   79], 90.00th=[   84], 95.00th=[   88],
     | 99.00th=[  102], 99.50th=[  106], 99.90th=[  110], 99.95th=[  110],
     | 99.99th=[  110]
   bw (  KiB/s): min=  200, max=  432, per=25.08%, avg=311.69, stdev=55.35, samples=32
   iops        : min=   50, max=  108, avg=77.88, stdev=13.88, samples=32
  write: IOPS=80, BW=322KiB/s (330kB/s)(5192KiB/16134msec)
    slat (usec): min=13, max=231, avg=54.78, stdev=28.94
    clat (msec): min=55, max=235, avg=131.98, stdev=30.21
     lat (msec): min=55, max=235, avg=132.03, stdev=30.21
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[   85], 10.00th=[   93], 20.00th=[  106],
     | 30.00th=[  115], 40.00th=[  124], 50.00th=[  133], 60.00th=[  140],
     | 70.00th=[  148], 80.00th=[  157], 90.00th=[  171], 95.00th=[  186],
     | 99.00th=[  201], 99.50th=[  205], 99.90th=[  226], 99.95th=[  236],
     | 99.99th=[  236]
   bw (  KiB/s): min=  272, max=  408, per=24.71%, avg=319.69, stdev=31.12, samples=32
   iops        : min=   68, max=  102, avg=79.88, stdev= 7.78, samples=32
  lat (usec)   : 750=0.04%
  lat (msec)   : 2=0.04%, 20=0.20%, 50=3.32%, 100=53.24%, 250=43.16%
  cpu          : usr=0.22%, sys=1.42%, ctx=3388, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20036: Thu Jun 20 20:57:50 2024
   read: IOPS=77, BW=308KiB/s (316kB/s)(4976KiB/16133msec)
    slat (usec): min=453, max=3028, avg=816.78, stdev=204.57
    clat (usec): min=816, max=128691, avg=68342.22, stdev=13253.07
     lat (usec): min=1531, max=129404, avg=69160.87, stdev=13264.73
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   47], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   66], 50.00th=[   69], 60.00th=[   72],
     | 70.00th=[   75], 80.00th=[   80], 90.00th=[   85], 95.00th=[   89],
     | 99.00th=[  102], 99.50th=[  104], 99.90th=[  129], 99.95th=[  129],
     | 99.99th=[  129]
   bw (  KiB/s): min=  200, max=  408, per=24.67%, avg=306.69, stdev=51.94, samples=32
   iops        : min=   50, max=  102, avg=76.62, stdev=13.04, samples=32
  write: IOPS=81, BW=326KiB/s (334kB/s)(5264KiB/16133msec)
    slat (usec): min=13, max=351, avg=56.21, stdev=30.85
    clat (msec): min=51, max=234, avg=130.57, stdev=30.11
     lat (msec): min=51, max=234, avg=130.63, stdev=30.11
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   83], 10.00th=[   92], 20.00th=[  105],
     | 30.00th=[  114], 40.00th=[  123], 50.00th=[  131], 60.00th=[  140],
     | 70.00th=[  146], 80.00th=[  157], 90.00th=[  171], 95.00th=[  182],
     | 99.00th=[  201], 99.50th=[  203], 99.90th=[  226], 99.95th=[  234],
     | 99.99th=[  234]
   bw (  KiB/s): min=  256, max=  392, per=25.13%, avg=325.19, stdev=37.13, samples=32
   iops        : min=   64, max=   98, avg=81.25, stdev= 9.26, samples=32
  lat (usec)   : 1000=0.08%
  lat (msec)   : 20=0.20%, 50=2.81%, 100=53.36%, 250=43.55%
  cpu          : usr=0.25%, sys=1.45%, ctx=3150, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20037: Thu Jun 20 20:57:50 2024
   read: IOPS=77, BW=309KiB/s (317kB/s)(4984KiB/16124msec)
    slat (usec): min=22, max=2449, avg=810.11, stdev=209.12
    clat (usec): min=650, max=127089, avg=67630.73, stdev=13697.94
     lat (usec): min=1337, max=127596, avg=68442.74, stdev=13713.99
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   47], 10.00th=[   54], 20.00th=[   59],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   69], 60.00th=[   71],
     | 70.00th=[   74], 80.00th=[   79], 90.00th=[   84], 95.00th=[   89],
     | 99.00th=[  101], 99.50th=[  104], 99.90th=[  110], 99.95th=[  128],
     | 99.99th=[  128]
   bw (  KiB/s): min=  168, max=  472, per=24.92%, avg=309.72, stdev=63.97, samples=32
   iops        : min=   42, max=  118, avg=77.41, stdev=16.01, samples=32
  write: IOPS=81, BW=326KiB/s (334kB/s)(5256KiB/16124msec)
    slat (usec): min=13, max=297, avg=55.03, stdev=31.08
    clat (msec): min=15, max=226, avg=131.28, stdev=30.93
     lat (msec): min=15, max=226, avg=131.33, stdev=30.93
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   83], 10.00th=[   92], 20.00th=[  106],
     | 30.00th=[  115], 40.00th=[  123], 50.00th=[  132], 60.00th=[  140],
     | 70.00th=[  148], 80.00th=[  157], 90.00th=[  171], 95.00th=[  184],
     | 99.00th=[  203], 99.50th=[  205], 99.90th=[  226], 99.95th=[  226],
     | 99.99th=[  226]
   bw (  KiB/s): min=  264, max=  408, per=24.98%, avg=323.22, stdev=43.64, samples=32
   iops        : min=   66, max=  102, avg=80.78, stdev=10.93, samples=32
  lat (usec)   : 750=0.04%, 1000=0.04%
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.04%, 20=0.51%, 50=2.93%
  lat (msec)   : 100=52.27%, 250=43.98%
  cpu          : usr=0.20%, sys=1.44%, ctx=3413, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20038: Thu Jun 20 20:57:50 2024
   read: IOPS=78, BW=314KiB/s (321kB/s)(5060KiB/16134msec)
    slat (usec): min=496, max=3016, avg=820.33, stdev=222.14
    clat (usec): min=1432, max=116004, avg=68079.12, stdev=12996.51
     lat (usec): min=1996, max=116894, avg=68901.26, stdev=13007.35
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   48], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   69], 60.00th=[   71],
     | 70.00th=[   74], 80.00th=[   79], 90.00th=[   84], 95.00th=[   90],
     | 99.00th=[  102], 99.50th=[  107], 99.90th=[  110], 99.95th=[  116],
     | 99.99th=[  116]
   bw (  KiB/s): min=  176, max=  448, per=25.24%, avg=313.69, stdev=66.98, samples=32
   iops        : min=   44, max=  112, avg=78.38, stdev=16.75, samples=32
  write: IOPS=80, BW=321KiB/s (329kB/s)(5180KiB/16134msec)
    slat (usec): min=13, max=253, avg=55.14, stdev=30.13
    clat (msec): min=23, max=222, avg=131.81, stdev=30.06
     lat (msec): min=23, max=222, avg=131.87, stdev=30.06
    clat percentiles (msec):
     |  1.00th=[   66],  5.00th=[   85], 10.00th=[   93], 20.00th=[  107],
     | 30.00th=[  115], 40.00th=[  124], 50.00th=[  132], 60.00th=[  140],
     | 70.00th=[  148], 80.00th=[  159], 90.00th=[  174], 95.00th=[  182],
     | 99.00th=[  203], 99.50th=[  203], 99.90th=[  222], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  232, max=  384, per=24.59%, avg=318.19, stdev=34.80, samples=32
   iops        : min=   58, max=   96, avg=79.50, stdev= 8.73, samples=32
  lat (msec)   : 2=0.08%, 20=0.20%, 50=2.85%, 100=53.05%, 250=43.83%
  cpu          : usr=0.25%, sys=1.42%, ctx=3300, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1244KiB/s (1274kB/s), 308KiB/s-314KiB/s (316kB/s-321kB/s), io=19.6MiB (20.5MB), run=16124-16134msec
  WRITE: bw=1295KiB/s (1326kB/s), 321KiB/s-326KiB/s (329kB/s-334kB/s), io=20.4MiB (21.4MB), run=16124-16134msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=20055: Thu Jun 20 20:57:52 2024
   read: IOPS=13, BW=556MiB/s (583MB/s)(40.0MiB/72msec)
    slat (nsec): min=6761.4k, max=6761.4k, avg=6761412.00, stdev= 0.00
    clat (nsec): min=60571k, max=60571k, avg=60570720.00, stdev= 0.00
     lat (nsec): min=67344k, max=67344k, avg=67344367.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[60556],  5.00th=[60556], 10.00th=[60556], 20.00th=[60556],
     | 30.00th=[60556], 40.00th=[60556], 50.00th=[60556], 60.00th=[60556],
     | 70.00th=[60556], 80.00th=[60556], 90.00th=[60556], 95.00th=[60556],
     | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556],
     | 99.99th=[60556]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.86%, ctx=3, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=556MiB/s (583MB/s), 556MiB/s-556MiB/s (583MB/s-583MB/s), io=40.0MiB (41.9MB), run=72-72msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck