== sanity test 398c: run fio to test AIO ================= 13:23:45 (1740680625)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.128829 s, 326 MB/s
osc.lustre-OST0000-osc-ffff88013032a800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=9445: Thu Feb 27 13:24:14 2025
  write: IOPS=91, BW=368KiB/s (377kB/s)(10.0MiB/27837msec)
    slat (usec): min=14, max=571, avg=63.23, stdev=38.71
    clat (msec): min=23, max=283, avg=173.78, stdev=35.94
     lat (msec): min=24, max=284, avg=173.85, stdev=35.94
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  114], 10.00th=[  132], 20.00th=[  148],
     | 30.00th=[  157], 40.00th=[  165], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  192], 80.00th=[  203], 90.00th=[  220], 95.00th=[  232],
     | 99.00th=[  259], 99.50th=[  268], 99.90th=[  275], 99.95th=[  279],
     | 99.99th=[  284]
   bw (  KiB/s): min=  264, max=  640, per=25.04%, avg=366.85, stdev=64.47, samples=55
   iops        : min=   66, max=  160, avg=91.64, stdev=16.10, samples=55
  lat (msec)   : 50=0.43%, 100=2.54%, 250=95.12%, 500=1.91%
  cpu          : usr=0.16%, sys=0.63%, ctx=2501, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9446: Thu Feb 27 13:24:14 2025
  write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27952msec)
    slat (usec): min=13, max=363, avg=60.76, stdev=38.36
    clat (msec): min=17, max=284, avg=174.27, stdev=35.74
     lat (msec): min=17, max=284, avg=174.34, stdev=35.74
    clat percentiles (msec):
     |  1.00th=[   91],  5.00th=[  114], 10.00th=[  129], 20.00th=[  148],
     | 30.00th=[  157], 40.00th=[  165], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  192], 80.00th=[  205], 90.00th=[  224], 95.00th=[  234],
     | 99.00th=[  259], 99.50th=[  268], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  284]
   bw (  KiB/s): min=  256, max=  512, per=24.78%, avg=363.07, stdev=58.36, samples=55
   iops        : min=   64, max=  128, avg=90.69, stdev=14.59, samples=55
  lat (msec)   : 20=0.04%, 50=0.08%, 100=1.84%, 250=96.37%, 500=1.68%
  cpu          : usr=0.17%, sys=0.55%, ctx=2345, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9447: Thu Feb 27 13:24:14 2025
  write: IOPS=91, BW=367KiB/s (375kB/s)(10.0MiB/27936msec)
    slat (usec): min=13, max=385, avg=62.90, stdev=37.52
    clat (msec): min=66, max=279, avg=174.31, stdev=34.50
     lat (msec): min=66, max=279, avg=174.37, stdev=34.50
    clat percentiles (msec):
     |  1.00th=[  100],  5.00th=[  115], 10.00th=[  132], 20.00th=[  148],
     | 30.00th=[  157], 40.00th=[  167], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  192], 80.00th=[  203], 90.00th=[  220], 95.00th=[  234],
     | 99.00th=[  255], 99.50th=[  264], 99.90th=[  275], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  264, max=  536, per=24.89%, avg=364.67, stdev=56.18, samples=55
   iops        : min=   66, max=  134, avg=91.09, stdev=14.03, samples=55
  lat (msec)   : 100=2.07%, 250=96.45%, 500=1.48%
  cpu          : usr=0.09%, sys=0.66%, ctx=2524, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9448: Thu Feb 27 13:24:14 2025
  write: IOPS=91, BW=367KiB/s (376kB/s)(10.0MiB/27869msec)
    slat (usec): min=14, max=400, avg=63.39, stdev=35.87
    clat (msec): min=65, max=280, avg=173.93, stdev=34.58
     lat (msec): min=65, max=280, avg=173.99, stdev=34.58
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  114], 10.00th=[  131], 20.00th=[  146],
     | 30.00th=[  157], 40.00th=[  165], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  192], 80.00th=[  203], 90.00th=[  220], 95.00th=[  234],
     | 99.00th=[  253], 99.50th=[  264], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  256, max=  520, per=25.00%, avg=366.29, stdev=56.67, samples=55
   iops        : min=   64, max=  130, avg=91.51, stdev=14.15, samples=55
  lat (msec)   : 100=1.99%, 250=96.60%, 500=1.41%
  cpu          : usr=0.14%, sys=0.64%, ctx=2567, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1465KiB/s (1501kB/s), 366KiB/s-368KiB/s (375kB/s-377kB/s), io=40.0MiB (41.9MB), run=27837-27952msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=9480: Thu Feb 27 13:24:31 2025
   read: IOPS=80, BW=322KiB/s (330kB/s)(5048KiB/15672msec)
    slat (usec): min=641, max=3615, avg=1118.39, stdev=302.34
    clat (msec): min=8, max=160, avg=64.69, stdev=14.60
     lat (msec): min=9, max=161, avg=65.81, stdev=14.67
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   47], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   61], 50.00th=[   63], 60.00th=[   66],
     | 70.00th=[   69], 80.00th=[   74], 90.00th=[   82], 95.00th=[   89],
     | 99.00th=[  115], 99.50th=[  125], 99.90th=[  150], 99.95th=[  161],
     | 99.99th=[  161]
   bw (  KiB/s): min=  224, max=  408, per=25.28%, avg=323.06, stdev=50.25, samples=31
   iops        : min=   56, max=  102, avg=80.74, stdev=12.53, samples=31
  write: IOPS=82, BW=331KiB/s (339kB/s)(5192KiB/15672msec)
    slat (usec): min=15, max=940, avg=57.42, stdev=45.72
    clat (msec): min=68, max=240, avg=128.64, stdev=26.18
     lat (msec): min=68, max=240, avg=128.69, stdev=26.18
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   87], 10.00th=[   96], 20.00th=[  107],
     | 30.00th=[  115], 40.00th=[  122], 50.00th=[  127], 60.00th=[  134],
     | 70.00th=[  142], 80.00th=[  150], 90.00th=[  163], 95.00th=[  176],
     | 99.00th=[  197], 99.50th=[  220], 99.90th=[  241], 99.95th=[  241],
     | 99.99th=[  241]
   bw (  KiB/s): min=  224, max=  392, per=24.89%, avg=331.06, stdev=37.47, samples=31
   iops        : min=   56, max=   98, avg=82.74, stdev= 9.39, samples=31
  lat (msec)   : 10=0.12%, 20=0.12%, 50=4.57%, 100=50.20%, 250=45.00%
  cpu          : usr=0.23%, sys=1.47%, ctx=3640, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9481: Thu Feb 27 13:24:31 2025
   read: IOPS=79, BW=318KiB/s (325kB/s)(4976KiB/15672msec)
    slat (usec): min=49, max=3479, avg=1129.84, stdev=308.63
    clat (msec): min=3, max=177, avg=64.29, stdev=15.65
     lat (msec): min=3, max=178, avg=65.42, stdev=15.73
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   46], 10.00th=[   49], 20.00th=[   54],
     | 30.00th=[   57], 40.00th=[   61], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   69], 80.00th=[   74], 90.00th=[   82], 95.00th=[   89],
     | 99.00th=[  118], 99.50th=[  124], 99.90th=[  150], 99.95th=[  178],
     | 99.99th=[  178]
   bw (  KiB/s): min=  216, max=  424, per=24.88%, avg=317.90, stdev=53.22, samples=31
   iops        : min=   54, max=  106, avg=79.45, stdev=13.30, samples=31
  write: IOPS=83, BW=336KiB/s (344kB/s)(5264KiB/15672msec)
    slat (usec): min=14, max=269, avg=54.39, stdev=26.86
    clat (msec): min=23, max=249, avg=128.44, stdev=26.82
     lat (msec): min=23, max=249, avg=128.50, stdev=26.82
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   87], 10.00th=[   96], 20.00th=[  107],
     | 30.00th=[  114], 40.00th=[  122], 50.00th=[  128], 60.00th=[  134],
     | 70.00th=[  142], 80.00th=[  153], 90.00th=[  163], 95.00th=[  176],
     | 99.00th=[  190], 99.50th=[  209], 99.90th=[  241], 99.95th=[  251],
     | 99.99th=[  251]
   bw (  KiB/s): min=  272, max=  408, per=25.28%, avg=336.23, stdev=30.66, samples=31
   iops        : min=   68, max=  102, avg=84.03, stdev= 7.66, samples=31
  lat (msec)   : 4=0.04%, 10=0.20%, 20=0.16%, 50=5.43%, 100=48.52%
  lat (msec)   : 250=45.66%
  cpu          : usr=0.20%, sys=1.49%, ctx=3588, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9482: Thu Feb 27 13:24:31 2025
   read: IOPS=79, BW=318KiB/s (325kB/s)(4984KiB/15697msec)
    slat (usec): min=643, max=4071, avg=1121.15, stdev=297.90
    clat (msec): min=9, max=117, avg=64.52, stdev=13.15
     lat (msec): min=10, max=119, avg=65.64, stdev=13.23
    clat percentiles (msec):
     |  1.00th=[   38],  5.00th=[   48], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   61], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   70], 80.00th=[   74], 90.00th=[   82], 95.00th=[   87],
     | 99.00th=[  105], 99.50th=[  111], 99.90th=[  116], 99.95th=[  117],
     | 99.99th=[  117]
   bw (  KiB/s): min=  208, max=  464, per=24.85%, avg=317.65, stdev=56.61, samples=31
   iops        : min=   52, max=  116, avg=79.39, stdev=14.15, samples=31
  write: IOPS=83, BW=335KiB/s (343kB/s)(5256KiB/15697msec)
    slat (usec): min=14, max=1473, avg=56.55, stdev=50.73
    clat (msec): min=35, max=243, avg=128.52, stdev=25.93
     lat (msec): min=35, max=243, avg=128.58, stdev=25.93
    clat percentiles (msec):
     |  1.00th=[   80],  5.00th=[   90], 10.00th=[   97], 20.00th=[  108],
     | 30.00th=[  115], 40.00th=[  122], 50.00th=[  128], 60.00th=[  134],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  163], 95.00th=[  176],
     | 99.00th=[  197], 99.50th=[  222], 99.90th=[  232], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  248, max=  408, per=24.95%, avg=331.84, stdev=35.47, samples=31
   iops        : min=   62, max=  102, avg=82.94, stdev= 8.87, samples=31
  lat (msec)   : 10=0.08%, 20=0.27%, 50=3.87%, 100=50.12%, 250=45.66%
  cpu          : usr=0.22%, sys=1.48%, ctx=3797, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9483: Thu Feb 27 13:24:31 2025
   read: IOPS=80, BW=323KiB/s (330kB/s)(5060KiB/15683msec)
    slat (usec): min=624, max=4492, avg=1124.64, stdev=324.59
    clat (msec): min=4, max=121, avg=64.44, stdev=13.54
     lat (msec): min=5, max=123, avg=65.57, stdev=13.62
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   47], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   61], 50.00th=[   64], 60.00th=[   66],
     | 70.00th=[   69], 80.00th=[   74], 90.00th=[   82], 95.00th=[   88],
     | 99.00th=[  113], 99.50th=[  117], 99.90th=[  122], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  232, max=  448, per=25.32%, avg=323.58, stdev=59.33, samples=31
   iops        : min=   58, max=  112, avg=80.87, stdev=14.83, samples=31
  write: IOPS=82, BW=330KiB/s (338kB/s)(5180KiB/15683msec)
    slat (usec): min=15, max=262, avg=56.48, stdev=28.30
    clat (msec): min=46, max=243, avg=129.27, stdev=26.22
     lat (msec): min=46, max=243, avg=129.33, stdev=26.22
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   89], 10.00th=[   96], 20.00th=[  108],
     | 30.00th=[  115], 40.00th=[  123], 50.00th=[  129], 60.00th=[  136],
     | 70.00th=[  142], 80.00th=[  150], 90.00th=[  163], 95.00th=[  174],
     | 99.00th=[  197], 99.50th=[  224], 99.90th=[  243], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  224, max=  392, per=24.70%, avg=328.48, stdev=32.32, samples=31
   iops        : min=   56, max=   98, avg=82.10, stdev= 8.08, samples=31
  lat (msec)   : 10=0.08%, 20=0.20%, 50=4.18%, 100=50.70%, 250=44.84%
  cpu          : usr=0.19%, sys=1.52%, ctx=3896, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1278KiB/s (1309kB/s), 318KiB/s-323KiB/s (325kB/s-330kB/s), io=19.6MiB (20.5MB), run=15672-15697msec
  WRITE: bw=1331KiB/s (1363kB/s), 330KiB/s-336KiB/s (338kB/s-344kB/s), io=20.4MiB (21.4MB), run=15672-15697msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=9500: Thu Feb 27 13:24:33 2025
   read: IOPS=5, BW=227MiB/s (238MB/s)(40.0MiB/176msec)
    slat (nsec): min=5499.2k, max=5499.2k, avg=5499221.00, stdev= 0.00
    clat (nsec): min=163411k, max=163411k, avg=163410630.00, stdev= 0.00
     lat (nsec): min=168919k, max=168919k, avg=168919171.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  163], 10.00th=[  163], 20.00th=[  163],
     | 30.00th=[  163], 40.00th=[  163], 50.00th=[  163], 60.00th=[  163],
     | 70.00th=[  163], 80.00th=[  163], 90.00th=[  163], 95.00th=[  163],
     | 99.00th=[  163], 99.50th=[  163], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=2.86%, ctx=15, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=40.0MiB (41.9MB), run=176-176msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout