== sanity test 398c: run fio to test AIO ================= 19:51:43 (1736988703)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.121017 s, 347 MB/s
osc.lustre-OST0000-osc-ffff88012aac5800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23426: Wed Jan 15 19:52:12 2025
  write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27596msec)
    slat (usec): min=12, max=281, avg=66.47, stdev=37.16
    clat (msec): min=33, max=370, avg=172.30, stdev=31.11
     lat (msec): min=33, max=370, avg=172.37, stdev=31.11
    clat percentiles (msec):
     |  1.00th=[  104],  5.00th=[  123], 10.00th=[  134], 20.00th=[  148],
     | 30.00th=[  157], 40.00th=[  165], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  188], 80.00th=[  199], 90.00th=[  211], 95.00th=[  220],
     | 99.00th=[  239], 99.50th=[  249], 99.90th=[  338], 99.95th=[  351],
     | 99.99th=[  372]
   bw (  KiB/s): min=  288, max=  544, per=25.01%, avg=370.13, stdev=46.80, samples=55
   iops        : min=   72, max=  136, avg=92.49, stdev=11.65, samples=55
  lat (msec)   : 50=0.04%, 100=0.74%, 250=98.75%, 500=0.47%
  cpu          : usr=0.15%, sys=0.72%, ctx=2116, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23427: Wed Jan 15 19:52:12 2025
  write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27654msec)
    slat (usec): min=12, max=426, avg=65.68, stdev=37.50
    clat (msec): min=77, max=323, avg=172.67, stdev=29.43
     lat (msec): min=77, max=324, avg=172.73, stdev=29.44
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  124], 10.00th=[  134], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  190], 80.00th=[  199], 90.00th=[  211], 95.00th=[  220],
     | 99.00th=[  239], 99.50th=[  247], 99.90th=[  292], 99.95th=[  305],
     | 99.99th=[  326]
   bw (  KiB/s): min=  295, max=  488, per=24.96%, avg=369.44, stdev=41.86, samples=55
   iops        : min=   73, max=  122, avg=92.35, stdev=10.49, samples=55
  lat (msec)   : 100=0.35%, 250=99.30%, 500=0.35%
  cpu          : usr=0.16%, sys=0.71%, ctx=2043, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23428: Wed Jan 15 19:52:12 2025
  write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27665msec)
    slat (usec): min=11, max=365, avg=64.00, stdev=37.81
    clat (msec): min=69, max=258, avg=172.75, stdev=28.57
     lat (msec): min=69, max=258, avg=172.82, stdev=28.57
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  124], 10.00th=[  136], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  188], 80.00th=[  199], 90.00th=[  211], 95.00th=[  218],
     | 99.00th=[  236], 99.50th=[  241], 99.90th=[  247], 99.95th=[  249],
     | 99.99th=[  259]
   bw (  KiB/s): min=  288, max=  472, per=24.90%, avg=368.51, stdev=44.49, samples=55
   iops        : min=   72, max=  118, avg=92.07, stdev=11.11, samples=55
  lat (msec)   : 100=0.39%, 250=99.57%, 500=0.04%
  cpu          : usr=0.12%, sys=0.70%, ctx=1954, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23429: Wed Jan 15 19:52:12 2025
  write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27641msec)
    slat (usec): min=12, max=365, avg=65.29, stdev=36.58
    clat (msec): min=53, max=330, avg=172.58, stdev=29.55
     lat (msec): min=53, max=330, avg=172.64, stdev=29.55
    clat percentiles (msec):
     |  1.00th=[  104],  5.00th=[  124], 10.00th=[  136], 20.00th=[  148],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  174], 60.00th=[  182],
     | 70.00th=[  188], 80.00th=[  197], 90.00th=[  209], 95.00th=[  220],
     | 99.00th=[  239], 99.50th=[  245], 99.90th=[  330], 99.95th=[  330],
     | 99.99th=[  330]
   bw (  KiB/s): min=  272, max=  496, per=25.01%, avg=370.18, stdev=42.98, samples=55
   iops        : min=   68, max=  124, avg=92.55, stdev=10.75, samples=55
  lat (msec)   : 100=0.43%, 250=99.30%, 500=0.27%
  cpu          : usr=0.13%, sys=0.74%, ctx=1955, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1481KiB/s (1516kB/s), 370KiB/s-371KiB/s (379kB/s-380kB/s), io=40.0MiB (41.9MB), run=27596-27665msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=23461: Wed Jan 15 19:52:28 2025
   read: IOPS=80, BW=323KiB/s (331kB/s)(5048KiB/15618msec)
    slat (usec): min=28, max=2271, avg=850.24, stdev=207.51
    clat (usec): min=1434, max=133098, avg=66743.81, stdev=13939.11
     lat (usec): min=1571, max=133971, avg=67595.97, stdev=13977.21
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   48], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   61], 40.00th=[   64], 50.00th=[   67], 60.00th=[   69],
     | 70.00th=[   73], 80.00th=[   78], 90.00th=[   84], 95.00th=[   89],
     | 99.00th=[  102], 99.50th=[  114], 99.90th=[  124], 99.95th=[  133],
     | 99.99th=[  133]
   bw (  KiB/s): min=  232, max=  440, per=25.32%, avg=324.06, stdev=51.45, samples=31
   iops        : min=   58, max=  110, avg=80.97, stdev=12.89, samples=31
  write: IOPS=83, BW=332KiB/s (340kB/s)(5192KiB/15618msec)
    slat (usec): min=13, max=300, avg=56.37, stdev=30.96
    clat (msec): min=25, max=236, avg=126.58, stdev=30.22
     lat (msec): min=25, max=237, avg=126.63, stdev=30.22
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   83], 10.00th=[   90], 20.00th=[  101],
     | 30.00th=[  110], 40.00th=[  117], 50.00th=[  126], 60.00th=[  134],
     | 70.00th=[  142], 80.00th=[  150], 90.00th=[  165], 95.00th=[  178],
     | 99.00th=[  211], 99.50th=[  218], 99.90th=[  236], 99.95th=[  236],
     | 99.99th=[  236]
   bw (  KiB/s): min=  232, max=  407, per=24.87%, avg=331.55, stdev=36.75, samples=31
   iops        : min=   58, max=  101, avg=82.84, stdev= 9.17, samples=31
  lat (msec)   : 2=0.08%, 4=0.12%, 10=0.08%, 20=0.23%, 50=2.70%
  lat (msec)   : 100=55.51%, 250=41.29%
  cpu          : usr=0.20%, sys=1.62%, ctx=2992, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23462: Wed Jan 15 19:52:28 2025
   read: IOPS=79, BW=318KiB/s (326kB/s)(4976KiB/15650msec)
    slat (usec): min=524, max=2408, avg=860.17, stdev=214.13
    clat (usec): min=823, max=120682, avg=66671.42, stdev=12962.54
     lat (usec): min=1484, max=122241, avg=67533.43, stdev=12975.42
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   49], 10.00th=[   53], 20.00th=[   57],
     | 30.00th=[   61], 40.00th=[   64], 50.00th=[   67], 60.00th=[   70],
     | 70.00th=[   72], 80.00th=[   77], 90.00th=[   83], 95.00th=[   87],
     | 99.00th=[  103], 99.50th=[  113], 99.90th=[  120], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  192, max=  408, per=24.71%, avg=316.32, stdev=48.35, samples=31
   iops        : min=   48, max=  102, avg=79.03, stdev=12.09, samples=31
  write: IOPS=84, BW=336KiB/s (344kB/s)(5264KiB/15650msec)
    slat (usec): min=12, max=872, avg=55.97, stdev=37.11
    clat (msec): min=57, max=228, avg=126.18, stdev=29.32
     lat (msec): min=57, max=228, avg=126.24, stdev=29.33
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   83], 10.00th=[   91], 20.00th=[  100],
     | 30.00th=[  109], 40.00th=[  116], 50.00th=[  125], 60.00th=[  133],
     | 70.00th=[  142], 80.00th=[  150], 90.00th=[  165], 95.00th=[  178],
     | 99.00th=[  209], 99.50th=[  211], 99.90th=[  220], 99.95th=[  230],
     | 99.99th=[  230]
   bw (  KiB/s): min=  232, max=  423, per=25.16%, avg=335.42, stdev=44.77, samples=31
   iops        : min=   58, max=  105, avg=83.81, stdev=11.19, samples=31
  lat (usec)   : 1000=0.04%
  lat (msec)   : 10=0.08%, 20=0.27%, 50=2.73%, 100=55.43%, 250=41.45%
  cpu          : usr=0.15%, sys=1.65%, ctx=2883, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23463: Wed Jan 15 19:52:28 2025
   read: IOPS=79, BW=318KiB/s (326kB/s)(4984KiB/15669msec)
    slat (usec): min=500, max=2898, avg=866.04, stdev=227.05
    clat (msec): min=10, max=120, avg=66.97, stdev=12.59
     lat (msec): min=11, max=122, avg=67.84, stdev=12.61
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   48], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   67], 60.00th=[   70],
     | 70.00th=[   73], 80.00th=[   77], 90.00th=[   83], 95.00th=[   88],
     | 99.00th=[  102], 99.50th=[  111], 99.90th=[  122], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  224, max=  448, per=24.87%, avg=318.39, stdev=52.96, samples=31
   iops        : min=   56, max=  112, avg=79.55, stdev=13.21, samples=31
  write: IOPS=83, BW=335KiB/s (343kB/s)(5256KiB/15669msec)
    slat (usec): min=12, max=347, avg=57.44, stdev=32.09
    clat (msec): min=40, max=223, avg=126.21, stdev=29.14
     lat (msec): min=40, max=223, avg=126.26, stdev=29.14
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[   83], 10.00th=[   89], 20.00th=[  102],
     | 30.00th=[  110], 40.00th=[  117], 50.00th=[  126], 60.00th=[  133],
     | 70.00th=[  140], 80.00th=[  150], 90.00th=[  163], 95.00th=[  178],
     | 99.00th=[  205], 99.50th=[  211], 99.90th=[  220], 99.95th=[  224],
     | 99.99th=[  224]
   bw (  KiB/s): min=  248, max=  416, per=24.95%, avg=332.58, stdev=43.35, samples=31
   iops        : min=   62, max=  104, avg=83.10, stdev=10.86, samples=31
  lat (msec)   : 20=0.20%, 50=2.97%, 100=54.65%, 250=42.19%
  cpu          : usr=0.18%, sys=1.63%, ctx=2999, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=23464: Wed Jan 15 19:52:28 2025
   read: IOPS=80, BW=324KiB/s (332kB/s)(5060KiB/15618msec)
    slat (usec): min=452, max=2923, avg=866.74, stdev=224.41
    clat (msec): min=8, max=143, avg=66.55, stdev=13.58
     lat (msec): min=9, max=144, avg=67.42, stdev=13.60
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   48], 10.00th=[   53], 20.00th=[   57],
     | 30.00th=[   61], 40.00th=[   64], 50.00th=[   67], 60.00th=[   69],
     | 70.00th=[   72], 80.00th=[   75], 90.00th=[   84], 95.00th=[   89],
     | 99.00th=[  112], 99.50th=[  118], 99.90th=[  142], 99.95th=[  144],
     | 99.99th=[  144]
   bw (  KiB/s): min=  232, max=  416, per=25.40%, avg=325.10, stdev=56.17, samples=31
   iops        : min=   58, max=  104, avg=81.23, stdev=14.04, samples=31
  write: IOPS=82, BW=332KiB/s (340kB/s)(5180KiB/15618msec)
    slat (usec): min=13, max=438, avg=57.85, stdev=32.95
    clat (msec): min=59, max=228, avg=126.79, stdev=29.82
     lat (msec): min=59, max=228, avg=126.85, stdev=29.82
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   82], 10.00th=[   90], 20.00th=[  101],
     | 30.00th=[  110], 40.00th=[  116], 50.00th=[  125], 60.00th=[  134],
     | 70.00th=[  142], 80.00th=[  153], 90.00th=[  165], 95.00th=[  180],
     | 99.00th=[  207], 99.50th=[  211], 99.90th=[  230], 99.95th=[  230],
     | 99.99th=[  230]
   bw (  KiB/s): min=  240, max=  432, per=24.76%, avg=330.00, stdev=44.21, samples=31
   iops        : min=   60, max=  108, avg=82.45, stdev=11.06, samples=31
  lat (msec)   : 10=0.08%, 20=0.31%, 50=3.16%, 100=54.80%, 250=41.64%
  cpu          : usr=0.26%, sys=1.54%, ctx=3112, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1281KiB/s (1311kB/s), 318KiB/s-324KiB/s (326kB/s-332kB/s), io=19.6MiB (20.5MB), run=15618-15669msec
  WRITE: bw=1333KiB/s (1365kB/s), 332KiB/s-336KiB/s (340kB/s-344kB/s), io=20.4MiB (21.4MB), run=15618-15669msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=23480: Wed Jan 15 19:52:30 2025
   read: IOPS=13, BW=541MiB/s (567MB/s)(40.0MiB/74msec)
    slat (nsec): min=10093k, max=10093k, avg=10092560.00, stdev= 0.00
    clat (nsec): min=59512k, max=59512k, avg=59512449.00, stdev= 0.00
     lat (nsec): min=69619k, max=69619k, avg=69619334.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[59507],  5.00th=[59507], 10.00th=[59507], 20.00th=[59507],
     | 30.00th=[59507], 40.00th=[59507], 50.00th=[59507], 60.00th=[59507],
     | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507],
     | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=12.33%, ctx=15, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=541MiB/s (567MB/s), 541MiB/s-541MiB/s (567MB/s-567MB/s), io=40.0MiB (41.9MB), run=74-74msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck