== sanity test 398c: run fio to test AIO ================= 01:35:32 (1721885732)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.107138 s, 391 MB/s
osc.lustre-OST0000-osc-ffff880136447000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=24324: Thu Jul 25 01:36:09 2024
  write: IOPS=72, BW=289KiB/s (296kB/s)(10.0MiB/35398msec)
    slat (usec): min=14, max=680, avg=103.39, stdev=55.83
    clat (msec): min=40, max=413, avg=220.89, stdev=44.65
     lat (msec): min=40, max=413, avg=220.99, stdev=44.66
    clat percentiles (msec):
     |  1.00th=[  118],  5.00th=[  148], 10.00th=[  165], 20.00th=[  186],
     | 30.00th=[  199], 40.00th=[  209], 50.00th=[  220], 60.00th=[  230],
     | 70.00th=[  245], 80.00th=[  259], 90.00th=[  279], 95.00th=[  296],
     | 99.00th=[  321], 99.50th=[  330], 99.90th=[  401], 99.95th=[  401],
     | 99.99th=[  414]
   bw (  KiB/s): min=  208, max=  464, per=25.02%, avg=289.20, stdev=51.72, samples=70
   iops        : min=   52, max=  116, avg=72.17, stdev=12.93, samples=70
  lat (msec)   : 50=0.20%, 100=0.27%, 250=73.98%, 500=25.55%
  cpu          : usr=0.11%, sys=0.89%, ctx=2994, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24325: Thu Jul 25 01:36:09 2024
  write: IOPS=72, BW=289KiB/s (296kB/s)(10.0MiB/35413msec)
    slat (usec): min=13, max=591, avg=101.43, stdev=56.37
    clat (msec): min=91, max=351, avg=220.87, stdev=43.29
     lat (msec): min=91, max=351, avg=220.97, stdev=43.30
    clat percentiles (msec):
     |  1.00th=[  120],  5.00th=[  150], 10.00th=[  169], 20.00th=[  186],
     | 30.00th=[  197], 40.00th=[  209], 50.00th=[  220], 60.00th=[  230],
     | 70.00th=[  243], 80.00th=[  259], 90.00th=[  279], 95.00th=[  292],
     | 99.00th=[  317], 99.50th=[  321], 99.90th=[  338], 99.95th=[  347],
     | 99.99th=[  351]
   bw (  KiB/s): min=  200, max=  408, per=24.94%, avg=288.29, stdev=50.31, samples=70
   iops        : min=   50, max=  102, avg=71.94, stdev=12.59, samples=70
  lat (msec)   : 100=0.62%, 250=73.48%, 500=25.90%
  cpu          : usr=0.12%, sys=0.88%, ctx=2874, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24326: Thu Jul 25 01:36:09 2024
  write: IOPS=72, BW=289KiB/s (296kB/s)(10.0MiB/35424msec)
    slat (usec): min=14, max=653, avg=101.55, stdev=57.16
    clat (msec): min=58, max=339, avg=220.86, stdev=42.73
     lat (msec): min=58, max=339, avg=220.97, stdev=42.73
    clat percentiles (msec):
     |  1.00th=[  123],  5.00th=[  148], 10.00th=[  167], 20.00th=[  184],
     | 30.00th=[  199], 40.00th=[  209], 50.00th=[  220], 60.00th=[  230],
     | 70.00th=[  243], 80.00th=[  259], 90.00th=[  279], 95.00th=[  292],
     | 99.00th=[  317], 99.50th=[  326], 99.90th=[  334], 99.95th=[  338],
     | 99.99th=[  338]
   bw (  KiB/s): min=  208, max=  440, per=24.91%, avg=287.99, stdev=48.34, samples=70
   iops        : min=   52, max=  110, avg=71.90, stdev=12.05, samples=70
  lat (msec)   : 100=0.12%, 250=74.53%, 500=25.35%
  cpu          : usr=0.12%, sys=0.88%, ctx=2877, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24327: Thu Jul 25 01:36:09 2024
  write: IOPS=72, BW=290KiB/s (296kB/s)(10.0MiB/35366msec)
    slat (usec): min=12, max=681, avg=101.25, stdev=55.28
    clat (msec): min=91, max=452, avg=220.53, stdev=44.43
     lat (msec): min=91, max=452, avg=220.63, stdev=44.44
    clat percentiles (msec):
     |  1.00th=[  118],  5.00th=[  148], 10.00th=[  167], 20.00th=[  186],
     | 30.00th=[  197], 40.00th=[  209], 50.00th=[  220], 60.00th=[  230],
     | 70.00th=[  243], 80.00th=[  257], 90.00th=[  279], 95.00th=[  292],
     | 99.00th=[  326], 99.50th=[  330], 99.90th=[  439], 99.95th=[  439],
     | 99.99th=[  451]
   bw (  KiB/s): min=  184, max=  423, per=25.04%, avg=289.43, stdev=49.77, samples=70
   iops        : min=   46, max=  105, avg=72.23, stdev=12.41, samples=70
  lat (msec)   : 100=0.62%, 250=73.83%, 500=25.55%
  cpu          : usr=0.14%, sys=0.84%, ctx=2885, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1156KiB/s (1184kB/s), 289KiB/s-290KiB/s (296kB/s-296kB/s), io=40.0MiB (41.9MB), run=35366-35424msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=24365: Thu Jul 25 01:36:30 2024
   read: IOPS=63, BW=255KiB/s (261kB/s)(5048KiB/19797msec)
    slat (usec): min=614, max=3781, avg=1341.07, stdev=401.80
    clat (msec): min=2, max=163, avg=81.02, stdev=17.67
     lat (msec): min=3, max=164, avg=82.36, stdev=17.80
    clat percentiles (msec):
     |  1.00th=[   23],  5.00th=[   56], 10.00th=[   63], 20.00th=[   68],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   81], 60.00th=[   84],
     | 70.00th=[   90], 80.00th=[   94], 90.00th=[  102], 95.00th=[  110],
     | 99.00th=[  128], 99.50th=[  136], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  152, max=  351, per=25.10%, avg=254.05, stdev=56.50, samples=39
   iops        : min=   38, max=   87, avg=63.44, stdev=14.06, samples=39
  write: IOPS=65, BW=262KiB/s (269kB/s)(5192KiB/19797msec)
    slat (usec): min=13, max=399, avg=68.44, stdev=41.93
    clat (msec): min=63, max=279, avg=163.34, stdev=35.98
     lat (msec): min=63, max=280, avg=163.41, stdev=35.99
    clat percentiles (msec):
     |  1.00th=[   93],  5.00th=[  109], 10.00th=[  117], 20.00th=[  132],
     | 30.00th=[  144], 40.00th=[  153], 50.00th=[  163], 60.00th=[  171],
     | 70.00th=[  182], 80.00th=[  192], 90.00th=[  211], 95.00th=[  226],
     | 99.00th=[  257], 99.50th=[  268], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  176, max=  328, per=24.75%, avg=260.82, stdev=37.92, samples=39
   iops        : min=   44, max=   82, avg=65.13, stdev= 9.46, samples=39
  lat (msec)   : 4=0.12%, 10=0.04%, 20=0.31%, 50=0.74%, 100=43.36%
  lat (msec)   : 250=54.61%, 500=0.82%
  cpu          : usr=0.23%, sys=1.65%, ctx=2972, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24366: Thu Jul 25 01:36:30 2024
   read: IOPS=63, BW=252KiB/s (258kB/s)(4976KiB/19735msec)
    slat (usec): min=67, max=3895, avg=1337.40, stdev=412.00
    clat (msec): min=3, max=167, avg=81.08, stdev=17.65
     lat (msec): min=3, max=168, avg=82.42, stdev=17.79
    clat percentiles (msec):
     |  1.00th=[   24],  5.00th=[   56], 10.00th=[   63], 20.00th=[   68],
     | 30.00th=[   73], 40.00th=[   78], 50.00th=[   81], 60.00th=[   85],
     | 70.00th=[   89], 80.00th=[   94], 90.00th=[  103], 95.00th=[  111],
     | 99.00th=[  128], 99.50th=[  138], 99.90th=[  167], 99.95th=[  167],
     | 99.99th=[  167]
   bw (  KiB/s): min=  128, max=  392, per=24.76%, avg=250.56, stdev=57.77, samples=39
   iops        : min=   32, max=   98, avg=62.56, stdev=14.48, samples=39
  write: IOPS=66, BW=267KiB/s (273kB/s)(5264KiB/19735msec)
    slat (usec): min=14, max=904, avg=69.54, stdev=49.00
    clat (msec): min=31, max=280, avg=161.78, stdev=37.61
     lat (msec): min=31, max=281, avg=161.85, stdev=37.62
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[  107], 10.00th=[  116], 20.00th=[  130],
     | 30.00th=[  142], 40.00th=[  153], 50.00th=[  161], 60.00th=[  169],
     | 70.00th=[  180], 80.00th=[  192], 90.00th=[  209], 95.00th=[  230],
     | 99.00th=[  257], 99.50th=[  264], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  167, max=  360, per=25.23%, avg=265.95, stdev=40.83, samples=39
   iops        : min=   41, max=   90, avg=66.41, stdev=10.28, samples=39
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.39%, 50=0.94%, 100=42.97%
  lat (msec)   : 250=54.80%, 500=0.82%
  cpu          : usr=0.24%, sys=1.62%, ctx=2941, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24367: Thu Jul 25 01:36:30 2024
   read: IOPS=62, BW=251KiB/s (257kB/s)(4984KiB/19820msec)
    slat (usec): min=679, max=3795, avg=1366.21, stdev=410.48
    clat (msec): min=2, max=133, avg=81.45, stdev=16.58
     lat (msec): min=3, max=136, avg=82.82, stdev=16.70
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   59], 10.00th=[   64], 20.00th=[   68],
     | 30.00th=[   74], 40.00th=[   78], 50.00th=[   81], 60.00th=[   85],
     | 70.00th=[   89], 80.00th=[   94], 90.00th=[  103], 95.00th=[  112],
     | 99.00th=[  122], 99.50th=[  128], 99.90th=[  133], 99.95th=[  134],
     | 99.99th=[  134]
   bw (  KiB/s): min=  176, max=  375, per=24.82%, avg=251.21, stdev=52.22, samples=39
   iops        : min=   44, max=   93, avg=62.74, stdev=13.00, samples=39
  write: IOPS=66, BW=265KiB/s (272kB/s)(5256KiB/19820msec)
    slat (usec): min=14, max=2019, avg=69.44, stdev=69.49
    clat (msec): min=69, max=276, avg=162.36, stdev=36.48
     lat (msec): min=69, max=276, avg=162.43, stdev=36.49
    clat percentiles (msec):
     |  1.00th=[   91],  5.00th=[  106], 10.00th=[  115], 20.00th=[  129],
     | 30.00th=[  142], 40.00th=[  153], 50.00th=[  161], 60.00th=[  171],
     | 70.00th=[  180], 80.00th=[  192], 90.00th=[  211], 95.00th=[  228],
     | 99.00th=[  257], 99.50th=[  268], 99.90th=[  275], 99.95th=[  275],
     | 99.99th=[  275]
   bw (  KiB/s): min=  192, max=  352, per=24.83%, avg=261.67, stdev=39.89, samples=39
   iops        : min=   48, max=   88, avg=65.36, stdev= 9.99, samples=39
  lat (msec)   : 4=0.04%, 20=0.35%, 50=0.74%, 100=42.85%, 250=55.35%
  lat (msec)   : 500=0.66%
  cpu          : usr=0.19%, sys=1.70%, ctx=3098, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24368: Thu Jul 25 01:36:30 2024
   read: IOPS=64, BW=256KiB/s (262kB/s)(5060KiB/19763msec)
    slat (usec): min=643, max=3193, avg=1336.75, stdev=388.66
    clat (usec): min=1348, max=155925, avg=81449.76, stdev=16341.01
     lat (msec): min=2, max=156, avg=82.79, stdev=16.48
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   59], 10.00th=[   64], 20.00th=[   69],
     | 30.00th=[   74], 40.00th=[   78], 50.00th=[   81], 60.00th=[   85],
     | 70.00th=[   90], 80.00th=[   94], 90.00th=[  102], 95.00th=[  110],
     | 99.00th=[  125], 99.50th=[  130], 99.90th=[  142], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  152, max=  368, per=25.31%, avg=256.10, stdev=53.71, samples=39
   iops        : min=   38, max=   92, avg=63.95, stdev=13.45, samples=39
  write: IOPS=65, BW=262KiB/s (268kB/s)(5180KiB/19763msec)
    slat (usec): min=13, max=1178, avg=68.92, stdev=55.27
    clat (msec): min=80, max=274, avg=162.73, stdev=36.79
     lat (msec): min=80, max=275, avg=162.80, stdev=36.80
    clat percentiles (msec):
     |  1.00th=[   95],  5.00th=[  107], 10.00th=[  117], 20.00th=[  130],
     | 30.00th=[  140], 40.00th=[  150], 50.00th=[  161], 60.00th=[  171],
     | 70.00th=[  182], 80.00th=[  194], 90.00th=[  213], 95.00th=[  230],
     | 99.00th=[  259], 99.50th=[  264], 99.90th=[  275], 99.95th=[  275],
     | 99.99th=[  275]
   bw (  KiB/s): min=  175, max=  328, per=24.69%, avg=260.21, stdev=38.49, samples=39
   iops        : min=   43, max=   82, avg=64.97, stdev= 9.73, samples=39
  lat (msec)   : 2=0.08%, 20=0.20%, 50=0.66%, 100=44.06%, 250=54.14%
  lat (msec)   : 500=0.86%
  cpu          : usr=0.29%, sys=1.54%, ctx=3092, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1013KiB/s (1037kB/s), 251KiB/s-256KiB/s (257kB/s-262kB/s), io=19.6MiB (20.5MB), run=19735-19820msec
  WRITE: bw=1054KiB/s (1079kB/s), 262KiB/s-267KiB/s (268kB/s-273kB/s), io=20.4MiB (21.4MB), run=19735-19820msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=24388: Thu Jul 25 01:36:31 2024
   read: IOPS=5, BW=219MiB/s (229MB/s)(40.0MiB/183msec)
    slat (nsec): min=5744.5k, max=5744.5k, avg=5744469.00, stdev= 0.00
    clat (nsec): min=169926k, max=169926k, avg=169925694.00, stdev= 0.00
     lat (nsec): min=175679k, max=175679k, avg=175679343.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  171],  5.00th=[  171], 10.00th=[  171], 20.00th=[  171],
     | 30.00th=[  171], 40.00th=[  171], 50.00th=[  171], 60.00th=[  171],
     | 70.00th=[  171], 80.00th=[  171], 90.00th=[  171], 95.00th=[  171],
     | 99.00th=[  171], 99.50th=[  171], 99.90th=[  171], 99.95th=[  171],
     | 99.99th=[  171]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.30%, ctx=15, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=219MiB/s (229MB/s), 219MiB/s-219MiB/s (229MB/s-229MB/s), io=40.0MiB (41.9MB), run=183-183msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout