== sanity test 398c: run fio to test AIO ================= 01:03:11 (1735106591)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.146153 s, 287 MB/s
osc.lustre-OST0000-osc-ffff88012db6c000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=4864: Wed Dec 25 01:03:43 2024
  write: IOPS=82, BW=330KiB/s (337kB/s)(10.0MiB/31071msec)
    slat (usec): min=14, max=512, avg=79.15, stdev=47.94
    clat (msec): min=12, max=333, avg=194.03, stdev=46.98
     lat (msec): min=12, max=334, avg=194.11, stdev=46.98
    clat percentiles (msec):
     |  1.00th=[   87],  5.00th=[  110], 10.00th=[  140], 20.00th=[  159],
     | 30.00th=[  171], 40.00th=[  182], 50.00th=[  194], 60.00th=[  205],
     | 70.00th=[  218], 80.00th=[  232], 90.00th=[  257], 95.00th=[  275],
     | 99.00th=[  300], 99.50th=[  313], 99.90th=[  330], 99.95th=[  330],
     | 99.99th=[  334]
   bw (  KiB/s): min=  200, max=  616, per=24.97%, avg=328.29, stdev=72.37, samples=62
   iops        : min=   50, max=  154, avg=82.00, stdev=18.09, samples=62
  lat (msec)   : 20=0.04%, 50=0.08%, 100=3.20%, 250=84.18%, 500=12.50%
  cpu          : usr=0.14%, sys=0.75%, ctx=2640, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4865: Wed Dec 25 01:03:43 2024
  write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31116msec)
    slat (usec): min=13, max=541, avg=79.59, stdev=50.28
    clat (msec): min=75, max=332, avg=194.22, stdev=45.79
     lat (msec): min=75, max=332, avg=194.30, stdev=45.79
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  118], 10.00th=[  140], 20.00th=[  159],
     | 30.00th=[  169], 40.00th=[  182], 50.00th=[  194], 60.00th=[  205],
     | 70.00th=[  215], 80.00th=[  232], 90.00th=[  257], 95.00th=[  275],
     | 99.00th=[  300], 99.50th=[  305], 99.90th=[  317], 99.95th=[  317],
     | 99.99th=[  334]
   bw (  KiB/s): min=  224, max=  592, per=24.91%, avg=327.53, stdev=67.39, samples=62
   iops        : min=   56, max=  148, avg=81.82, stdev=16.85, samples=62
  lat (msec)   : 100=2.42%, 250=85.43%, 500=12.15%
  cpu          : usr=0.21%, sys=0.66%, ctx=2604, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4866: Wed Dec 25 01:03:43 2024
  write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31124msec)
    slat (usec): min=13, max=434, avg=79.70, stdev=50.27
    clat (msec): min=51, max=334, avg=194.30, stdev=45.65
     lat (msec): min=52, max=334, avg=194.38, stdev=45.66
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  121], 10.00th=[  140], 20.00th=[  159],
     | 30.00th=[  169], 40.00th=[  182], 50.00th=[  192], 60.00th=[  203],
     | 70.00th=[  215], 80.00th=[  232], 90.00th=[  257], 95.00th=[  279],
     | 99.00th=[  300], 99.50th=[  309], 99.90th=[  321], 99.95th=[  330],
     | 99.99th=[  334]
   bw (  KiB/s): min=  231, max=  568, per=24.92%, avg=327.65, stdev=66.58, samples=62
   iops        : min=   57, max=  142, avg=81.84, stdev=16.65, samples=62
  lat (msec)   : 100=2.46%, 250=85.47%, 500=12.07%
  cpu          : usr=0.13%, sys=0.76%, ctx=2556, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4867: Wed Dec 25 01:03:43 2024
  write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31132msec)
    slat (usec): min=13, max=363, avg=78.01, stdev=49.68
    clat (msec): min=75, max=332, avg=194.29, stdev=45.74
     lat (msec): min=75, max=332, avg=194.37, stdev=45.75
    clat percentiles (msec):
     |  1.00th=[   90],  5.00th=[  117], 10.00th=[  142], 20.00th=[  159],
     | 30.00th=[  171], 40.00th=[  182], 50.00th=[  192], 60.00th=[  203],
     | 70.00th=[  215], 80.00th=[  230], 90.00th=[  257], 95.00th=[  279],
     | 99.00th=[  300], 99.50th=[  305], 99.90th=[  326], 99.95th=[  330],
     | 99.99th=[  334]
   bw (  KiB/s): min=  208, max=  608, per=24.91%, avg=327.53, stdev=69.54, samples=62
   iops        : min=   52, max=  152, avg=81.82, stdev=17.36, samples=62
  lat (msec)   : 100=2.58%, 250=85.12%, 500=12.30%
  cpu          : usr=0.17%, sys=0.67%, ctx=2658, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1316KiB/s (1347kB/s), 329KiB/s-330KiB/s (337kB/s-337kB/s), io=40.0MiB (41.9MB), run=31071-31132msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=4902: Wed Dec 25 01:03:59 2024
   read: IOPS=84, BW=339KiB/s (347kB/s)(5048KiB/14896msec)
    slat (usec): min=722, max=2701, avg=1178.08, stdev=290.82
    clat (usec): min=1221, max=123227, avg=60439.96, stdev=12118.49
     lat (msec): min=2, max=124, avg=61.62, stdev=12.17
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   43], 10.00th=[   47], 20.00th=[   52],
     | 30.00th=[   56], 40.00th=[   58], 50.00th=[   61], 60.00th=[   64],
     | 70.00th=[   66], 80.00th=[   69], 90.00th=[   72], 95.00th=[   79],
     | 99.00th=[  107], 99.50th=[  113], 99.90th=[  122], 99.95th=[  124],
     | 99.99th=[  124]
   bw (  KiB/s): min=  224, max=  432, per=25.16%, avg=338.45, stdev=57.16, samples=29
   iops        : min=   56, max=  108, avg=84.59, stdev=14.28, samples=29
  write: IOPS=87, BW=349KiB/s (357kB/s)(5192KiB/14896msec)
    slat (usec): min=14, max=375, avg=56.95, stdev=29.66
    clat (msec): min=55, max=201, avg=123.17, stdev=26.88
     lat (msec): min=55, max=201, avg=123.23, stdev=26.88
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[   80], 10.00th=[   87], 20.00th=[   97],
     | 30.00th=[  108], 40.00th=[  116], 50.00th=[  125], 60.00th=[  133],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  159], 95.00th=[  165],
     | 99.00th=[  184], 99.50th=[  188], 99.90th=[  192], 99.95th=[  201],
     | 99.99th=[  201]
   bw (  KiB/s): min=  232, max=  448, per=24.83%, avg=347.55, stdev=39.68, samples=29
   iops        : min=   58, max=  112, avg=86.86, stdev= 9.91, samples=29
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.12%, 20=0.20%, 50=7.66%
  lat (msec)   : 100=52.03%, 250=39.92%
  cpu          : usr=0.21%, sys=1.66%, ctx=3300, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4903: Wed Dec 25 01:03:59 2024
   read: IOPS=83, BW=334KiB/s (342kB/s)(4976KiB/14914msec)
    slat (usec): min=711, max=3643, avg=1167.02, stdev=285.34
    clat (msec): min=2, max=121, avg=60.14, stdev=11.24
     lat (msec): min=3, max=125, avg=61.30, stdev=11.29
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   45], 10.00th=[   48], 20.00th=[   52],
     | 30.00th=[   56], 40.00th=[   58], 50.00th=[   61], 60.00th=[   64],
     | 70.00th=[   66], 80.00th=[   68], 90.00th=[   72], 95.00th=[   78],
     | 99.00th=[   88], 99.50th=[   92], 99.90th=[  118], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  224, max=  472, per=24.61%, avg=331.03, stdev=64.94, samples=29
   iops        : min=   56, max=  118, avg=82.76, stdev=16.23, samples=29
  write: IOPS=88, BW=353KiB/s (361kB/s)(5264KiB/14914msec)
    slat (usec): min=13, max=330, avg=55.49, stdev=29.65
    clat (msec): min=39, max=197, avg=122.99, stdev=27.53
     lat (msec): min=39, max=197, avg=123.05, stdev=27.53
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[   79], 10.00th=[   86], 20.00th=[   96],
     | 30.00th=[  108], 40.00th=[  115], 50.00th=[  124], 60.00th=[  132],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  159], 95.00th=[  169],
     | 99.00th=[  182], 99.50th=[  188], 99.90th=[  197], 99.95th=[  199],
     | 99.99th=[  199]
   bw (  KiB/s): min=  272, max=  488, per=25.00%, avg=350.07, stdev=46.63, samples=29
   iops        : min=   68, max=  122, avg=87.52, stdev=11.66, samples=29
  lat (msec)   : 4=0.12%, 10=0.04%, 20=0.23%, 50=7.38%, 100=52.15%
  lat (msec)   : 250=40.08%
  cpu          : usr=0.28%, sys=1.60%, ctx=3029, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4904: Wed Dec 25 01:03:59 2024
   read: IOPS=83, BW=334KiB/s (342kB/s)(4984KiB/14914msec)
    slat (usec): min=696, max=3191, avg=1168.43, stdev=272.12
    clat (msec): min=2, max=121, avg=59.89, stdev=11.87
     lat (msec): min=3, max=123, avg=61.06, stdev=11.93
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[   43], 10.00th=[   47], 20.00th=[   52],
     | 30.00th=[   55], 40.00th=[   58], 50.00th=[   61], 60.00th=[   63],
     | 70.00th=[   66], 80.00th=[   68], 90.00th=[   73], 95.00th=[   79],
     | 99.00th=[   88], 99.50th=[  106], 99.90th=[  121], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  224, max=  480, per=24.91%, avg=335.03, stdev=69.14, samples=29
   iops        : min=   56, max=  120, avg=83.66, stdev=17.34, samples=29
  write: IOPS=88, BW=352KiB/s (361kB/s)(5256KiB/14914msec)
    slat (usec): min=14, max=767, avg=57.63, stdev=36.42
    clat (msec): min=30, max=195, avg=123.14, stdev=27.18
     lat (msec): min=30, max=195, avg=123.20, stdev=27.18
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   79], 10.00th=[   86], 20.00th=[   99],
     | 30.00th=[  108], 40.00th=[  115], 50.00th=[  125], 60.00th=[  133],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  159], 95.00th=[  167],
     | 99.00th=[  182], 99.50th=[  188], 99.90th=[  197], 99.95th=[  197],
     | 99.99th=[  197]
   bw (  KiB/s): min=  264, max=  463, per=24.92%, avg=348.83, stdev=45.62, samples=29
   iops        : min=   66, max=  115, avg=87.10, stdev=11.37, samples=29
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.39%, 50=7.97%, 100=51.17%
  lat (msec)   : 250=40.39%
  cpu          : usr=0.15%, sys=1.74%, ctx=3197, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4905: Wed Dec 25 01:03:59 2024
   read: IOPS=85, BW=341KiB/s (349kB/s)(5060KiB/14839msec)
    slat (usec): min=53, max=3410, avg=1165.14, stdev=311.24
    clat (usec): min=1994, max=146369, avg=60415.28, stdev=12143.02
     lat (msec): min=2, max=148, avg=61.58, stdev=12.22
    clat percentiles (msec):
     |  1.00th=[   28],  5.00th=[   46], 10.00th=[   49], 20.00th=[   53],
     | 30.00th=[   56], 40.00th=[   58], 50.00th=[   61], 60.00th=[   63],
     | 70.00th=[   66], 80.00th=[   68], 90.00th=[   72], 95.00th=[   79],
     | 99.00th=[   89], 99.50th=[  117], 99.90th=[  144], 99.95th=[  146],
     | 99.99th=[  146]
   bw (  KiB/s): min=  216, max=  448, per=25.43%, avg=342.03, stdev=59.07, samples=29
   iops        : min=   54, max=  112, avg=85.48, stdev=14.73, samples=29
  write: IOPS=87, BW=349KiB/s (357kB/s)(5180KiB/14839msec)
    slat (usec): min=12, max=387, avg=57.30, stdev=32.86
    clat (msec): min=25, max=207, avg=122.93, stdev=28.68
     lat (msec): min=25, max=207, avg=122.99, stdev=28.68
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   78], 10.00th=[   84], 20.00th=[   95],
     | 30.00th=[  106], 40.00th=[  115], 50.00th=[  125], 60.00th=[  133],
     | 70.00th=[  140], 80.00th=[  148], 90.00th=[  159], 95.00th=[  169],
     | 99.00th=[  190], 99.50th=[  194], 99.90th=[  201], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=  272, max=  480, per=24.83%, avg=347.55, stdev=42.87, samples=29
   iops        : min=   68, max=  120, avg=86.86, stdev=10.74, samples=29
  lat (msec)   : 2=0.04%, 4=0.16%, 10=0.12%, 20=0.08%, 50=7.19%
  lat (msec)   : 100=53.12%, 250=39.30%
  cpu          : usr=0.23%, sys=1.68%, ctx=3172, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1346KiB/s (1378kB/s), 334KiB/s-341KiB/s (342kB/s-349kB/s), io=19.6MiB (20.5MB), run=14839-14914msec
  WRITE: bw=1401KiB/s (1434kB/s), 349KiB/s-353KiB/s (357kB/s-361kB/s), io=20.4MiB (21.4MB), run=14839-14914msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=4921: Wed Dec 25 01:04:01 2024
   read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec)
    slat (nsec): min=6063.0k, max=6063.0k, avg=6063049.00, stdev= 0.00
    clat (nsec): min=89517k, max=89517k, avg=89516763.00, stdev= 0.00
     lat (nsec): min=95591k, max=95591k, avg=95590854.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[89654],  5.00th=[89654], 10.00th=[89654], 20.00th=[89654],
     | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654],
     | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654],
     | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.06%, ctx=21, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout