== sanity test 398c: run fio to test AIO ================= 21:35:11 (1736994911)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0971627 s, 432 MB/s
osc.lustre-OST0000-osc-ffff88008bfdb000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=24236: Wed Jan 15 21:35:36 2025
  write: IOPS=109, BW=437KiB/s (447kB/s)(10.0MiB/23446msec)
    slat (usec): min=9, max=338, avg=75.98, stdev=42.46
    clat (msec): min=62, max=225, avg=146.38, stdev=27.49
     lat (msec): min=62, max=225, avg=146.46, stdev=27.50
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[  103], 10.00th=[  112], 20.00th=[  124],
     | 30.00th=[  132], 40.00th=[  140], 50.00th=[  146], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  171], 90.00th=[  182], 95.00th=[  190],
     | 99.00th=[  207], 99.50th=[  211], 99.90th=[  220], 99.95th=[  224],
     | 99.99th=[  226]
   bw (  KiB/s): min=  352, max=  712, per=24.96%, avg=435.30, stdev=59.14, samples=46
   iops        : min=   88, max=  178, avg=108.83, stdev=14.78, samples=46
  lat (msec)   : 100=4.26%, 250=95.74%
  cpu          : usr=0.23%, sys=0.90%, ctx=2154, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24237: Wed Jan 15 21:35:36 2025
  write: IOPS=109, BW=437KiB/s (448kB/s)(10.0MiB/23411msec)
    slat (usec): min=11, max=244, avg=76.88, stdev=42.65
    clat (msec): min=12, max=246, avg=146.18, stdev=28.26
     lat (msec): min=12, max=246, avg=146.26, stdev=28.27
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[  100], 10.00th=[  112], 20.00th=[  125],
     | 30.00th=[  132], 40.00th=[  140], 50.00th=[  146], 60.00th=[  155],
     | 70.00th=[  163], 80.00th=[  171], 90.00th=[  180], 95.00th=[  190],
     | 99.00th=[  207], 99.50th=[  213], 99.90th=[  226], 99.95th=[  239],
     | 99.99th=[  247]
   bw (  KiB/s): min=  328, max=  736, per=25.02%, avg=436.41, stdev=63.13, samples=46
   iops        : min=   82, max=  184, avg=109.02, stdev=15.78, samples=46
  lat (msec)   : 20=0.08%, 50=0.74%, 100=4.18%, 250=95.00%
  cpu          : usr=0.22%, sys=0.91%, ctx=2215, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24238: Wed Jan 15 21:35:36 2025
  write: IOPS=109, BW=436KiB/s (447kB/s)(10.0MiB/23467msec)
    slat (usec): min=10, max=411, avg=72.25, stdev=42.06
    clat (msec): min=45, max=234, avg=146.52, stdev=27.33
     lat (msec): min=45, max=234, avg=146.60, stdev=27.33
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[  103], 10.00th=[  112], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  148], 60.00th=[  157],
     | 70.00th=[  163], 80.00th=[  171], 90.00th=[  180], 95.00th=[  190],
     | 99.00th=[  207], 99.50th=[  211], 99.90th=[  228], 99.95th=[  234],
     | 99.99th=[  234]
   bw (  KiB/s): min=  352, max=  712, per=24.96%, avg=435.24, stdev=58.03, samples=46
   iops        : min=   88, max=  178, avg=108.76, stdev=14.51, samples=46
  lat (msec)   : 50=0.08%, 100=4.57%, 250=95.35%
  cpu          : usr=0.18%, sys=0.93%, ctx=1933, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24239: Wed Jan 15 21:35:36 2025
  write: IOPS=109, BW=436KiB/s (447kB/s)(10.0MiB/23482msec)
    slat (usec): min=9, max=373, avg=73.03, stdev=44.11
    clat (msec): min=45, max=225, avg=146.61, stdev=27.65
     lat (msec): min=45, max=225, avg=146.68, stdev=27.66
    clat percentiles (msec):
     |  1.00th=[   78],  5.00th=[  103], 10.00th=[  112], 20.00th=[  125],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  146], 60.00th=[  155],
     | 70.00th=[  163], 80.00th=[  171], 90.00th=[  182], 95.00th=[  192],
     | 99.00th=[  205], 99.50th=[  209], 99.90th=[  220], 99.95th=[  224],
     | 99.99th=[  226]
   bw (  KiB/s): min=  336, max=  720, per=24.90%, avg=434.24, stdev=61.66, samples=46
   iops        : min=   84, max=  180, avg=108.54, stdev=15.41, samples=46
  lat (msec)   : 50=0.04%, 100=4.61%, 250=95.35%
  cpu          : usr=0.21%, sys=0.87%, ctx=2112, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1744KiB/s (1786kB/s), 436KiB/s-437KiB/s (447kB/s-448kB/s), io=40.0MiB (41.9MB), run=23411-23482msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=24268: Wed Jan 15 21:35:51 2025
   read: IOPS=90, BW=362KiB/s (371kB/s)(5048KiB/13940msec)
    slat (usec): min=508, max=3155, avg=870.74, stdev=224.27
    clat (msec): min=10, max=161, avg=58.50, stdev=12.01
     lat (msec): min=11, max=162, avg=59.37, stdev=12.01
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   43], 10.00th=[   46], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   55], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   67], 90.00th=[   71], 95.00th=[   78],
     | 99.00th=[   94], 99.50th=[  114], 99.90th=[  153], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  240, max=  456, per=25.20%, avg=362.63, stdev=53.55, samples=27
   iops        : min=   60, max=  114, avg=90.63, stdev=13.44, samples=27
  write: IOPS=93, BW=372KiB/s (381kB/s)(5192KiB/13940msec)
    slat (usec): min=10, max=611, avg=63.00, stdev=41.22
    clat (msec): min=56, max=181, avg=113.83, stdev=23.22
     lat (msec): min=56, max=181, avg=113.89, stdev=23.22
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   78], 10.00th=[   84], 20.00th=[   93],
     | 30.00th=[  101], 40.00th=[  107], 50.00th=[  113], 60.00th=[  118],
     | 70.00th=[  127], 80.00th=[  136], 90.00th=[  146], 95.00th=[  153],
     | 99.00th=[  169], 99.50th=[  171], 99.90th=[  176], 99.95th=[  182],
     | 99.99th=[  182]
   bw (  KiB/s): min=  328, max=  440, per=24.78%, avg=371.22, stdev=28.58, samples=27
   iops        : min=   82, max=  110, avg=92.78, stdev= 7.13, samples=27
  lat (msec)   : 20=0.12%, 50=10.27%, 100=53.67%, 250=35.94%
  cpu          : usr=0.39%, sys=1.87%, ctx=2981, majf=0, minf=36
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24269: Wed Jan 15 21:35:51 2025
   read: IOPS=89, BW=357KiB/s (366kB/s)(4976KiB/13939msec)
    slat (usec): min=21, max=4752, avg=874.00, stdev=253.28
    clat (usec): min=1221, max=162482, avg=59035.68, stdev=12926.62
     lat (usec): min=1269, max=163457, avg=59912.34, stdev=12936.45
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   43], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   65], 80.00th=[   68], 90.00th=[   73], 95.00th=[   79],
     | 99.00th=[   93], 99.50th=[  122], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  248, max=  488, per=24.79%, avg=356.74, stdev=62.73, samples=27
   iops        : min=   62, max=  122, avg=89.19, stdev=15.68, samples=27
  write: IOPS=94, BW=378KiB/s (387kB/s)(5264KiB/13939msec)
    slat (usec): min=9, max=326, avg=61.51, stdev=38.68
    clat (msec): min=11, max=191, avg=112.63, stdev=24.63
     lat (msec): min=11, max=191, avg=112.69, stdev=24.63
    clat percentiles (msec):
     |  1.00th=[   54],  5.00th=[   74], 10.00th=[   83], 20.00th=[   93],
     | 30.00th=[  101], 40.00th=[  106], 50.00th=[  112], 60.00th=[  118],
     | 70.00th=[  127], 80.00th=[  133], 90.00th=[  146], 95.00th=[  150],
     | 99.00th=[  171], 99.50th=[  182], 99.90th=[  190], 99.95th=[  192],
     | 99.99th=[  192]
   bw (  KiB/s): min=  320, max=  432, per=25.24%, avg=378.07, stdev=33.77, samples=27
   iops        : min=   80, max=  108, avg=94.52, stdev= 8.44, samples=27
  lat (msec)   : 2=0.04%, 10=0.12%, 20=0.20%, 50=9.30%, 100=54.38%
  lat (msec)   : 250=35.98%
  cpu          : usr=0.27%, sys=1.97%, ctx=2920, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24270: Wed Jan 15 21:35:51 2025
   read: IOPS=89, BW=358KiB/s (366kB/s)(4984KiB/13928msec)
    slat (usec): min=21, max=3178, avg=875.41, stdev=246.30
    clat (msec): min=7, max=151, avg=58.21, stdev=11.92
     lat (msec): min=8, max=152, avg=59.09, stdev=11.92
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   43], 10.00th=[   47], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   56], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   63], 80.00th=[   67], 90.00th=[   71], 95.00th=[   77],
     | 99.00th=[   91], 99.50th=[   94], 99.90th=[  153], 99.95th=[  153],
     | 99.99th=[  153]
   bw (  KiB/s): min=  264, max=  464, per=25.06%, avg=360.56, stdev=54.83, samples=27
   iops        : min=   66, max=  116, avg=90.11, stdev=13.70, samples=27
  write: IOPS=94, BW=377KiB/s (386kB/s)(5256KiB/13928msec)
    slat (usec): min=12, max=865, avg=62.31, stdev=43.01
    clat (msec): min=28, max=194, avg=113.36, stdev=24.01
     lat (msec): min=28, max=194, avg=113.42, stdev=24.01
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   75], 10.00th=[   84], 20.00th=[   93],
     | 30.00th=[  101], 40.00th=[  107], 50.00th=[  112], 60.00th=[  118],
     | 70.00th=[  126], 80.00th=[  136], 90.00th=[  146], 95.00th=[  155],
     | 99.00th=[  174], 99.50th=[  176], 99.90th=[  190], 99.95th=[  194],
     | 99.99th=[  194]
   bw (  KiB/s): min=  320, max=  472, per=25.08%, avg=375.67, stdev=39.45, samples=27
   iops        : min=   80, max=  118, avg=93.89, stdev= 9.89, samples=27
  lat (msec)   : 10=0.16%, 20=0.20%, 50=9.84%, 100=53.36%, 250=36.45%
  cpu          : usr=0.35%, sys=1.92%, ctx=2869, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24271: Wed Jan 15 21:35:51 2025
   read: IOPS=90, BW=363KiB/s (372kB/s)(5060KiB/13939msec)
    slat (usec): min=494, max=3116, avg=875.99, stdev=222.90
    clat (usec): min=921, max=130836, avg=58511.74, stdev=11752.35
     lat (msec): min=2, max=131, avg=59.39, stdev=11.75
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   43], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   59], 60.00th=[   61],
     | 70.00th=[   63], 80.00th=[   67], 90.00th=[   71], 95.00th=[   78],
     | 99.00th=[   91], 99.50th=[   94], 99.90th=[  130], 99.95th=[  131],
     | 99.99th=[  131]
   bw (  KiB/s): min=  264, max=  456, per=25.26%, avg=363.48, stdev=43.35, samples=27
   iops        : min=   66, max=  114, avg=90.81, stdev=10.86, samples=27
  write: IOPS=92, BW=372KiB/s (381kB/s)(5180KiB/13939msec)
    slat (usec): min=13, max=885, avg=63.20, stdev=44.31
    clat (msec): min=49, max=191, avg=113.95, stdev=23.65
     lat (msec): min=49, max=191, avg=114.02, stdev=23.66
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   77], 10.00th=[   84], 20.00th=[   93],
     | 30.00th=[  102], 40.00th=[  108], 50.00th=[  114], 60.00th=[  120],
     | 70.00th=[  127], 80.00th=[  134], 90.00th=[  146], 95.00th=[  155],
     | 99.00th=[  171], 99.50th=[  178], 99.90th=[  190], 99.95th=[  192],
     | 99.99th=[  192]
   bw (  KiB/s): min=  311, max=  432, per=24.66%, avg=369.41, stdev=33.97, samples=27
   iops        : min=   77, max=  108, avg=92.30, stdev= 8.57, samples=27
  lat (usec)   : 1000=0.04%
  lat (msec)   : 4=0.04%, 10=0.12%, 20=0.27%, 50=8.83%, 100=54.38%
  lat (msec)   : 250=36.33%
  cpu          : usr=0.39%, sys=1.86%, ctx=3039, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1440KiB/s (1474kB/s), 357KiB/s-363KiB/s (366kB/s-372kB/s), io=19.6MiB (20.5MB), run=13928-13940msec
  WRITE: bw=1499KiB/s (1535kB/s), 372KiB/s-378KiB/s (381kB/s-387kB/s), io=20.4MiB (21.4MB), run=13928-13940msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=24286: Wed Jan 15 21:35:52 2025
   read: IOPS=13, BW=541MiB/s (567MB/s)(40.0MiB/74msec)
    slat (nsec): min=8795.7k, max=8795.7k, avg=8795657.00, stdev= 0.00
    clat (nsec): min=61078k, max=61078k, avg=61078298.00, stdev= 0.00
     lat (nsec): min=69900k, max=69900k, avg=69900259.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[61080],  5.00th=[61080], 10.00th=[61080], 20.00th=[61080],
     | 30.00th=[61080], 40.00th=[61080], 50.00th=[61080], 60.00th=[61080],
     | 70.00th=[61080], 80.00th=[61080], 90.00th=[61080], 95.00th=[61080],
     | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61080]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=12.33%, ctx=3, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=541MiB/s (567MB/s), 541MiB/s-541MiB/s (567MB/s-567MB/s), io=40.0MiB (41.9MB), run=74-74msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck