== sanity test 398c: run fio to test AIO ================= 19:31:34 (1736987494)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0969099 s, 433 MB/s
osc.lustre-OST0000-osc-ffff8800a9520000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=23987: Wed Jan 15 19:31:58 2025
  write: IOPS=110, BW=443KiB/s (454kB/s)(10.0MiB/23090msec)
    slat (usec): min=12, max=4459, avg=73.00, stdev=95.77
    clat (msec): min=37, max=214, avg=144.15, stdev=27.52
     lat (msec): min=37, max=214, avg=144.22, stdev=27.53
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   92], 10.00th=[  111], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  146], 60.00th=[  155],
     | 70.00th=[  159], 80.00th=[  167], 90.00th=[  178], 95.00th=[  186],
     | 99.00th=[  199], 99.50th=[  207], 99.90th=[  211], 99.95th=[  211],
     | 99.99th=[  215]
   bw (  KiB/s): min=  351, max=  774, per=24.95%, avg=442.41, stdev=71.53, samples=46
   iops        : min=   87, max=  193, avg=110.48, stdev=17.90, samples=46
  lat (msec)   : 50=0.04%, 100=6.72%, 250=93.24%
  cpu          : usr=0.28%, sys=0.84%, ctx=2045, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23988: Wed Jan 15 19:31:58 2025
  write: IOPS=111, BW=445KiB/s (456kB/s)(10.0MiB/23002msec)
    slat (usec): min=12, max=914, avg=77.16, stdev=50.42
    clat (msec): min=24, max=218, avg=143.61, stdev=28.50
     lat (msec): min=24, max=218, avg=143.69, stdev=28.50
    clat percentiles (msec):
     |  1.00th=[   52],  5.00th=[   88], 10.00th=[  111], 20.00th=[  125],
     | 30.00th=[  132], 40.00th=[  140], 50.00th=[  148], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  167], 90.00th=[  178], 95.00th=[  184],
     | 99.00th=[  199], 99.50th=[  203], 99.90th=[  209], 99.95th=[  209],
     | 99.99th=[  220]
   bw (  KiB/s): min=  360, max=  870, per=24.95%, avg=442.39, stdev=81.74, samples=46
   iops        : min=   90, max=  217, avg=110.46, stdev=20.43, samples=46
  lat (msec)   : 50=0.74%, 100=6.37%, 250=92.89%
  cpu          : usr=0.21%, sys=0.92%, ctx=2354, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23989: Wed Jan 15 19:31:58 2025
  write: IOPS=111, BW=444KiB/s (455kB/s)(10.0MiB/23042msec)
    slat (usec): min=13, max=1730, avg=76.48, stdev=57.44
    clat (msec): min=27, max=221, avg=143.84, stdev=28.02
     lat (msec): min=27, max=221, avg=143.92, stdev=28.02
    clat percentiles (msec):
     |  1.00th=[   64],  5.00th=[   90], 10.00th=[  110], 20.00th=[  124],
     | 30.00th=[  132], 40.00th=[  140], 50.00th=[  148], 60.00th=[  153],
     | 70.00th=[  161], 80.00th=[  167], 90.00th=[  178], 95.00th=[  186],
     | 99.00th=[  197], 99.50th=[  203], 99.90th=[  220], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  368, max=  798, per=24.95%, avg=442.39, stdev=72.51, samples=46
   iops        : min=   92, max=  199, avg=110.46, stdev=18.13, samples=46
  lat (msec)   : 50=0.51%, 100=6.37%, 250=93.12%
  cpu          : usr=0.27%, sys=0.87%, ctx=2195, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=23990: Wed Jan 15 19:31:58 2025
  write: IOPS=110, BW=443KiB/s (454kB/s)(10.0MiB/23097msec)
    slat (usec): min=12, max=6515, avg=75.04, stdev=140.65
    clat (msec): min=37, max=221, avg=144.19, stdev=27.94
     lat (msec): min=37, max=221, avg=144.26, stdev=27.94
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   90], 10.00th=[  110], 20.00th=[  125],
     | 30.00th=[  132], 40.00th=[  140], 50.00th=[  146], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  167], 90.00th=[  178], 95.00th=[  186],
     | 99.00th=[  199], 99.50th=[  207], 99.90th=[  213], 99.95th=[  222],
     | 99.99th=[  222]
   bw (  KiB/s): min=  343, max=  782, per=24.95%, avg=442.43, stdev=73.11, samples=46
   iops        : min=   85, max=  195, avg=110.50, stdev=18.25, samples=46
  lat (msec)   : 50=0.23%, 100=7.23%, 250=92.54%
  cpu          : usr=0.25%, sys=0.87%, ctx=2021, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1773KiB/s (1816kB/s), 443KiB/s-445KiB/s (454kB/s-456kB/s), io=40.0MiB (41.9MB), run=23002-23097msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=24018: Wed Jan 15 19:32:13 2025
   read: IOPS=90, BW=360KiB/s (369kB/s)(5048KiB/14006msec)
    slat (usec): min=512, max=3393, avg=888.61, stdev=250.71
    clat (msec): min=9, max=125, avg=59.12, stdev=10.87
     lat (msec): min=10, max=126, avg=60.01, stdev=10.88
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   44], 10.00th=[   48], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   57], 50.00th=[   59], 60.00th=[   62],
     | 70.00th=[   64], 80.00th=[   68], 90.00th=[   72], 95.00th=[   78],
     | 99.00th=[   93], 99.50th=[   95], 99.90th=[  126], 99.95th=[  126],
     | 99.99th=[  126]
   bw (  KiB/s): min=  224, max=  479, per=25.07%, avg=358.46, stdev=60.76, samples=28
   iops        : min=   56, max=  119, avg=89.54, stdev=15.10, samples=28
  write: IOPS=92, BW=371KiB/s (380kB/s)(5192KiB/14006msec)
    slat (usec): min=12, max=421, avg=61.12, stdev=33.33
    clat (msec): min=49, max=184, avg=114.02, stdev=25.26
     lat (msec): min=49, max=184, avg=114.09, stdev=25.27
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   75], 10.00th=[   83], 20.00th=[   91],
     | 30.00th=[   99], 40.00th=[  106], 50.00th=[  113], 60.00th=[  122],
     | 70.00th=[  130], 80.00th=[  138], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  171], 99.50th=[  176], 99.90th=[  180], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  319, max=  440, per=24.76%, avg=368.46, stdev=33.59, samples=28
   iops        : min=   79, max=  110, avg=92.04, stdev= 8.40, samples=28
  lat (msec)   : 10=0.04%, 20=0.08%, 50=8.40%, 100=57.27%, 250=34.22%
  cpu          : usr=0.33%, sys=1.90%, ctx=2980, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24019: Wed Jan 15 19:32:13 2025
   read: IOPS=88, BW=355KiB/s (363kB/s)(4976KiB/14025msec)
    slat (usec): min=492, max=2783, avg=892.39, stdev=236.68
    clat (usec): min=989, max=115730, avg=58577.40, stdev=11440.94
     lat (usec): min=1757, max=116508, avg=59472.01, stdev=11446.94
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   43], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   53], 40.00th=[   57], 50.00th=[   59], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   67], 90.00th=[   72], 95.00th=[   78],
     | 99.00th=[   86], 99.50th=[   94], 99.90th=[  115], 99.95th=[  116],
     | 99.99th=[  116]
   bw (  KiB/s): min=  232, max=  472, per=24.71%, avg=353.29, stdev=60.03, samples=28
   iops        : min=   58, max=  118, avg=88.21, stdev=15.01, samples=28
  write: IOPS=93, BW=375KiB/s (384kB/s)(5264KiB/14025msec)
    slat (usec): min=14, max=409, avg=61.88, stdev=34.21
    clat (msec): min=40, max=182, avg=114.08, stdev=25.94
     lat (msec): min=40, max=182, avg=114.15, stdev=25.94
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   74], 10.00th=[   83], 20.00th=[   90],
     | 30.00th=[   99], 40.00th=[  106], 50.00th=[  115], 60.00th=[  123],
     | 70.00th=[  130], 80.00th=[  138], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  171], 99.50th=[  176], 99.90th=[  180], 99.95th=[  182],
     | 99.99th=[  182]
   bw (  KiB/s): min=  296, max=  424, per=25.11%, avg=373.57, stdev=34.62, samples=28
   iops        : min=   74, max=  106, avg=93.29, stdev= 8.56, samples=28
  lat (usec)   : 1000=0.04%
  lat (msec)   : 10=0.16%, 20=0.20%, 50=8.98%, 100=55.86%, 250=34.77%
  cpu          : usr=0.30%, sys=1.92%, ctx=3070, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24020: Wed Jan 15 19:32:13 2025
   read: IOPS=88, BW=355KiB/s (364kB/s)(4984KiB/14032msec)
    slat (usec): min=513, max=3297, avg=892.32, stdev=249.21
    clat (usec): min=6517, max=95055, avg=58364.06, stdev=10434.60
     lat (usec): min=7455, max=96181, avg=59258.40, stdev=10445.55
    clat percentiles (usec):
     |  1.00th=[34866],  5.00th=[42206], 10.00th=[46400], 20.00th=[50070],
     | 30.00th=[53216], 40.00th=[56361], 50.00th=[58459], 60.00th=[61080],
     | 70.00th=[63177], 80.00th=[66847], 90.00th=[70779], 95.00th=[76022],
     | 99.00th=[83362], 99.50th=[84411], 99.90th=[92799], 99.95th=[94897],
     | 99.99th=[94897]
   bw (  KiB/s): min=  223, max=  496, per=24.81%, avg=354.71, stdev=69.39, samples=28
   iops        : min=   55, max=  124, avg=88.57, stdev=17.31, samples=28
  write: IOPS=93, BW=375KiB/s (384kB/s)(5256KiB/14032msec)
    slat (usec): min=13, max=969, avg=62.31, stdev=41.65
    clat (msec): min=35, max=184, avg=114.47, stdev=25.62
     lat (msec): min=35, max=184, avg=114.54, stdev=25.62
    clat percentiles (msec):
     |  1.00th=[   63],  5.00th=[   73], 10.00th=[   82], 20.00th=[   91],
     | 30.00th=[  100], 40.00th=[  107], 50.00th=[  114], 60.00th=[  123],
     | 70.00th=[  131], 80.00th=[  138], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  169], 99.50th=[  171], 99.90th=[  184], 99.95th=[  184],
     | 99.99th=[  184]
   bw (  KiB/s): min=  288, max=  456, per=24.99%, avg=371.86, stdev=41.52, samples=28
   iops        : min=   72, max=  114, avg=92.86, stdev=10.39, samples=28
  lat (msec)   : 10=0.12%, 20=0.12%, 50=9.96%, 100=54.18%, 250=35.62%
  cpu          : usr=0.30%, sys=1.91%, ctx=3046, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24021: Wed Jan 15 19:32:13 2025
   read: IOPS=90, BW=362KiB/s (370kB/s)(5060KiB/13996msec)
    slat (usec): min=11, max=2875, avg=889.17, stdev=250.21
    clat (usec): min=906, max=116147, avg=58937.68, stdev=11853.57
     lat (usec): min=1125, max=117104, avg=59828.83, stdev=11875.25
    clat percentiles (msec):
     |  1.00th=[   29],  5.00th=[   43], 10.00th=[   47], 20.00th=[   51],
     | 30.00th=[   53], 40.00th=[   56], 50.00th=[   59], 60.00th=[   62],
     | 70.00th=[   64], 80.00th=[   68], 90.00th=[   73], 95.00th=[   79],
     | 99.00th=[   93], 99.50th=[   96], 99.90th=[  116], 99.95th=[  116],
     | 99.99th=[  116]
   bw (  KiB/s): min=  272, max=  464, per=25.14%, avg=359.56, stdev=47.81, samples=27
   iops        : min=   68, max=  116, avg=89.78, stdev=11.95, samples=27
  write: IOPS=92, BW=370KiB/s (379kB/s)(5180KiB/13996msec)
    slat (usec): min=11, max=303, avg=61.18, stdev=32.06
    clat (msec): min=13, max=184, avg=114.30, stdev=26.28
     lat (msec): min=13, max=184, avg=114.37, stdev=26.28
    clat percentiles (msec):
     |  1.00th=[   54],  5.00th=[   73], 10.00th=[   82], 20.00th=[   92],
     | 30.00th=[  100], 40.00th=[  107], 50.00th=[  114], 60.00th=[  123],
     | 70.00th=[  131], 80.00th=[  140], 90.00th=[  148], 95.00th=[  157],
     | 99.00th=[  171], 99.50th=[  176], 99.90th=[  180], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  320, max=  463, per=24.70%, avg=367.56, stdev=35.63, samples=27
   iops        : min=   80, max=  115, avg=91.78, stdev= 8.83, samples=27
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.08%, 20=0.43%, 50=9.34%, 100=55.00%, 250=35.12%
  cpu          : usr=0.40%, sys=1.86%, ctx=3000, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1430KiB/s (1464kB/s), 355KiB/s-362KiB/s (363kB/s-370kB/s), io=19.6MiB (20.5MB), run=13996-14032msec
  WRITE: bw=1489KiB/s (1525kB/s), 370KiB/s-375KiB/s (379kB/s-384kB/s), io=20.4MiB (21.4MB), run=13996-14032msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=24036: Wed Jan 15 19:32:14 2025
   read: IOPS=12, BW=519MiB/s (545MB/s)(40.0MiB/77msec)
    slat (nsec): min=8706.8k, max=8706.8k, avg=8706849.00, stdev= 0.00
    clat (nsec): min=64988k, max=64988k, avg=64987527.00, stdev= 0.00
     lat (nsec): min=73715k, max=73715k, avg=73714528.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[64750],  5.00th=[64750], 10.00th=[64750], 20.00th=[64750],
     | 30.00th=[64750], 40.00th=[64750], 50.00th=[64750], 60.00th=[64750],
     | 70.00th=[64750], 80.00th=[64750], 90.00th=[64750], 95.00th=[64750],
     | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.84%, ctx=14, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=40.0MiB (41.9MB), run=77-77msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck