== sanity test 398c: run fio to test AIO ================= 07:07:00 (1747825620)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.075439 s, 556 MB/s
osc.lustre-OST0000-osc-ffff8800ab870800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=4783: Wed May 21 07:07:25 2025
  write: IOPS=109, BW=438KiB/s (449kB/s)(10.0MiB/23360msec)
    slat (usec): min=12, max=364, avg=67.01, stdev=39.77
    clat (msec): min=60, max=244, avg=145.85, stdev=26.77
     lat (msec): min=60, max=244, avg=145.92, stdev=26.78
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[   97], 10.00th=[  112], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  148], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  169], 90.00th=[  180], 95.00th=[  190],
     | 99.00th=[  207], 99.50th=[  211], 99.90th=[  224], 99.95th=[  224],
     | 99.99th=[  245]
   bw (  KiB/s): min=  352, max=  664, per=24.95%, avg=437.30, stdev=60.12, samples=46
   iops        : min=   88, max=  166, avg=109.26, stdev=15.06, samples=46
  lat (msec)   : 100=5.62%, 250=94.38%
  cpu          : usr=0.19%, sys=0.82%, ctx=2089, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4784: Wed May 21 07:07:25 2025
  write: IOPS=109, BW=439KiB/s (449kB/s)(10.0MiB/23341msec)
    slat (usec): min=11, max=681, avg=64.64, stdev=39.69
    clat (msec): min=61, max=230, avg=145.74, stdev=26.85
     lat (msec): min=61, max=230, avg=145.80, stdev=26.86
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[   97], 10.00th=[  112], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  148], 60.00th=[  155],
     | 70.00th=[  159], 80.00th=[  169], 90.00th=[  180], 95.00th=[  190],
     | 99.00th=[  205], 99.50th=[  211], 99.90th=[  226], 99.95th=[  230],
     | 99.99th=[  230]
   bw (  KiB/s): min=  344, max=  664, per=24.95%, avg=437.35, stdev=60.40, samples=46
   iops        : min=   86, max=  166, avg=109.30, stdev=15.09, samples=46
  lat (msec)   : 100=5.31%, 250=94.69%
  cpu          : usr=0.23%, sys=0.78%, ctx=1885, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4785: Wed May 21 07:07:25 2025
  write: IOPS=109, BW=438KiB/s (449kB/s)(10.0MiB/23360msec)
    slat (usec): min=12, max=384, avg=65.34, stdev=41.09
    clat (msec): min=63, max=233, avg=145.86, stdev=26.09
     lat (msec): min=63, max=233, avg=145.93, stdev=26.09
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[  102], 10.00th=[  113], 20.00th=[  125],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  148], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  169], 90.00th=[  180], 95.00th=[  188],
     | 99.00th=[  203], 99.50th=[  207], 99.90th=[  220], 99.95th=[  224],
     | 99.99th=[  234]
   bw (  KiB/s): min=  360, max=  640, per=24.94%, avg=437.20, stdev=55.54, samples=46
   iops        : min=   90, max=  160, avg=109.28, stdev=13.89, samples=46
  lat (msec)   : 100=4.73%, 250=95.27%
  cpu          : usr=0.24%, sys=0.75%, ctx=1881, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4786: Wed May 21 07:07:25 2025
  write: IOPS=109, BW=439KiB/s (450kB/s)(10.0MiB/23314msec)
    slat (usec): min=11, max=311, avg=64.73, stdev=37.30
    clat (msec): min=18, max=302, avg=145.56, stdev=29.06
     lat (msec): min=18, max=302, avg=145.63, stdev=29.06
    clat percentiles (msec):
     |  1.00th=[   66],  5.00th=[   96], 10.00th=[  111], 20.00th=[  124],
     | 30.00th=[  133], 40.00th=[  140], 50.00th=[  146], 60.00th=[  153],
     | 70.00th=[  161], 80.00th=[  167], 90.00th=[  180], 95.00th=[  190],
     | 99.00th=[  209], 99.50th=[  224], 99.90th=[  300], 99.95th=[  305],
     | 99.99th=[  305]
   bw (  KiB/s): min=  344, max=  784, per=25.07%, avg=439.41, stdev=70.71, samples=46
   iops        : min=   86, max=  196, avg=109.80, stdev=17.67, samples=46
  lat (msec)   : 20=0.04%, 50=0.12%, 100=6.05%, 250=93.48%, 500=0.31%
  cpu          : usr=0.17%, sys=0.86%, ctx=1821, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1753KiB/s (1796kB/s), 438KiB/s-439KiB/s (449kB/s-450kB/s), io=40.0MiB (41.9MB), run=23314-23360msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=4814: Wed May 21 07:07:40 2025
   read: IOPS=86, BW=346KiB/s (354kB/s)(5048KiB/14595msec)
    slat (usec): min=495, max=2624, avg=854.96, stdev=184.14
    clat (usec): min=860, max=142250, avg=61728.75, stdev=12316.27
     lat (usec): min=1736, max=143119, avg=62585.59, stdev=12325.31
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   46], 10.00th=[   50], 20.00th=[   54],
     | 30.00th=[   57], 40.00th=[   59], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   67], 80.00th=[   71], 90.00th=[   75], 95.00th=[   81],
     | 99.00th=[   91], 99.50th=[  101], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  216, max=  464, per=25.25%, avg=346.97, stdev=63.62, samples=29
   iops        : min=   54, max=  116, avg=86.69, stdev=15.96, samples=29
  write: IOPS=88, BW=356KiB/s (364kB/s)(5192KiB/14595msec)
    slat (usec): min=13, max=336, avg=58.17, stdev=32.10
    clat (msec): min=50, max=262, avg=118.76, stdev=26.27
     lat (msec): min=50, max=262, avg=118.82, stdev=26.26
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   79], 10.00th=[   87], 20.00th=[   97],
     | 30.00th=[  105], 40.00th=[  112], 50.00th=[  118], 60.00th=[  124],
     | 70.00th=[  131], 80.00th=[  140], 90.00th=[  153], 95.00th=[  165],
     | 99.00th=[  180], 99.50th=[  188], 99.90th=[  245], 99.95th=[  264],
     | 99.99th=[  264]
   bw (  KiB/s): min=  296, max=  432, per=24.81%, avg=354.97, stdev=25.50, samples=29
   iops        : min=   74, max=  108, avg=88.69, stdev= 6.37, samples=29
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.12%, 10=0.12%, 20=0.12%, 50=4.96%, 100=56.29%
  lat (msec)   : 250=38.32%, 500=0.04%
  cpu          : usr=0.14%, sys=1.84%, ctx=3231, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4815: Wed May 21 07:07:40 2025
   read: IOPS=85, BW=341KiB/s (349kB/s)(4976KiB/14595msec)
    slat (usec): min=457, max=2970, avg=866.21, stdev=210.23
    clat (usec): min=1773, max=98673, avg=61219.26, stdev=11516.37
     lat (usec): min=2943, max=99627, avg=62087.41, stdev=11529.37
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[45351], 10.00th=[49546], 20.00th=[53216],
     | 30.00th=[56361], 40.00th=[58459], 50.00th=[61080], 60.00th=[63177],
     | 70.00th=[65799], 80.00th=[69731], 90.00th=[74974], 95.00th=[80217],
     | 99.00th=[90702], 99.50th=[95945], 99.90th=[98042], 99.95th=[99091],
     | 99.99th=[99091]
   bw (  KiB/s): min=  208, max=  504, per=24.87%, avg=341.72, stdev=68.68, samples=29
   iops        : min=   52, max=  126, avg=85.38, stdev=17.22, samples=29
  write: IOPS=90, BW=361KiB/s (369kB/s)(5264KiB/14595msec)
    slat (usec): min=13, max=306, avg=58.16, stdev=31.94
    clat (msec): min=39, max=214, avg=118.53, stdev=26.46
     lat (msec): min=39, max=214, avg=118.59, stdev=26.46
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   77], 10.00th=[   86], 20.00th=[   97],
     | 30.00th=[  105], 40.00th=[  112], 50.00th=[  118], 60.00th=[  125],
     | 70.00th=[  131], 80.00th=[  140], 90.00th=[  155], 95.00th=[  163],
     | 99.00th=[  180], 99.50th=[  186], 99.90th=[  207], 99.95th=[  215],
     | 99.99th=[  215]
   bw (  KiB/s): min=  312, max=  456, per=25.09%, avg=359.10, stdev=36.01, samples=29
   iops        : min=   78, max=  114, avg=89.72, stdev= 9.02, samples=29
  lat (msec)   : 2=0.04%, 10=0.20%, 20=0.16%, 50=5.51%, 100=55.00%
  lat (msec)   : 250=39.10%
  cpu          : usr=0.29%, sys=1.67%, ctx=3302, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4816: Wed May 21 07:07:40 2025
   read: IOPS=85, BW=341KiB/s (350kB/s)(4984KiB/14595msec)
    slat (usec): min=526, max=2684, avg=863.95, stdev=207.35
    clat (msec): min=9, max=100, avg=61.75, stdev=11.07
     lat (msec): min=10, max=101, avg=62.62, stdev=11.08
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   46], 10.00th=[   50], 20.00th=[   53],
     | 30.00th=[   56], 40.00th=[   59], 50.00th=[   62], 60.00th=[   64],
     | 70.00th=[   67], 80.00th=[   71], 90.00th=[   78], 95.00th=[   81],
     | 99.00th=[   88], 99.50th=[   92], 99.90th=[  101], 99.95th=[  101],
     | 99.99th=[  101]
   bw (  KiB/s): min=  248, max=  480, per=24.97%, avg=343.10, stdev=64.97, samples=29
   iops        : min=   62, max=  120, avg=85.72, stdev=16.30, samples=29
  write: IOPS=90, BW=360KiB/s (369kB/s)(5256KiB/14595msec)
    slat (usec): min=12, max=462, avg=59.76, stdev=35.12
    clat (msec): min=38, max=223, avg=118.09, stdev=25.89
     lat (msec): min=38, max=224, avg=118.15, stdev=25.89
    clat percentiles (msec):
     |  1.00th=[   65],  5.00th=[   79], 10.00th=[   86], 20.00th=[   96],
     | 30.00th=[  104], 40.00th=[  111], 50.00th=[  118], 60.00th=[  124],
     | 70.00th=[  130], 80.00th=[  140], 90.00th=[  155], 95.00th=[  165],
     | 99.00th=[  178], 99.50th=[  186], 99.90th=[  188], 99.95th=[  224],
     | 99.99th=[  224]
   bw (  KiB/s): min=  280, max=  416, per=25.06%, avg=358.55, stdev=32.81, samples=29
   iops        : min=   70, max=  104, avg=89.59, stdev= 8.22, samples=29
  lat (msec)   : 10=0.04%, 20=0.08%, 50=5.78%, 100=55.82%, 250=38.28%
  cpu          : usr=0.25%, sys=1.73%, ctx=3175, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4817: Wed May 21 07:07:40 2025
   read: IOPS=87, BW=348KiB/s (356kB/s)(5060KiB/14540msec)
    slat (usec): min=29, max=2714, avg=865.03, stdev=222.37
    clat (usec): min=1769, max=149020, avg=61770.22, stdev=12226.44
     lat (usec): min=1805, max=149972, avg=62637.15, stdev=12254.16
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   45], 10.00th=[   50], 20.00th=[   53],
     | 30.00th=[   56], 40.00th=[   59], 50.00th=[   62], 60.00th=[   65],
     | 70.00th=[   67], 80.00th=[   71], 90.00th=[   75], 95.00th=[   82],
     | 99.00th=[   92], 99.50th=[  100], 99.90th=[  148], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  216, max=  488, per=25.31%, avg=347.79, stdev=67.80, samples=29
   iops        : min=   54, max=  122, avg=86.90, stdev=16.97, samples=29
  write: IOPS=89, BW=356KiB/s (365kB/s)(5180KiB/14540msec)
    slat (usec): min=13, max=916, avg=58.69, stdev=41.26
    clat (msec): min=13, max=238, avg=118.25, stdev=27.41
     lat (msec): min=13, max=239, avg=118.31, stdev=27.41
    clat percentiles (msec):
     |  1.00th=[   53],  5.00th=[   78], 10.00th=[   86], 20.00th=[   96],
     | 30.00th=[  105], 40.00th=[  112], 50.00th=[  118], 60.00th=[  125],
     | 70.00th=[  131], 80.00th=[  140], 90.00th=[  155], 95.00th=[  165],
     | 99.00th=[  182], 99.50th=[  209], 99.90th=[  230], 99.95th=[  239],
     | 99.99th=[  239]
   bw (  KiB/s): min=  280, max=  424, per=24.75%, avg=354.14, stdev=45.09, samples=29
   iops        : min=   70, max=  106, avg=88.48, stdev=11.31, samples=29
  lat (msec)   : 2=0.04%, 10=0.04%, 20=0.39%, 50=5.90%, 100=55.23%
  lat (msec)   : 250=38.40%
  cpu          : usr=0.22%, sys=1.76%, ctx=3234, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1375KiB/s (1408kB/s), 341KiB/s-348KiB/s (349kB/s-356kB/s), io=19.6MiB (20.5MB), run=14540-14595msec
  WRITE: bw=1431KiB/s (1466kB/s), 356KiB/s-361KiB/s (364kB/s-369kB/s), io=20.4MiB (21.4MB), run=14540-14595msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=4832: Wed May 21 07:07:42 2025
   read: IOPS=11, BW=449MiB/s (471MB/s)(40.0MiB/89msec)
    slat (nsec): min=7983.1k, max=7983.1k, avg=7983148.00, stdev= 0.00
    clat (nsec): min=77407k, max=77407k, avg=77407151.00, stdev= 0.00
     lat (nsec): min=85405k, max=85405k, avg=85405453.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[77071],  5.00th=[77071], 10.00th=[77071], 20.00th=[77071],
     | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[77071],
     | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071],
     | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.95%, ctx=21, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=40.0MiB (41.9MB), run=89-89msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck