== sanity test 398c: run fio to test AIO ================= 03:02:37 (1735372957)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.139186 s, 301 MB/s
osc.lustre-OST0000-osc-ffff88012bfc4000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=26274: Sat Dec 28 03:02:44 2024
  write: IOPS=421, BW=1686KiB/s (1727kB/s)(10.0MiB/6073msec)
    slat (usec): min=13, max=582, avg=86.56, stdev=53.61
    clat (usec): min=3177, max=87602, avg=37757.82, stdev=8778.67
     lat (usec): min=3330, max=87663, avg=37846.44, stdev=8785.68
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[20317], 10.00th=[22152], 20.00th=[30802],
     | 30.00th=[35914], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44827], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[52691], 99.90th=[74974], 99.95th=[80217],
     | 99.99th=[87557]
   bw (  KiB/s): min= 1416, max= 2488, per=25.11%, avg=1688.00, stdev=376.84, samples=12
   iops        : min=  354, max=  622, avg=422.00, stdev=94.21, samples=12
  lat (msec)   : 4=0.08%, 10=0.08%, 20=3.63%, 50=95.62%, 100=0.59%
  cpu          : usr=0.63%, sys=4.28%, ctx=1590, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26275: Sat Dec 28 03:02:44 2024
  write: IOPS=421, BW=1685KiB/s (1725kB/s)(10.0MiB/6078msec)
    slat (usec): min=12, max=2034, avg=86.01, stdev=69.62
    clat (usec): min=4473, max=80033, avg=37740.55, stdev=8629.33
     lat (usec): min=4602, max=80155, avg=37828.49, stdev=8636.57
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[20055], 10.00th=[21890], 20.00th=[30802],
     | 30.00th=[35914], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[74974], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1424, max= 2448, per=25.10%, avg=1687.33, stdev=366.30, samples=12
   iops        : min=  356, max=  612, avg=421.83, stdev=91.57, samples=12
  lat (msec)   : 10=0.08%, 20=4.22%, 50=95.23%, 100=0.47%
  cpu          : usr=0.56%, sys=4.21%, ctx=1261, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26276: Sat Dec 28 03:02:44 2024
  write: IOPS=420, BW=1682KiB/s (1722kB/s)(10.0MiB/6088msec)
    slat (usec): min=12, max=675, avg=75.68, stdev=47.35
    clat (usec): min=16789, max=72461, avg=37859.36, stdev=8428.37
     lat (usec): min=16828, max=72554, avg=37936.99, stdev=8442.25
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[20579], 10.00th=[22414], 20.00th=[30802],
     | 30.00th=[35914], 40.00th=[39584], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44827], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[63701], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1408, max= 2464, per=24.98%, avg=1679.33, stdev=374.79, samples=12
   iops        : min=  352, max=  616, avg=419.83, stdev=93.70, samples=12
  lat (msec)   : 20=3.83%, 50=95.86%, 100=0.31%
  cpu          : usr=0.76%, sys=3.53%, ctx=1101, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26277: Sat Dec 28 03:02:44 2024
  write: IOPS=420, BW=1681KiB/s (1721kB/s)(10.0MiB/6092msec)
    slat (usec): min=13, max=499, avg=77.91, stdev=47.53
    clat (usec): min=13423, max=67900, avg=37809.57, stdev=8322.58
     lat (usec): min=13460, max=67926, avg=37889.21, stdev=8336.52
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[20317], 10.00th=[22938], 20.00th=[30802],
     | 30.00th=[35914], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[45876], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[48497], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1416, max= 2440, per=24.97%, avg=1678.67, stdev=349.63, samples=12
   iops        : min=  354, max=  610, avg=419.67, stdev=87.41, samples=12
  lat (msec)   : 20=3.44%, 50=96.37%, 100=0.20%
  cpu          : usr=0.84%, sys=3.76%, ctx=1182, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6724KiB/s (6885kB/s), 1681KiB/s-1686KiB/s (1721kB/s-1727kB/s), io=40.0MiB (41.9MB), run=6073-6092msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=26291: Sat Dec 28 03:02:48 2024
   read: IOPS=452, BW=1811KiB/s (1854kB/s)(5048KiB/2788msec)
    slat (usec): min=590, max=3831, avg=1138.71, stdev=403.33
    clat (usec): min=9754, max=45834, avg=16495.96, stdev=4800.67
     lat (usec): min=10782, max=47420, avg=17636.45, stdev=5070.29
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[11863], 10.00th=[12518], 20.00th=[13435],
     | 30.00th=[14222], 40.00th=[14746], 50.00th=[15401], 60.00th=[16057],
     | 70.00th=[16909], 80.00th=[17957], 90.00th=[21103], 95.00th=[27657],
     | 99.00th=[35914], 99.50th=[38536], 99.90th=[44303], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min= 1112, max= 2040, per=24.95%, avg=1793.60, stdev=387.58, samples=5
   iops        : min=  278, max=  510, avg=448.40, stdev=96.90, samples=5
  write: IOPS=465, BW=1862KiB/s (1907kB/s)(5192KiB/2788msec)
    slat (usec): min=12, max=238, avg=40.89, stdev=20.58
    clat (usec): min=10336, max=48720, avg=16821.59, stdev=4688.58
     lat (usec): min=10413, max=48777, avg=16863.43, stdev=4696.46
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13042], 10.00th=[13698], 20.00th=[14222],
     | 30.00th=[14746], 40.00th=[15139], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17433], 90.00th=[20055], 95.00th=[29492],
     | 99.00th=[34866], 99.50th=[36439], 99.90th=[45876], 99.95th=[48497],
     | 99.99th=[48497]
   bw (  KiB/s): min= 1040, max= 2168, per=25.16%, avg=1883.20, stdev=473.85, samples=5
   iops        : min=  260, max=  542, avg=470.80, stdev=118.46, samples=5
  lat (msec)   : 10=0.08%, 20=89.26%, 50=10.66%
  cpu          : usr=0.93%, sys=7.39%, ctx=2251, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26292: Sat Dec 28 03:02:48 2024
   read: IOPS=455, BW=1822KiB/s (1866kB/s)(4976KiB/2731msec)
    slat (usec): min=65, max=3559, avg=1116.78, stdev=395.85
    clat (usec): min=1177, max=40602, avg=16175.26, stdev=4580.64
     lat (usec): min=2176, max=42630, avg=17293.83, stdev=4832.80
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11863], 10.00th=[12518], 20.00th=[13304],
     | 30.00th=[14091], 40.00th=[14615], 50.00th=[15270], 60.00th=[15795],
     | 70.00th=[16319], 80.00th=[17433], 90.00th=[20317], 95.00th=[27657],
     | 99.00th=[34866], 99.50th=[36439], 99.90th=[40109], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1048, max= 2040, per=25.01%, avg=1798.40, stdev=421.82, samples=5
   iops        : min=  262, max=  510, avg=449.60, stdev=105.46, samples=5
  write: IOPS=481, BW=1927KiB/s (1974kB/s)(5264KiB/2731msec)
    slat (usec): min=12, max=1349, avg=44.85, stdev=60.82
    clat (usec): min=9000, max=41629, avg=16719.21, stdev=4562.90
     lat (usec): min=9059, max=41706, avg=16765.12, stdev=4572.20
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[12911], 10.00th=[13435], 20.00th=[14091],
     | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17433], 90.00th=[20317], 95.00th=[28443],
     | 99.00th=[35390], 99.50th=[37487], 99.90th=[39060], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1288, max= 2112, per=25.76%, avg=1928.00, stdev=358.89, samples=5
   iops        : min=  322, max=  528, avg=482.00, stdev=89.72, samples=5
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.23%, 20=89.41%, 50=10.20%
  cpu          : usr=1.03%, sys=7.33%, ctx=2201, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26293: Sat Dec 28 03:02:48 2024
   read: IOPS=446, BW=1786KiB/s (1829kB/s)(4984KiB/2791msec)
    slat (usec): min=605, max=3312, avg=1111.37, stdev=379.20
    clat (usec): min=4293, max=47180, avg=16392.10, stdev=4725.65
     lat (usec): min=5078, max=48705, avg=17505.26, stdev=4983.77
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12125], 10.00th=[12649], 20.00th=[13435],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15533], 60.00th=[16057],
     | 70.00th=[16581], 80.00th=[17433], 90.00th=[20055], 95.00th=[27919],
     | 99.00th=[35390], 99.50th=[38536], 99.90th=[42730], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1080, max= 2048, per=24.83%, avg=1785.60, stdev=399.01, samples=5
   iops        : min=  270, max=  512, avg=446.40, stdev=99.75, samples=5
  write: IOPS=470, BW=1883KiB/s (1928kB/s)(5256KiB/2791msec)
    slat (usec): min=12, max=996, avg=43.45, stdev=34.94
    clat (usec): min=11075, max=47055, avg=16980.42, stdev=4782.50
     lat (usec): min=11127, max=47082, avg=17024.89, stdev=4792.32
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13042], 10.00th=[13698], 20.00th=[14353],
     | 30.00th=[14746], 40.00th=[15270], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[20579], 95.00th=[29492],
     | 99.00th=[34866], 99.50th=[37487], 99.90th=[45876], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1040, max= 2176, per=25.07%, avg=1876.80, stdev=473.34, samples=5
   iops        : min=  260, max=  544, avg=469.20, stdev=118.34, samples=5
  lat (msec)   : 10=0.27%, 20=89.02%, 50=10.70%
  cpu          : usr=0.75%, sys=7.53%, ctx=2211, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26294: Sat Dec 28 03:02:48 2024
   read: IOPS=454, BW=1819KiB/s (1863kB/s)(5060KiB/2781msec)
    slat (usec): min=611, max=3221, avg=1121.14, stdev=374.26
    clat (usec): min=4950, max=43841, avg=16260.74, stdev=4591.76
     lat (usec): min=6418, max=45533, avg=17383.77, stdev=4849.90
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[11600], 10.00th=[12518], 20.00th=[13173],
     | 30.00th=[13960], 40.00th=[14615], 50.00th=[15401], 60.00th=[15926],
     | 70.00th=[16581], 80.00th=[17695], 90.00th=[20579], 95.00th=[27132],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[42206], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1096, max= 2072, per=25.30%, avg=1819.20, stdev=408.81, samples=5
   iops        : min=  274, max=  518, avg=454.80, stdev=102.20, samples=5
  write: IOPS=465, BW=1863KiB/s (1907kB/s)(5180KiB/2781msec)
    slat (usec): min=12, max=356, avg=41.81, stdev=26.74
    clat (usec): min=10590, max=45659, avg=16890.06, stdev=4535.22
     lat (usec): min=10628, max=45704, avg=16932.89, stdev=4547.30
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13042], 10.00th=[13698], 20.00th=[14353],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[20055], 95.00th=[28967],
     | 99.00th=[34341], 99.50th=[35914], 99.90th=[45351], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min= 1056, max= 2096, per=24.69%, avg=1848.00, stdev=445.35, samples=5
   iops        : min=  264, max=  524, avg=462.00, stdev=111.34, samples=5
  lat (msec)   : 10=0.20%, 20=89.34%, 50=10.47%
  cpu          : usr=0.83%, sys=7.55%, ctx=2219, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7190KiB/s (7363kB/s), 1786KiB/s-1822KiB/s (1829kB/s-1866kB/s), io=19.6MiB (20.5MB), run=2731-2791msec
  WRITE: bw=7485KiB/s (7665kB/s), 1862KiB/s-1927KiB/s (1907kB/s-1974kB/s), io=20.4MiB (21.4MB), run=2731-2791msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=26300: Sat Dec 28 03:02:50 2024
   read: IOPS=6, BW=261MiB/s (274MB/s)(40.0MiB/153msec)
    slat (nsec): min=6296.0k, max=6296.0k, avg=6296984.00, stdev= 0.00
    clat (nsec): min=138360k, max=138360k, avg=138360281.00, stdev= 0.00
     lat (nsec): min=144666k, max=144666k, avg=144666230.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  138],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  138], 40.00th=[  138], 50.00th=[  138], 60.00th=[  138],
     | 70.00th=[  138], 80.00th=[  138], 90.00th=[  138], 95.00th=[  138],
     | 99.00th=[  138], 99.50th=[  138], 99.90th=[  138], 99.95th=[  138],
     | 99.99th=[  138]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.66%, sys=3.29%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=261MiB/s (274MB/s), 261MiB/s-261MiB/s (274MB/s-274MB/s), io=40.0MiB (41.9MB), run=153-153msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout