== sanity test 398c: run fio to test AIO ================= 03:10:42 (1715584242)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.149602 s, 280 MB/s
osc.lustre-OST0000-osc-ffff88012ab39800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=17950: Mon May 13 03:10:51 2024
  write: IOPS=372, BW=1491KiB/s (1527kB/s)(10.0MiB/6868msec)
    slat (usec): min=25, max=477, avg=79.12, stdev=42.00
    clat (usec): min=20735, max=79213, avg=42733.35, stdev=3956.63
     lat (usec): min=20840, max=79392, avg=42814.44, stdev=3958.25
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[34866], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[41681], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[47973], 99.50th=[51643], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1400, max= 1688, per=24.97%, avg=1488.77, stdev=88.23, samples=13
   iops        : min=  350, max=  422, avg=372.15, stdev=22.08, samples=13
  lat (msec)   : 50=99.34%, 100=0.66%
  cpu          : usr=0.41%, sys=3.70%, ctx=1209, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17951: Mon May 13 03:10:51 2024
  write: IOPS=374, BW=1496KiB/s (1532kB/s)(10.0MiB/6843msec)
    slat (usec): min=26, max=405, avg=84.84, stdev=43.27
    clat (usec): min=5216, max=83345, avg=42579.57, stdev=4259.60
     lat (usec): min=5288, max=83384, avg=42666.21, stdev=4259.03
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[34341], 10.00th=[36963], 20.00th=[40633],
     | 30.00th=[41681], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[47973], 99.50th=[48497], 99.90th=[73925], 99.95th=[79168],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1368, max= 1792, per=25.03%, avg=1492.46, stdev=107.06, samples=13
   iops        : min=  342, max=  448, avg=373.08, stdev=26.79, samples=13
  lat (msec)   : 10=0.12%, 20=0.16%, 50=99.34%, 100=0.39%
  cpu          : usr=0.67%, sys=3.89%, ctx=1500, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17952: Mon May 13 03:10:51 2024
  write: IOPS=374, BW=1496KiB/s (1532kB/s)(10.0MiB/6843msec)
    slat (usec): min=26, max=533, avg=83.18, stdev=45.37
    clat (usec): min=6489, max=78919, avg=42582.57, stdev=4087.30
     lat (usec): min=6609, max=79004, avg=42667.66, stdev=4084.12
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[34341], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[41681], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[47973], 99.50th=[50070], 99.90th=[60031], 99.95th=[64750],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1376, max= 1792, per=25.07%, avg=1495.08, stdev=104.79, samples=13
   iops        : min=  344, max=  448, avg=373.77, stdev=26.20, samples=13
  lat (msec)   : 10=0.08%, 20=0.12%, 50=99.30%, 100=0.51%
  cpu          : usr=0.60%, sys=3.71%, ctx=1307, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=17953: Mon May 13 03:10:51 2024
  write: IOPS=372, BW=1492KiB/s (1527kB/s)(10.0MiB/6865msec)
    slat (usec): min=18, max=544, avg=74.90, stdev=43.76
    clat (usec): min=9906, max=81561, avg=42718.79, stdev=3966.88
     lat (usec): min=10061, max=81608, avg=42795.43, stdev=3967.73
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[34866], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[41681], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303],
     | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[46924],
     | 99.00th=[48497], 99.50th=[51119], 99.90th=[58459], 99.95th=[77071],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1408, max= 1688, per=24.96%, avg=1488.38, stdev=89.24, samples=13
   iops        : min=  352, max=  422, avg=372.08, stdev=22.32, samples=13
  lat (msec)   : 10=0.04%, 50=99.41%, 100=0.55%
  cpu          : usr=0.51%, sys=3.25%, ctx=1014, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5964KiB/s (6107kB/s), 1491KiB/s-1496KiB/s (1527kB/s-1532kB/s), io=40.0MiB (41.9MB), run=6843-6868msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=17968: Mon May 13 03:10:55 2024
   read: IOPS=425, BW=1702KiB/s (1743kB/s)(5048KiB/2966msec)
    slat (usec): min=661, max=3115, avg=1364.28, stdev=361.52
    clat (usec): min=9431, max=40142, avg=17410.19, stdev=3978.39
     lat (usec): min=10810, max=42633, avg=18777.31, stdev=4151.38
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12518], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[16188], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[21627], 95.00th=[25297],
     | 99.00th=[32113], 99.50th=[33817], 99.90th=[37487], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1272, max= 1944, per=25.07%, avg=1696.00, stdev=265.09, samples=5
   iops        : min=  318, max=  486, avg=424.00, stdev=66.27, samples=5
  write: IOPS=437, BW=1751KiB/s (1793kB/s)(5192KiB/2966msec)
    slat (usec): min=15, max=1184, avg=71.42, stdev=53.41
    clat (usec): min=10400, max=37971, avg=17913.93, stdev=3959.67
     lat (usec): min=10433, max=38010, avg=17986.98, stdev=3962.40
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18744], 80.00th=[20055], 90.00th=[21890], 95.00th=[24511],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1304, max= 2080, per=25.33%, avg=1784.00, stdev=310.82, samples=5
   iops        : min=  326, max=  520, avg=446.00, stdev=77.70, samples=5
  lat (msec)   : 10=0.12%, 20=81.60%, 50=18.28%
  cpu          : usr=1.38%, sys=10.52%, ctx=2431, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17969: Mon May 13 03:10:55 2024
   read: IOPS=419, BW=1678KiB/s (1718kB/s)(4976KiB/2966msec)
    slat (usec): min=701, max=3142, avg=1363.53, stdev=362.65
    clat (usec): min=9833, max=38493, avg=17523.48, stdev=4348.73
     lat (usec): min=10894, max=41248, avg=18889.96, stdev=4525.89
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12518], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15270], 40.00th=[16057], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[21627], 95.00th=[25560],
     | 99.00th=[34866], 99.50th=[36963], 99.90th=[38011], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1152, max= 1944, per=24.88%, avg=1683.20, stdev=319.23, samples=5
   iops        : min=  288, max=  486, avg=420.80, stdev=79.81, samples=5
  write: IOPS=443, BW=1775KiB/s (1817kB/s)(5264KiB/2966msec)
    slat (usec): min=15, max=1498, avg=73.26, stdev=56.46
    clat (usec): min=10567, max=38056, avg=17873.69, stdev=3816.34
     lat (usec): min=10600, max=38137, avg=17948.72, stdev=3819.15
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13566], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[21627], 95.00th=[25560],
     | 99.00th=[32900], 99.50th=[34866], 99.90th=[36439], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1464, max= 2104, per=25.63%, avg=1804.80, stdev=254.78, samples=5
   iops        : min=  366, max=  526, avg=451.20, stdev=63.70, samples=5
  lat (msec)   : 10=0.04%, 20=81.91%, 50=18.05%
  cpu          : usr=1.25%, sys=10.86%, ctx=2498, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17970: Mon May 13 03:10:55 2024
   read: IOPS=420, BW=1683KiB/s (1723kB/s)(4984KiB/2962msec)
    slat (usec): min=29, max=3647, avg=1353.39, stdev=381.50
    clat (usec): min=3015, max=39041, avg=17405.50, stdev=4087.57
     lat (usec): min=3066, max=42692, avg=18761.72, stdev=4244.22
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[12125], 10.00th=[13173], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15926], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[21890], 95.00th=[23987],
     | 99.00th=[32900], 99.50th=[34866], 99.90th=[36963], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1336, max= 2096, per=24.98%, avg=1690.40, stdev=305.23, samples=5
   iops        : min=  334, max=  524, avg=422.60, stdev=76.31, samples=5
  write: IOPS=443, BW=1774KiB/s (1817kB/s)(5256KiB/2962msec)
    slat (usec): min=16, max=1115, avg=72.10, stdev=48.38
    clat (usec): min=9018, max=38645, avg=18073.90, stdev=4014.99
     lat (usec): min=9067, max=38708, avg=18147.91, stdev=4017.96
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13566], 10.00th=[14484], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[20055], 90.00th=[22152], 95.00th=[26608],
     | 99.00th=[33817], 99.50th=[35390], 99.90th=[36439], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1352, max= 2075, per=25.59%, avg=1802.20, stdev=270.06, samples=5
   iops        : min=  338, max=  518, avg=450.40, stdev=67.33, samples=5
  lat (msec)   : 4=0.04%, 10=0.31%, 20=80.00%, 50=19.65%
  cpu          : usr=1.01%, sys=11.01%, ctx=2396, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=17971: Mon May 13 03:10:55 2024
   read: IOPS=426, BW=1707KiB/s (1748kB/s)(5060KiB/2965msec)
    slat (usec): min=707, max=3712, avg=1370.17, stdev=372.78
    clat (usec): min=4756, max=39081, avg=17318.60, stdev=3968.26
     lat (usec): min=5786, max=42238, avg=18691.72, stdev=4140.54
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12387], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15926], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[21627], 95.00th=[25035],
     | 99.00th=[31589], 99.50th=[33817], 99.90th=[37487], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1317, max= 1968, per=25.34%, avg=1714.60, stdev=254.00, samples=5
   iops        : min=  329, max=  492, avg=428.60, stdev=63.60, samples=5
  write: IOPS=436, BW=1747KiB/s (1789kB/s)(5180KiB/2965msec)
    slat (usec): min=14, max=505, avg=70.78, stdev=40.82
    clat (usec): min=2273, max=38132, avg=17996.60, stdev=3907.83
     lat (usec): min=2399, max=38214, avg=18069.21, stdev=3914.34
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13698], 10.00th=[14353], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16581], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18744], 80.00th=[19792], 90.00th=[21627], 95.00th=[25822],
     | 99.00th=[33817], 99.50th=[34866], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 1277, max= 2072, per=25.09%, avg=1767.40, stdev=311.84, samples=5
   iops        : min=  319, max=  518, avg=441.80, stdev=78.06, samples=5
  lat (msec)   : 4=0.04%, 10=0.27%, 20=81.41%, 50=18.28%
  cpu          : usr=1.25%, sys=10.70%, ctx=2524, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6766KiB/s (6928kB/s), 1678KiB/s-1707KiB/s (1718kB/s-1748kB/s), io=19.6MiB (20.5MB), run=2962-2966msec
  WRITE: bw=7044KiB/s (7213kB/s), 1747KiB/s-1775KiB/s (1789kB/s-1817kB/s), io=20.4MiB (21.4MB), run=2962-2966msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=17977: Mon May 13 03:10:56 2024
   read: IOPS=7, BW=305MiB/s (320MB/s)(40.0MiB/131msec)
    slat (nsec): min=6091.7k, max=6091.7k, avg=6091666.00, stdev= 0.00
    clat (nsec): min=117805k, max=117805k, avg=117805218.00, stdev= 0.00
     lat (nsec): min=123907k, max=123907k, avg=123907019.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  118],  5.00th=[  118], 10.00th=[  118], 20.00th=[  118],
     | 30.00th=[  118], 40.00th=[  118], 50.00th=[  118], 60.00th=[  118],
     | 70.00th=[  118], 80.00th=[  118], 90.00th=[  118], 95.00th=[  118],
     | 99.00th=[  118], 99.50th=[  118], 99.90th=[  118], 99.95th=[  118],
     | 99.99th=[  118]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.85%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=305MiB/s (320MB/s), 305MiB/s-305MiB/s (320MB/s-320MB/s), io=40.0MiB (41.9MB), run=131-131msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout