== sanity test 398c: run fio to test AIO ================= 20:18:04 (1718324284)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.133284 s, 315 MB/s
osc.lustre-OST0000-osc-ffff88009c398800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=843: Thu Jun 13 20:18:12 2024
  write: IOPS=401, BW=1604KiB/s (1643kB/s)(10.0MiB/6383msec)
    slat (usec): min=15, max=544, avg=75.04, stdev=41.39
    clat (usec): min=12154, max=55893, avg=39718.02, stdev=7473.43
     lat (usec): min=12290, max=55947, avg=39794.95, stdev=7485.80
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[22938], 10.00th=[26608], 20.00th=[34341],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44303], 80.00th=[45351], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[51643], 99.50th=[52691], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[55837]
   bw (  KiB/s): min= 1344, max= 2056, per=24.77%, avg=1589.42, stdev=230.13, samples=12
   iops        : min=  336, max=  514, avg=397.33, stdev=57.54, samples=12
  lat (msec)   : 20=0.78%, 50=96.45%, 100=2.77%
  cpu          : usr=0.53%, sys=3.67%, ctx=1120, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=846: Thu Jun 13 20:18:12 2024
  write: IOPS=401, BW=1607KiB/s (1645kB/s)(10.0MiB/6374msec)
    slat (usec): min=14, max=499, avg=82.90, stdev=44.81
    clat (usec): min=9108, max=54679, avg=39656.14, stdev=7515.58
     lat (usec): min=9136, max=54710, avg=39741.00, stdev=7526.38
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[23462], 10.00th=[26608], 20.00th=[34866],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[47449], 95.00th=[48497],
     | 99.00th=[51643], 99.50th=[53216], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1320, max= 2104, per=24.87%, avg=1596.08, stdev=243.63, samples=12
   iops        : min=  330, max=  526, avg=399.00, stdev=60.92, samples=12
  lat (msec)   : 10=0.04%, 20=1.72%, 50=95.94%, 100=2.30%
  cpu          : usr=0.66%, sys=3.81%, ctx=1703, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=847: Thu Jun 13 20:18:12 2024
  write: IOPS=401, BW=1607KiB/s (1645kB/s)(10.0MiB/6374msec)
    slat (usec): min=14, max=390, avg=79.25, stdev=43.18
    clat (usec): min=3567, max=54713, avg=39660.63, stdev=7555.55
     lat (usec): min=3724, max=54850, avg=39741.81, stdev=7567.59
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[23200], 10.00th=[26608], 20.00th=[34866],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[47449], 95.00th=[48497],
     | 99.00th=[51119], 99.50th=[53740], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1296, max= 2088, per=24.84%, avg=1594.08, stdev=243.36, samples=12
   iops        : min=  324, max=  522, avg=398.50, stdev=60.86, samples=12
  lat (msec)   : 4=0.04%, 10=0.08%, 20=1.64%, 50=96.05%, 100=2.19%
  cpu          : usr=0.55%, sys=3.78%, ctx=1342, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=848: Thu Jun 13 20:18:12 2024
  write: IOPS=401, BW=1605KiB/s (1643kB/s)(10.0MiB/6381msec)
    slat (usec): min=14, max=925, avg=87.08, stdev=51.72
    clat (usec): min=5035, max=54720, avg=39694.38, stdev=7513.48
     lat (usec): min=5143, max=54809, avg=39783.49, stdev=7524.13
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[22676], 10.00th=[26608], 20.00th=[34866],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44303], 80.00th=[44827], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[51643], 99.50th=[53216], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1360, max= 2075, per=24.85%, avg=1594.33, stdev=232.88, samples=12
   iops        : min=  340, max=  518, avg=398.50, stdev=58.09, samples=12
  lat (msec)   : 10=0.08%, 20=1.21%, 50=96.17%, 100=2.54%
  cpu          : usr=0.61%, sys=4.09%, ctx=1825, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6417KiB/s (6571kB/s), 1604KiB/s-1607KiB/s (1643kB/s-1645kB/s), io=40.0MiB (41.9MB), run=6374-6383msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=864: Thu Jun 13 20:18:17 2024
   read: IOPS=301, BW=1206KiB/s (1235kB/s)(5048KiB/4185msec)
    slat (usec): min=20, max=3279, avg=1702.52, stdev=347.32
    clat (usec): min=13250, max=45311, avg=24832.61, stdev=4187.57
     lat (usec): min=14633, max=47193, avg=26538.68, stdev=4279.94
    clat percentiles (usec):
     |  1.00th=[16581],  5.00th=[19006], 10.00th=[20055], 20.00th=[21627],
     | 30.00th=[22414], 40.00th=[23462], 50.00th=[24249], 60.00th=[25297],
     | 70.00th=[26608], 80.00th=[27919], 90.00th=[30016], 95.00th=[31851],
     | 99.00th=[39060], 99.50th=[39060], 99.90th=[43779], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min= 1040, max= 1368, per=25.07%, avg=1201.00, stdev=91.16, samples=8
   iops        : min=  260, max=  342, avg=300.25, stdev=22.79, samples=8
  write: IOPS=310, BW=1241KiB/s (1270kB/s)(5192KiB/4185msec)
    slat (usec): min=19, max=532, avg=85.54, stdev=41.01
    clat (usec): min=10636, max=43444, avg=25648.24, stdev=3438.99
     lat (usec): min=10665, max=43508, avg=25736.01, stdev=3439.53
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[21103], 10.00th=[21890], 20.00th=[22938],
     | 30.00th=[23987], 40.00th=[24773], 50.00th=[25297], 60.00th=[26084],
     | 70.00th=[26870], 80.00th=[27919], 90.00th=[29492], 95.00th=[31851],
     | 99.00th=[36439], 99.50th=[38536], 99.90th=[40109], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  984, max= 1376, per=24.78%, avg=1236.00, stdev=122.08, samples=8
   iops        : min=  246, max=  344, avg=309.00, stdev=30.52, samples=8
  lat (msec)   : 20=6.88%, 50=93.12%
  cpu          : usr=1.15%, sys=9.70%, ctx=2227, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=865: Thu Jun 13 20:18:17 2024
   read: IOPS=298, BW=1193KiB/s (1222kB/s)(4976KiB/4171msec)
    slat (usec): min=18, max=3169, avg=1719.97, stdev=337.90
    clat (usec): min=12844, max=44225, avg=24754.45, stdev=4003.99
     lat (usec): min=14733, max=45815, avg=26477.86, stdev=4089.29
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[19268], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22414], 40.00th=[23200], 50.00th=[24249], 60.00th=[25297],
     | 70.00th=[26346], 80.00th=[27919], 90.00th=[29492], 95.00th=[31851],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[41681], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min=  928, max= 1288, per=24.69%, avg=1183.00, stdev=119.69, samples=8
   iops        : min=  232, max=  322, avg=295.75, stdev=29.92, samples=8
  write: IOPS=315, BW=1262KiB/s (1292kB/s)(5264KiB/4171msec)
    slat (usec): min=20, max=1788, avg=87.57, stdev=74.38
    clat (usec): min=10399, max=40624, avg=25541.68, stdev=3359.18
     lat (usec): min=10563, max=40663, avg=25631.31, stdev=3359.26
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[20841], 10.00th=[21890], 20.00th=[22938],
     | 30.00th=[23987], 40.00th=[24511], 50.00th=[25297], 60.00th=[26084],
     | 70.00th=[26870], 80.00th=[27919], 90.00th=[29230], 95.00th=[31327],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1080, max= 1416, per=25.24%, avg=1259.00, stdev=110.35, samples=8
   iops        : min=  270, max=  354, avg=314.75, stdev=27.59, samples=8
  lat (msec)   : 20=6.05%, 50=93.95%
  cpu          : usr=1.13%, sys=9.52%, ctx=2224, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=866: Thu Jun 13 20:18:17 2024
   read: IOPS=297, BW=1190KiB/s (1219kB/s)(4984KiB/4188msec)
    slat (usec): min=19, max=3260, avg=1708.97, stdev=347.60
    clat (usec): min=7243, max=42500, avg=24828.52, stdev=4028.31
     lat (usec): min=9164, max=45071, avg=26541.19, stdev=4112.89
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[19006], 10.00th=[20055], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24511], 60.00th=[25297],
     | 70.00th=[26608], 80.00th=[27919], 90.00th=[30016], 95.00th=[31851],
     | 99.00th=[36439], 99.50th=[38536], 99.90th=[42206], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1000, max= 1296, per=24.78%, avg=1187.00, stdev=105.26, samples=8
   iops        : min=  250, max=  324, avg=296.75, stdev=26.32, samples=8
  write: IOPS=313, BW=1255KiB/s (1285kB/s)(5256KiB/4188msec)
    slat (usec): min=21, max=1818, avg=84.02, stdev=60.02
    clat (usec): min=3732, max=38863, avg=25703.73, stdev=3395.56
     lat (usec): min=3798, max=38923, avg=25789.69, stdev=3393.56
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[20841], 10.00th=[21890], 20.00th=[23200],
     | 30.00th=[23987], 40.00th=[24773], 50.00th=[25560], 60.00th=[26346],
     | 70.00th=[27132], 80.00th=[27919], 90.00th=[29230], 95.00th=[31327],
     | 99.00th=[37487], 99.50th=[38011], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1024, max= 1376, per=24.86%, avg=1240.00, stdev=110.27, samples=8
   iops        : min=  256, max=  344, avg=310.00, stdev=27.57, samples=8
  lat (msec)   : 4=0.04%, 10=0.04%, 20=5.74%, 50=94.18%
  cpu          : usr=1.10%, sys=9.46%, ctx=2178, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=867: Thu Jun 13 20:18:17 2024
   read: IOPS=302, BW=1209KiB/s (1238kB/s)(5060KiB/4185msec)
    slat (usec): min=19, max=3342, avg=1705.31, stdev=352.63
    clat (usec): min=2097, max=46314, avg=24814.88, stdev=4223.36
     lat (usec): min=3645, max=48772, avg=26523.63, stdev=4309.13
    clat percentiles (usec):
     |  1.00th=[16450],  5.00th=[19006], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22414], 40.00th=[23462], 50.00th=[24249], 60.00th=[25297],
     | 70.00th=[26608], 80.00th=[28181], 90.00th=[30016], 95.00th=[31851],
     | 99.00th=[38011], 99.50th=[39060], 99.90th=[41157], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  992, max= 1272, per=25.03%, avg=1199.00, stdev=89.95, samples=8
   iops        : min=  248, max=  318, avg=299.75, stdev=22.49, samples=8
  write: IOPS=309, BW=1238KiB/s (1267kB/s)(5180KiB/4185msec)
    slat (usec): min=20, max=452, avg=83.18, stdev=42.80
    clat (usec): min=10665, max=50380, avg=25666.25, stdev=3554.47
     lat (usec): min=10771, max=50453, avg=25751.29, stdev=3552.91
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[20841], 10.00th=[21890], 20.00th=[22938],
     | 30.00th=[23987], 40.00th=[24773], 50.00th=[25297], 60.00th=[26084],
     | 70.00th=[26870], 80.00th=[28181], 90.00th=[29492], 95.00th=[31851],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[45351], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min= 1016, max= 1368, per=24.84%, avg=1239.00, stdev=107.96, samples=8
   iops        : min=  254, max=  342, avg=309.75, stdev=26.99, samples=8
  lat (msec)   : 4=0.04%, 20=6.56%, 50=93.36%, 100=0.04%
  cpu          : usr=1.34%, sys=9.11%, ctx=2232, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4792KiB/s (4907kB/s), 1190KiB/s-1209KiB/s (1219kB/s-1238kB/s), io=19.6MiB (20.5MB), run=4171-4188msec
  WRITE: bw=4989KiB/s (5108kB/s), 1238KiB/s-1262KiB/s (1267kB/s-1292kB/s), io=20.4MiB (21.4MB), run=4171-4188msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=879: Thu Jun 13 20:18:19 2024
   read: IOPS=6, BW=265MiB/s (278MB/s)(40.0MiB/151msec)
    slat (nsec): min=5587.5k, max=5587.5k, avg=5587541.00, stdev= 0.00
    clat (nsec): min=144794k, max=144794k, avg=144793828.00, stdev= 0.00
     lat (nsec): min=150390k, max=150390k, avg=150390309.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  146],  5.00th=[  146], 10.00th=[  146], 20.00th=[  146],
     | 30.00th=[  146], 40.00th=[  146], 50.00th=[  146], 60.00th=[  146],
     | 70.00th=[  146], 80.00th=[  146], 90.00th=[  146], 95.00th=[  146],
     | 99.00th=[  146], 99.50th=[  146], 99.90th=[  146], 99.95th=[  146],
     | 99.99th=[  146]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.33%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=40.0MiB (41.9MB), run=151-151msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout