== sanity test 398c: run fio to test AIO ================= 13:49:18 (1740682158)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0969818 s, 432 MB/s
osc.lustre-OST0000-osc-ffff8800885b7000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=31150: Thu Feb 27 13:49:26 2025
  write: IOPS=365, BW=1463KiB/s (1498kB/s)(10.0MiB/6998msec)
    slat (usec): min=24, max=438, avg=79.00, stdev=41.19
    clat (usec): min=12365, max=76109, avg=43432.85, stdev=5706.23
     lat (usec): min=12496, max=76261, avg=43513.62, stdev=5708.59
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[32900], 10.00th=[35914], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45351], 60.00th=[46400],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[49546], 99.50th=[52691], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1328, max= 1792, per=25.10%, avg=1467.46, stdev=141.30, samples=13
   iops        : min=  332, max=  448, avg=366.85, stdev=35.32, samples=13
  lat (msec)   : 20=0.74%, 50=98.55%, 100=0.70%
  cpu          : usr=0.61%, sys=3.40%, ctx=1385, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=31151: Thu Feb 27 13:49:26 2025
  write: IOPS=366, BW=1467KiB/s (1502kB/s)(10.0MiB/6980msec)
    slat (usec): min=24, max=3950, avg=83.54, stdev=88.14
    clat (usec): min=4169, max=76138, avg=43355.55, stdev=5644.60
     lat (usec): min=4271, max=76200, avg=43441.04, stdev=5642.02
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[32900], 10.00th=[35914], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45351], 60.00th=[46400],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[49546], 99.50th=[50070], 99.90th=[66323], 99.95th=[71828],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1336, max= 1792, per=25.14%, avg=1469.92, stdev=146.36, samples=13
   iops        : min=  334, max=  448, avg=367.46, stdev=36.59, samples=13
  lat (msec)   : 10=0.08%, 20=0.82%, 50=98.59%, 100=0.51%
  cpu          : usr=0.72%, sys=3.68%, ctx=1491, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=31152: Thu Feb 27 13:49:26 2025
  write: IOPS=367, BW=1468KiB/s (1503kB/s)(10.0MiB/6975msec)
    slat (usec): min=24, max=542, avg=81.26, stdev=44.76
    clat (usec): min=5122, max=71092, avg=43344.23, stdev=5854.65
     lat (usec): min=5170, max=71249, avg=43427.22, stdev=5855.37
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[32375], 10.00th=[35390], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45351], 60.00th=[46400],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[49546], 99.50th=[51643], 99.90th=[61604], 99.95th=[66323],
     | 99.99th=[70779]
   bw (  KiB/s): min= 1320, max= 1864, per=25.22%, avg=1474.54, stdev=151.86, samples=13
   iops        : min=  330, max=  466, avg=368.62, stdev=37.97, samples=13
  lat (msec)   : 10=0.08%, 20=0.59%, 50=98.48%, 100=0.86%
  cpu          : usr=0.63%, sys=3.67%, ctx=1444, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=31153: Thu Feb 27 13:49:26 2025
  write: IOPS=365, BW=1462KiB/s (1497kB/s)(10.0MiB/7005msec)
    slat (usec): min=25, max=2535, avg=80.15, stdev=64.47
    clat (usec): min=6893, max=53260, avg=43449.81, stdev=5273.20
     lat (usec): min=7080, max=53351, avg=43531.66, stdev=5267.46
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[32900], 10.00th=[35914], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45351], 60.00th=[46400],
     | 70.00th=[46924], 80.00th=[47449], 90.00th=[47973], 95.00th=[48497],
     | 99.00th=[49546], 99.50th=[50594], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1333, max= 1696, per=24.87%, avg=1454.07, stdev=117.27, samples=14
   iops        : min=  333, max=  424, avg=363.50, stdev=29.34, samples=14
  lat (msec)   : 10=0.20%, 20=0.16%, 50=98.91%, 100=0.74%
  cpu          : usr=0.57%, sys=3.54%, ctx=1355, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5847KiB/s (5988kB/s), 1462KiB/s-1468KiB/s (1497kB/s-1503kB/s), io=40.0MiB (41.9MB), run=6975-7005msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=31168: Thu Feb 27 13:49:31 2025
   read: IOPS=362, BW=1451KiB/s (1486kB/s)(5048KiB/3479msec)
    slat (usec): min=711, max=3072, avg=1349.74, stdev=343.92
    clat (usec): min=3568, max=39205, avg=20506.29, stdev=4061.81
     lat (usec): min=4693, max=41795, avg=21858.18, stdev=4244.79
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15664], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21365], 80.00th=[22414], 90.00th=[24511], 95.00th=[29754],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[38536], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min=  976, max= 1568, per=24.63%, avg=1419.50, stdev=221.82, samples=6
   iops        : min=  244, max=  392, avg=354.83, stdev=55.44, samples=6
  write: IOPS=373, BW=1492KiB/s (1528kB/s)(5192KiB/3479msec)
    slat (usec): min=14, max=1580, avg=50.80, stdev=52.17
    clat (usec): min=13716, max=37751, avg=21141.62, stdev=3538.64
     lat (usec): min=13759, max=37814, avg=21193.52, stdev=3545.80
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[17171], 10.00th=[17695], 20.00th=[18744],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[24511], 95.00th=[30016],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[35914], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min=  920, max= 1720, per=24.86%, avg=1491.33, stdev=286.67, samples=6
   iops        : min=  230, max=  430, avg=372.83, stdev=71.67, samples=6
  lat (msec)   : 4=0.04%, 20=45.90%, 50=54.06%
  cpu          : usr=0.92%, sys=6.87%, ctx=2580, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=31169: Thu Feb 27 13:49:31 2025
   read: IOPS=357, BW=1432KiB/s (1466kB/s)(4976KiB/3475msec)
    slat (usec): min=745, max=3100, avg=1341.16, stdev=358.68
    clat (usec): min=3601, max=39250, avg=20498.86, stdev=3898.38
     lat (usec): min=4955, max=41844, avg=21842.21, stdev=4081.24
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15664], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[18482], 40.00th=[19268], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21365], 80.00th=[22414], 90.00th=[24773], 95.00th=[28181],
     | 99.00th=[33817], 99.50th=[35914], 99.90th=[36439], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min=  928, max= 1632, per=24.35%, avg=1403.33, stdev=245.72, samples=6
   iops        : min=  232, max=  408, avg=350.83, stdev=61.43, samples=6
  write: IOPS=378, BW=1515KiB/s (1551kB/s)(5264KiB/3475msec)
    slat (usec): min=13, max=1251, avg=49.07, stdev=45.48
    clat (usec): min=14026, max=38845, avg=21288.72, stdev=3723.06
     lat (usec): min=14077, max=38914, avg=21338.76, stdev=3730.07
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[17171], 10.00th=[17957], 20.00th=[18744],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22676], 90.00th=[25035], 95.00th=[30540],
     | 99.00th=[35390], 99.50th=[37487], 99.90th=[38536], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1064, max= 1720, per=25.06%, avg=1503.33, stdev=231.76, samples=6
   iops        : min=  266, max=  430, avg=375.83, stdev=57.94, samples=6
  lat (msec)   : 4=0.04%, 10=0.08%, 20=44.34%, 50=55.55%
  cpu          : usr=0.86%, sys=6.76%, ctx=2511, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=31170: Thu Feb 27 13:49:31 2025
   read: IOPS=357, BW=1431KiB/s (1466kB/s)(4984KiB/3482msec)
    slat (usec): min=767, max=3116, avg=1352.14, stdev=342.31
    clat (usec): min=3759, max=38858, avg=20402.68, stdev=3928.21
     lat (usec): min=5175, max=40678, avg=21756.82, stdev=4096.12
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15533], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[19268], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[25035], 95.00th=[27919],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[39060], 99.95th=[39060],
     | 99.99th=[39060]
   bw (  KiB/s): min=  904, max= 1632, per=24.38%, avg=1404.83, stdev=263.83, samples=6
   iops        : min=  226, max=  408, avg=351.17, stdev=65.96, samples=6
  write: IOPS=377, BW=1509KiB/s (1546kB/s)(5256KiB/3482msec)
    slat (usec): min=13, max=485, avg=52.41, stdev=36.89
    clat (usec): min=13932, max=37620, avg=21238.64, stdev=3688.99
     lat (usec): min=13973, max=37752, avg=21292.16, stdev=3696.78
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17171], 10.00th=[17695], 20.00th=[18744],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[25035], 95.00th=[30278],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[36963], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min=  992, max= 1744, per=24.94%, avg=1496.67, stdev=263.64, samples=6
   iops        : min=  248, max=  436, avg=374.17, stdev=65.91, samples=6
  lat (msec)   : 4=0.08%, 10=0.04%, 20=44.45%, 50=55.43%
  cpu          : usr=1.03%, sys=6.81%, ctx=2529, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=31171: Thu Feb 27 13:49:31 2025
   read: IOPS=363, BW=1454KiB/s (1489kB/s)(5060KiB/3479msec)
    slat (usec): min=25, max=3034, avg=1330.69, stdev=364.37
    clat (usec): min=4301, max=40107, avg=20697.79, stdev=4021.19
     lat (usec): min=5637, max=42422, avg=22030.56, stdev=4183.18
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15533], 10.00th=[16450], 20.00th=[17695],
     | 30.00th=[18744], 40.00th=[19268], 50.00th=[20055], 60.00th=[21103],
     | 70.00th=[21890], 80.00th=[22938], 90.00th=[25035], 95.00th=[28705],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[38536], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1048, max= 1552, per=24.80%, avg=1429.33, stdev=191.31, samples=6
   iops        : min=  262, max=  388, avg=357.33, stdev=47.83, samples=6
  write: IOPS=372, BW=1489KiB/s (1525kB/s)(5180KiB/3479msec)
    slat (usec): min=13, max=333, avg=48.46, stdev=32.09
    clat (usec): min=9142, max=39452, avg=21303.75, stdev=3691.35
     lat (usec): min=9175, max=39508, avg=21353.26, stdev=3698.27
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17171], 10.00th=[17957], 20.00th=[18744],
     | 30.00th=[19530], 40.00th=[20055], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21890], 80.00th=[22676], 90.00th=[25297], 95.00th=[30016],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[39060], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 1048, max= 1608, per=24.64%, avg=1478.67, stdev=216.82, samples=6
   iops        : min=  262, max=  402, avg=369.67, stdev=54.21, samples=6
  lat (msec)   : 10=0.20%, 20=43.05%, 50=56.76%
  cpu          : usr=0.75%, sys=6.87%, ctx=2386, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5763KiB/s (5902kB/s), 1431KiB/s-1454KiB/s (1466kB/s-1489kB/s), io=19.6MiB (20.5MB), run=3475-3482msec
  WRITE: bw=6000KiB/s (6144kB/s), 1489KiB/s-1515KiB/s (1525kB/s-1551kB/s), io=20.4MiB (21.4MB), run=3475-3482msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=31178: Thu Feb 27 13:49:32 2025
   read: IOPS=6, BW=258MiB/s (271MB/s)(40.0MiB/155msec)
    slat (nsec): min=5569.2k, max=5569.2k, avg=5569234.00, stdev= 0.00
    clat (nsec): min=142136k, max=142136k, avg=142136194.00, stdev= 0.00
     lat (nsec): min=147714k, max=147714k, avg=147714447.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  142],  5.00th=[  142], 10.00th=[  142], 20.00th=[  142],
     | 30.00th=[  142], 40.00th=[  142], 50.00th=[  142], 60.00th=[  142],
     | 70.00th=[  142], 80.00th=[  142], 90.00th=[  142], 95.00th=[  142],
     | 99.00th=[  142], 99.50th=[  142], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.90%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=258MiB/s (271MB/s), 258MiB/s-258MiB/s (271MB/s-271MB/s), io=40.0MiB (41.9MB), run=155-155msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout