== sanity test 398c: run fio to test AIO ================= 12:46:48 (1737136008)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.103817 s, 404 MB/s
osc.lustre-OST0000-osc-ffff880085bc9000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=21388: Fri Jan 17 12:46:55 2025
  write: IOPS=433, BW=1736KiB/s (1778kB/s)(10.0MiB/5899msec)
    slat (usec): min=12, max=1063, avg=61.72, stdev=79.27
    clat (usec): min=6832, max=62112, avg=36760.89, stdev=9079.40
     lat (usec): min=6878, max=62175, avg=36824.15, stdev=9086.02
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[24249], 10.00th=[26084], 20.00th=[29754],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[35390], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[50070], 95.00th=[54264],
     | 99.00th=[57934], 99.50th=[59507], 99.90th=[61604], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min= 1464, max= 2296, per=24.94%, avg=1731.64, stdev=240.44, samples=11
   iops        : min=  366, max=  574, avg=432.91, stdev=60.11, samples=11
  lat (msec)   : 10=0.08%, 20=2.19%, 50=87.73%, 100=10.00%
  cpu          : usr=0.53%, sys=3.81%, ctx=1121, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21389: Fri Jan 17 12:46:55 2025
  write: IOPS=436, BW=1747KiB/s (1789kB/s)(10.0MiB/5860msec)
    slat (usec): min=11, max=1476, avg=66.49, stdev=89.13
    clat (usec): min=3078, max=68140, avg=36519.01, stdev=9280.58
     lat (usec): min=3147, max=68175, avg=36586.84, stdev=9283.38
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[23987], 10.00th=[26346], 20.00th=[29492],
     | 30.00th=[31589], 40.00th=[33424], 50.00th=[35390], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[49021], 95.00th=[53740],
     | 99.00th=[59507], 99.50th=[61604], 99.90th=[65274], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1464, max= 2384, per=25.15%, avg=1745.82, stdev=260.45, samples=11
   iops        : min=  366, max=  596, avg=436.45, stdev=65.11, samples=11
  lat (msec)   : 4=0.04%, 10=0.47%, 20=2.54%, 50=88.55%, 100=8.40%
  cpu          : usr=0.80%, sys=3.72%, ctx=1357, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21390: Fri Jan 17 12:46:55 2025
  write: IOPS=434, BW=1738KiB/s (1780kB/s)(10.0MiB/5891msec)
    slat (usec): min=11, max=1171, avg=61.61, stdev=73.78
    clat (usec): min=6841, max=68426, avg=36701.54, stdev=9008.32
     lat (usec): min=6880, max=68454, avg=36764.84, stdev=9011.46
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[24773], 10.00th=[26608], 20.00th=[29754],
     | 30.00th=[31589], 40.00th=[33424], 50.00th=[35390], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[48497], 95.00th=[53740],
     | 99.00th=[58983], 99.50th=[62129], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1456, max= 2288, per=24.98%, avg=1734.18, stdev=238.40, samples=11
   iops        : min=  364, max=  572, avg=433.55, stdev=59.60, samples=11
  lat (msec)   : 10=0.12%, 20=2.23%, 50=89.18%, 100=8.48%
  cpu          : usr=1.34%, sys=3.06%, ctx=1210, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=21391: Fri Jan 17 12:46:55 2025
  write: IOPS=434, BW=1737KiB/s (1779kB/s)(10.0MiB/5895msec)
    slat (usec): min=10, max=21404, avg=74.40, stdev=429.10
    clat (usec): min=3002, max=65798, avg=36622.52, stdev=9261.68
     lat (usec): min=7216, max=65826, avg=36698.48, stdev=9245.04
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[24249], 10.00th=[26346], 20.00th=[29492],
     | 30.00th=[31589], 40.00th=[33162], 50.00th=[35390], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[49546], 95.00th=[54789],
     | 99.00th=[58459], 99.50th=[59507], 99.90th=[65274], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1432, max= 2248, per=24.96%, avg=1733.09, stdev=235.86, samples=11
   iops        : min=  358, max=  562, avg=433.27, stdev=58.97, samples=11
  lat (msec)   : 4=0.04%, 10=0.16%, 20=2.66%, 50=87.30%, 100=9.84%
  cpu          : usr=0.42%, sys=4.55%, ctx=1411, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6944KiB/s (7110kB/s), 1736KiB/s-1747KiB/s (1778kB/s-1789kB/s), io=40.0MiB (41.9MB), run=5860-5899msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=21406: Fri Jan 17 12:47:01 2025
   read: IOPS=254, BW=1019KiB/s (1044kB/s)(5048KiB/4952msec)
    slat (usec): min=747, max=6426, avg=1745.86, stdev=646.05
    clat (usec): min=7504, max=59979, avg=29397.43, stdev=6844.97
     lat (usec): min=8613, max=62055, avg=31146.61, stdev=7062.39
    clat percentiles (usec):
     |  1.00th=[16712],  5.00th=[19792], 10.00th=[21365], 20.00th=[23462],
     | 30.00th=[25035], 40.00th=[26870], 50.00th=[28705], 60.00th=[30540],
     | 70.00th=[32900], 80.00th=[35390], 90.00th=[38536], 95.00th=[40633],
     | 99.00th=[47449], 99.50th=[49021], 99.90th=[58459], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  824, max= 1208, per=24.66%, avg=998.89, stdev=140.30, samples=9
   iops        : min=  206, max=  302, avg=249.67, stdev=35.11, samples=9
  write: IOPS=262, BW=1048KiB/s (1074kB/s)(5192KiB/4952msec)
    slat (usec): min=11, max=2831, avg=62.26, stdev=112.82
    clat (usec): min=16697, max=63728, avg=30389.29, stdev=6696.75
     lat (usec): min=16743, max=63763, avg=30452.96, stdev=6699.72
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[21103], 10.00th=[22676], 20.00th=[24511],
     | 30.00th=[25822], 40.00th=[27657], 50.00th=[30016], 60.00th=[31589],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[39060], 95.00th=[41157],
     | 99.00th=[52167], 99.50th=[54264], 99.90th=[62129], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  880, max= 1344, per=24.81%, avg=1046.00, stdev=158.30, samples=9
   iops        : min=  220, max=  336, avg=261.44, stdev=39.63, samples=9
  lat (msec)   : 10=0.08%, 20=3.83%, 50=95.23%, 100=0.86%
  cpu          : usr=0.99%, sys=7.19%, ctx=2271, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21407: Fri Jan 17 12:47:01 2025
   read: IOPS=252, BW=1012KiB/s (1036kB/s)(4976KiB/4918msec)
    slat (usec): min=23, max=5839, avg=1712.75, stdev=651.78
    clat (usec): min=1070, max=60047, avg=29229.47, stdev=7100.45
     lat (usec): min=2296, max=62480, avg=30945.29, stdev=7341.01
    clat percentiles (usec):
     |  1.00th=[16188],  5.00th=[19530], 10.00th=[20841], 20.00th=[22938],
     | 30.00th=[24773], 40.00th=[26608], 50.00th=[28705], 60.00th=[30540],
     | 70.00th=[32637], 80.00th=[35390], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[49021], 99.50th=[52167], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  816, max= 1240, per=24.66%, avg=998.89, stdev=141.08, samples=9
   iops        : min=  204, max=  310, avg=249.67, stdev=35.28, samples=9
  write: IOPS=267, BW=1070KiB/s (1096kB/s)(5264KiB/4918msec)
    slat (usec): min=13, max=1744, avg=57.73, stdev=67.16
    clat (usec): min=8248, max=58161, avg=30381.01, stdev=7064.74
     lat (usec): min=8273, max=58212, avg=30440.38, stdev=7068.12
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[21103], 10.00th=[22414], 20.00th=[24249],
     | 30.00th=[25560], 40.00th=[27132], 50.00th=[29754], 60.00th=[31851],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[39060], 95.00th=[42730],
     | 99.00th=[51119], 99.50th=[52691], 99.90th=[55837], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  912, max= 1416, per=25.31%, avg=1067.22, stdev=154.74, samples=9
   iops        : min=  228, max=  354, avg=266.78, stdev=38.68, samples=9
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.12%, 20=4.10%, 50=94.61%
  lat (msec)   : 100=1.05%
  cpu          : usr=0.49%, sys=7.63%, ctx=2251, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21408: Fri Jan 17 12:47:01 2025
   read: IOPS=251, BW=1007KiB/s (1031kB/s)(4984KiB/4950msec)
    slat (usec): min=725, max=5654, avg=1753.48, stdev=654.87
    clat (usec): min=2886, max=65078, avg=29367.68, stdev=7183.62
     lat (usec): min=4421, max=66741, avg=31124.65, stdev=7414.01
    clat percentiles (usec):
     |  1.00th=[16712],  5.00th=[19530], 10.00th=[21103], 20.00th=[22938],
     | 30.00th=[25035], 40.00th=[26870], 50.00th=[28443], 60.00th=[30278],
     | 70.00th=[32375], 80.00th=[34866], 90.00th=[38536], 95.00th=[41681],
     | 99.00th=[50594], 99.50th=[51119], 99.90th=[62129], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  880, max= 1232, per=24.77%, avg=1003.22, stdev=118.40, samples=9
   iops        : min=  220, max=  308, avg=250.78, stdev=29.58, samples=9
  write: IOPS=265, BW=1062KiB/s (1087kB/s)(5256KiB/4950msec)
    slat (usec): min=12, max=2803, avg=62.93, stdev=106.53
    clat (usec): min=16524, max=58196, avg=30445.94, stdev=6655.59
     lat (usec): min=16560, max=58232, avg=30510.63, stdev=6659.87
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[21627], 10.00th=[22938], 20.00th=[24511],
     | 30.00th=[26084], 40.00th=[27657], 50.00th=[29754], 60.00th=[31589],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[49546], 99.50th=[52691], 99.90th=[57934], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  808, max= 1312, per=24.87%, avg=1048.67, stdev=172.23, samples=9
   iops        : min=  202, max=  328, avg=262.11, stdev=43.09, samples=9
  lat (msec)   : 4=0.04%, 10=0.08%, 20=4.14%, 50=94.65%, 100=1.09%
  cpu          : usr=0.00%, sys=8.32%, ctx=2243, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=21409: Fri Jan 17 12:47:01 2025
   read: IOPS=255, BW=1021KiB/s (1046kB/s)(5060KiB/4955msec)
    slat (usec): min=749, max=8528, avg=1763.02, stdev=720.52
    clat (usec): min=10603, max=66998, avg=29627.59, stdev=6814.61
     lat (usec): min=11980, max=68682, avg=31393.72, stdev=7066.50
    clat percentiles (usec):
     |  1.00th=[16450],  5.00th=[19530], 10.00th=[21103], 20.00th=[23462],
     | 30.00th=[25560], 40.00th=[27395], 50.00th=[29492], 60.00th=[31065],
     | 70.00th=[32900], 80.00th=[35390], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[47449], 99.50th=[49021], 99.90th=[52167], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  832, max= 1256, per=25.06%, avg=1014.89, stdev=140.68, samples=9
   iops        : min=  208, max=  314, avg=253.67, stdev=35.18, samples=9
  write: IOPS=261, BW=1045KiB/s (1070kB/s)(5180KiB/4955msec)
    slat (usec): min=12, max=1280, avg=60.99, stdev=75.31
    clat (usec): min=15161, max=52825, avg=30251.91, stdev=6523.36
     lat (usec): min=15197, max=52864, avg=30314.24, stdev=6529.77
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[21103], 10.00th=[22938], 20.00th=[24511],
     | 30.00th=[25822], 40.00th=[27395], 50.00th=[29230], 60.00th=[31065],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[39060], 95.00th=[42206],
     | 99.00th=[48497], 99.50th=[51643], 99.90th=[52167], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min=  856, max= 1256, per=24.45%, avg=1030.78, stdev=142.25, samples=9
   iops        : min=  214, max=  314, avg=257.67, stdev=35.56, samples=9
  lat (msec)   : 20=4.22%, 50=95.23%, 100=0.55%
  cpu          : usr=0.48%, sys=7.67%, ctx=2245, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4050KiB/s (4147kB/s), 1007KiB/s-1021KiB/s (1031kB/s-1046kB/s), io=19.6MiB (20.5MB), run=4918-4955msec
  WRITE: bw=4216KiB/s (4318kB/s), 1045KiB/s-1070KiB/s (1070kB/s-1096kB/s), io=20.4MiB (21.4MB), run=4918-4955msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=21417: Fri Jan 17 12:47:02 2025
   read: IOPS=10, BW=412MiB/s (432MB/s)(40.0MiB/97msec)
    slat (nsec): min=10728k, max=10728k, avg=10727657.00, stdev= 0.00
    clat (nsec): min=81604k, max=81604k, avg=81604084.00, stdev= 0.00
     lat (nsec): min=92355k, max=92355k, avg=92355431.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[81265],  5.00th=[81265], 10.00th=[81265], 20.00th=[81265],
     | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265],
     | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265],
     | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.38%, ctx=6, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=412MiB/s (432MB/s), 412MiB/s-412MiB/s (432MB/s-432MB/s), io=40.0MiB (41.9MB), run=97-97msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck