== sanity test 398c: run fio to test AIO ================= 15:23:14 (1741634594)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0903344 s, 464 MB/s
osc.lustre-OST0000-osc-ffff88012b71c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=25829: Mon Mar 10 15:23:21 2025
  write: IOPS=458, BW=1832KiB/s (1876kB/s)(10.0MiB/5589msec)
    slat (usec): min=12, max=1409, avg=67.43, stdev=51.33
    clat (usec): min=15877, max=76068, avg=34711.64, stdev=10258.54
     lat (usec): min=15920, max=76166, avg=34780.91, stdev=10275.50
    clat percentiles (usec):
     |  1.00th=[16712],  5.00th=[18220], 10.00th=[20579], 20.00th=[22676],
     | 30.00th=[26084], 40.00th=[31589], 50.00th=[36963], 60.00th=[40633],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[46400], 95.00th=[47973],
     | 99.00th=[50594], 99.50th=[51643], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1344, max= 2688, per=25.06%, avg=1835.27, stdev=467.48, samples=11
   iops        : min=  336, max=  672, avg=458.82, stdev=116.87, samples=11
  lat (msec)   : 20=8.01%, 50=90.43%, 100=1.56%
  cpu          : usr=0.63%, sys=3.60%, ctx=1098, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25830: Mon Mar 10 15:23:21 2025
  write: IOPS=458, BW=1834KiB/s (1878kB/s)(10.0MiB/5584msec)
    slat (usec): min=12, max=329, avg=66.76, stdev=39.36
    clat (usec): min=4162, max=67150, avg=34640.84, stdev=10260.21
     lat (usec): min=4211, max=67241, avg=34709.31, stdev=10277.04
    clat percentiles (usec):
     |  1.00th=[16581],  5.00th=[18744], 10.00th=[20579], 20.00th=[22676],
     | 30.00th=[26608], 40.00th=[31851], 50.00th=[36963], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[44303], 90.00th=[46400], 95.00th=[47973],
     | 99.00th=[50070], 99.50th=[50594], 99.90th=[66847], 99.95th=[66847],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1384, max= 2648, per=25.02%, avg=1832.36, stdev=445.45, samples=11
   iops        : min=  346, max=  662, avg=458.09, stdev=111.36, samples=11
  lat (msec)   : 10=0.12%, 20=7.97%, 50=89.69%, 100=2.23%
  cpu          : usr=0.72%, sys=3.64%, ctx=1257, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25831: Mon Mar 10 15:23:21 2025
  write: IOPS=457, BW=1831KiB/s (1875kB/s)(10.0MiB/5592msec)
    slat (usec): min=12, max=292, avg=59.24, stdev=39.15
    clat (usec): min=5189, max=65408, avg=34654.84, stdev=10218.80
     lat (usec): min=5247, max=65546, avg=34715.69, stdev=10238.01
    clat percentiles (usec):
     |  1.00th=[16712],  5.00th=[18482], 10.00th=[20841], 20.00th=[22676],
     | 30.00th=[26084], 40.00th=[31327], 50.00th=[36439], 60.00th=[40633],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[46400], 95.00th=[47449],
     | 99.00th=[50594], 99.50th=[51119], 99.90th=[60556], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min= 1328, max= 2560, per=25.03%, avg=1833.09, stdev=441.98, samples=11
   iops        : min=  332, max=  640, avg=458.27, stdev=110.50, samples=11
  lat (msec)   : 10=0.16%, 20=7.30%, 50=90.78%, 100=1.76%
  cpu          : usr=0.64%, sys=3.20%, ctx=825, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=25832: Mon Mar 10 15:23:21 2025
  write: IOPS=459, BW=1836KiB/s (1881kB/s)(10.0MiB/5576msec)
    slat (usec): min=12, max=1166, avg=70.76, stdev=51.23
    clat (usec): min=2517, max=84697, avg=34622.16, stdev=10489.02
     lat (usec): min=2599, max=84769, avg=34694.63, stdev=10501.59
    clat percentiles (usec):
     |  1.00th=[16450],  5.00th=[18220], 10.00th=[20317], 20.00th=[22676],
     | 30.00th=[26608], 40.00th=[31589], 50.00th=[36963], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[44303], 90.00th=[46924], 95.00th=[47973],
     | 99.00th=[50594], 99.50th=[51643], 99.90th=[80217], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1392, max= 2712, per=25.14%, avg=1841.09, stdev=457.18, samples=11
   iops        : min=  348, max=  678, avg=460.27, stdev=114.29, samples=11
  lat (msec)   : 4=0.04%, 10=0.08%, 20=8.36%, 50=89.73%, 100=1.80%
  cpu          : usr=0.66%, sys=3.80%, ctx=1427, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7325KiB/s (7501kB/s), 1831KiB/s-1836KiB/s (1875kB/s-1881kB/s), io=40.0MiB (41.9MB), run=5576-5592msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=25846: Mon Mar 10 15:23:25 2025
   read: IOPS=415, BW=1661KiB/s (1701kB/s)(5048KiB/3039msec)
    slat (usec): min=626, max=4229, avg=1202.71, stdev=382.35
    clat (usec): min=5430, max=39760, avg=17851.36, stdev=5236.49
     lat (usec): min=6402, max=41933, avg=19055.85, stdev=5506.07
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13042], 10.00th=[13698], 20.00th=[14484],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[19006], 90.00th=[25560], 95.00th=[31589],
     | 99.00th=[35914], 99.50th=[38536], 99.90th=[39584], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min=  856, max= 1952, per=24.91%, avg=1644.67, stdev=415.13, samples=6
   iops        : min=  214, max=  488, avg=411.17, stdev=103.78, samples=6
  write: IOPS=427, BW=1708KiB/s (1749kB/s)(5192KiB/3039msec)
    slat (usec): min=13, max=1951, avg=47.97, stdev=58.94
    clat (usec): min=5396, max=38349, avg=18359.11, stdev=4822.20
     lat (usec): min=5424, max=38381, avg=18408.16, stdev=4836.18
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14222], 10.00th=[14746], 20.00th=[15401],
     | 30.00th=[16057], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[26870], 95.00th=[31065],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[38011], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min=  904, max= 2003, per=24.76%, avg=1701.83, stdev=406.09, samples=6
   iops        : min=  226, max=  500, avg=425.33, stdev=101.41, samples=6
  lat (msec)   : 10=0.27%, 20=84.02%, 50=15.70%
  cpu          : usr=0.92%, sys=7.18%, ctx=2497, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25847: Mon Mar 10 15:23:25 2025
   read: IOPS=410, BW=1642KiB/s (1681kB/s)(4976KiB/3031msec)
    slat (usec): min=654, max=4402, avg=1195.34, stdev=398.52
    clat (usec): min=6127, max=42951, avg=17837.77, stdev=5222.57
     lat (usec): min=7349, max=46034, avg=19035.19, stdev=5488.84
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13042], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15139], 40.00th=[15926], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[19006], 90.00th=[24773], 95.00th=[31065],
     | 99.00th=[37487], 99.50th=[38011], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min=  872, max= 1904, per=24.71%, avg=1631.33, stdev=399.22, samples=6
   iops        : min=  218, max=  476, avg=407.83, stdev=99.80, samples=6
  write: IOPS=434, BW=1737KiB/s (1778kB/s)(5264KiB/3031msec)
    slat (usec): min=12, max=252, avg=46.61, stdev=26.20
    clat (usec): min=11168, max=42370, avg=18514.25, stdev=5056.63
     lat (usec): min=11201, max=42412, avg=18561.95, stdev=5068.23
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14091], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[27919], 95.00th=[31065],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[40633], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min=  992, max= 1960, per=25.28%, avg=1738.00, stdev=388.47, samples=6
   iops        : min=  248, max=  490, avg=434.50, stdev=97.12, samples=6
  lat (msec)   : 10=0.04%, 20=83.36%, 50=16.60%
  cpu          : usr=0.73%, sys=7.39%, ctx=2471, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25848: Mon Mar 10 15:23:25 2025
   read: IOPS=412, BW=1652KiB/s (1692kB/s)(4984KiB/3017msec)
    slat (usec): min=32, max=3754, avg=1208.41, stdev=406.26
    clat (usec): min=3528, max=43794, avg=17913.31, stdev=5217.21
     lat (usec): min=3606, max=46198, avg=19123.71, stdev=5491.93
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13304], 10.00th=[13698], 20.00th=[14615],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[25560], 95.00th=[31327],
     | 99.00th=[35914], 99.50th=[37487], 99.90th=[42206], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min=  968, max= 1904, per=25.03%, avg=1652.67, stdev=348.79, samples=6
   iops        : min=  242, max=  476, avg=413.17, stdev=87.20, samples=6
  write: IOPS=435, BW=1742KiB/s (1784kB/s)(5256KiB/3017msec)
    slat (usec): min=12, max=1039, avg=50.31, stdev=52.48
    clat (usec): min=11603, max=41613, avg=18431.38, stdev=5028.14
     lat (usec): min=11651, max=41734, avg=18482.86, stdev=5038.55
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14222], 10.00th=[14746], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19268], 90.00th=[27657], 95.00th=[30802],
     | 99.00th=[36439], 99.50th=[38011], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1008, max= 2051, per=25.26%, avg=1736.50, stdev=395.45, samples=6
   iops        : min=  252, max=  512, avg=434.00, stdev=98.74, samples=6
  lat (msec)   : 4=0.08%, 10=0.12%, 20=83.16%, 50=16.64%
  cpu          : usr=0.99%, sys=7.19%, ctx=2443, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=25849: Mon Mar 10 15:23:25 2025
   read: IOPS=416, BW=1666KiB/s (1706kB/s)(5060KiB/3037msec)
    slat (usec): min=637, max=4096, avg=1209.92, stdev=400.84
    clat (usec): min=10362, max=42367, avg=17924.92, stdev=5211.48
     lat (usec): min=11249, max=44304, avg=19136.76, stdev=5492.67
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13173], 10.00th=[13829], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[19006], 90.00th=[25297], 95.00th=[31065],
     | 99.00th=[36439], 99.50th=[37487], 99.90th=[42206], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min=  880, max= 1936, per=25.12%, avg=1658.67, stdev=402.35, samples=6
   iops        : min=  220, max=  484, avg=414.67, stdev=100.59, samples=6
  write: IOPS=426, BW=1706KiB/s (1747kB/s)(5180KiB/3037msec)
    slat (usec): min=13, max=903, avg=48.26, stdev=38.78
    clat (usec): min=11645, max=40595, avg=18359.31, stdev=4820.93
     lat (usec): min=11682, max=40682, avg=18408.88, stdev=4833.02
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14091], 10.00th=[14615], 20.00th=[15401],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[19268], 90.00th=[26608], 95.00th=[30802],
     | 99.00th=[34866], 99.50th=[35914], 99.90th=[39060], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min=  928, max= 1976, per=24.67%, avg=1696.00, stdev=391.00, samples=6
   iops        : min=  232, max=  494, avg=424.00, stdev=97.75, samples=6
  lat (msec)   : 20=84.26%, 50=15.74%
  cpu          : usr=0.82%, sys=7.38%, ctx=2576, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6603KiB/s (6762kB/s), 1642KiB/s-1666KiB/s (1681kB/s-1706kB/s), io=19.6MiB (20.5MB), run=3017-3039msec
  WRITE: bw=6875KiB/s (7040kB/s), 1706KiB/s-1742KiB/s (1747kB/s-1784kB/s), io=20.4MiB (21.4MB), run=3017-3039msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=25855: Mon Mar 10 15:23:26 2025
   read: IOPS=10, BW=440MiB/s (461MB/s)(40.0MiB/91msec)
    slat (nsec): min=5319.1k, max=5319.1k, avg=5319077.00, stdev= 0.00
    clat (nsec): min=81200k, max=81200k, avg=81200378.00, stdev= 0.00
     lat (nsec): min=86528k, max=86528k, avg=86528118.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[81265],  5.00th=[81265], 10.00th=[81265], 20.00th=[81265],
     | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265],
     | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265],
     | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.56%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=440MiB/s (461MB/s), 440MiB/s-440MiB/s (461MB/s-461MB/s), io=40.0MiB (41.9MB), run=91-91msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout