== sanity test 398c: run fio to test AIO ================= 01:16:16 (1743743776)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.128935 s, 325 MB/s
osc.lustre-OST0000-osc-ffff88013073f000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30468: Fri Apr  4 01:16:25 2025
  write: IOPS=375, BW=1502KiB/s (1538kB/s)(10.0MiB/6817msec)
    slat (usec): min=14, max=755, avg=80.17, stdev=41.91
    clat (usec): min=6673, max=80387, avg=42358.52, stdev=5252.61
     lat (usec): min=6822, max=80435, avg=42440.63, stdev=5253.81
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[33424], 10.00th=[35390], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[48497],
     | 99.00th=[49546], 99.50th=[51643], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1400, max= 1736, per=25.16%, avg=1503.46, stdev=103.83, samples=13
   iops        : min=  350, max=  434, avg=375.85, stdev=25.97, samples=13
  lat (msec)   : 10=0.12%, 20=0.31%, 50=99.02%, 100=0.55%
  cpu          : usr=0.70%, sys=3.83%, ctx=1573, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30469: Fri Apr  4 01:16:25 2025
  write: IOPS=374, BW=1499KiB/s (1535kB/s)(10.0MiB/6831msec)
    slat (usec): min=16, max=505, avg=66.95, stdev=39.18
    clat (usec): min=9527, max=57904, avg=42434.87, stdev=4561.11
     lat (usec): min=9642, max=58044, avg=42503.64, stdev=4566.59
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[34341], 10.00th=[35390], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[41681], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[48497],
     | 99.00th=[49546], 99.50th=[49546], 99.90th=[51643], 99.95th=[52691],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1400, max= 1664, per=25.05%, avg=1496.69, stdev=96.60, samples=13
   iops        : min=  350, max=  416, avg=374.15, stdev=24.17, samples=13
  lat (msec)   : 10=0.04%, 50=99.61%, 100=0.35%
  cpu          : usr=0.47%, sys=3.31%, ctx=865, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30470: Fri Apr  4 01:16:25 2025
  write: IOPS=373, BW=1494KiB/s (1530kB/s)(10.0MiB/6853msec)
    slat (usec): min=16, max=1144, avg=66.51, stdev=48.89
    clat (usec): min=9348, max=74316, avg=42493.17, stdev=4634.29
     lat (usec): min=9429, max=74393, avg=42561.43, stdev=4640.25
    clat percentiles (usec):
     |  1.00th=[30540],  5.00th=[34341], 10.00th=[35914], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[48497],
     | 99.00th=[49546], 99.50th=[49546], 99.90th=[66323], 99.95th=[69731],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1328, max= 1664, per=24.96%, avg=1491.85, stdev=108.14, samples=13
   iops        : min=  332, max=  416, avg=372.92, stdev=27.06, samples=13
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.53%, 100=0.35%
  cpu          : usr=0.54%, sys=3.04%, ctx=911, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30471: Fri Apr  4 01:16:25 2025
  write: IOPS=373, BW=1495KiB/s (1531kB/s)(10.0MiB/6848msec)
    slat (usec): min=14, max=747, avg=70.57, stdev=39.86
    clat (usec): min=5684, max=79307, avg=42492.03, stdev=4640.81
     lat (usec): min=5799, max=79364, avg=42564.52, stdev=4642.57
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[33817], 10.00th=[35390], 20.00th=[39060],
     | 30.00th=[40633], 40.00th=[41681], 50.00th=[42730], 60.00th=[44303],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47449], 95.00th=[48497],
     | 99.00th=[49021], 99.50th=[49546], 99.90th=[59507], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1352, max= 1664, per=25.07%, avg=1498.23, stdev=99.12, samples=13
   iops        : min=  338, max=  416, avg=374.54, stdev=24.79, samples=13
  lat (msec)   : 10=0.04%, 20=0.04%, 50=99.61%, 100=0.31%
  cpu          : usr=0.70%, sys=3.23%, ctx=1068, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5977KiB/s (6120kB/s), 1494KiB/s-1502KiB/s (1530kB/s-1538kB/s), io=40.0MiB (41.9MB), run=6817-6853msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30487: Fri Apr  4 01:16:31 2025
   read: IOPS=256, BW=1024KiB/s (1049kB/s)(5048KiB/4928msec)
    slat (usec): min=807, max=3415, avg=1924.02, stdev=379.09
    clat (usec): min=8373, max=59228, avg=29072.44, stdev=5073.65
     lat (usec): min=10451, max=61242, avg=30999.74, stdev=5193.99
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[21365], 10.00th=[23200], 20.00th=[24773],
     | 30.00th=[26346], 40.00th=[27657], 50.00th=[28705], 60.00th=[30278],
     | 70.00th=[31589], 80.00th=[33162], 90.00th=[35390], 95.00th=[37487],
     | 99.00th=[41157], 99.50th=[44303], 99.90th=[56886], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  840, max= 1280, per=25.11%, avg=1019.78, stdev=132.26, samples=9
   iops        : min=  210, max=  320, avg=254.89, stdev=33.04, samples=9
  write: IOPS=263, BW=1054KiB/s (1079kB/s)(5192KiB/4928msec)
    slat (usec): min=16, max=2395, avg=70.29, stdev=90.24
    clat (usec): min=4120, max=56884, avg=30096.23, stdev=4372.65
     lat (usec): min=4206, max=56960, avg=30168.10, stdev=4379.43
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[23200], 10.00th=[24773], 20.00th=[26608],
     | 30.00th=[27919], 40.00th=[28967], 50.00th=[30278], 60.00th=[31065],
     | 70.00th=[32113], 80.00th=[33424], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[40109], 99.50th=[42206], 99.90th=[55313], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  880, max= 1136, per=25.04%, avg=1058.89, stdev=75.57, samples=9
   iops        : min=  220, max=  284, avg=264.67, stdev=18.90, samples=9
  lat (msec)   : 10=0.08%, 20=1.33%, 50=98.28%, 100=0.31%
  cpu          : usr=1.04%, sys=6.58%, ctx=2377, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30488: Fri Apr  4 01:16:31 2025
   read: IOPS=252, BW=1011KiB/s (1035kB/s)(4976KiB/4922msec)
    slat (usec): min=54, max=4355, avg=1919.90, stdev=395.46
    clat (usec): min=4958, max=52927, avg=29177.75, stdev=5011.16
     lat (usec): min=5015, max=54928, avg=31100.63, stdev=5149.57
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[21103], 10.00th=[22938], 20.00th=[25297],
     | 30.00th=[26608], 40.00th=[27919], 50.00th=[28967], 60.00th=[30278],
     | 70.00th=[31327], 80.00th=[32900], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[41157], 99.50th=[44303], 99.90th=[50594], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min=  872, max= 1240, per=24.72%, avg=1004.11, stdev=118.85, samples=9
   iops        : min=  218, max=  310, avg=251.00, stdev=29.71, samples=9
  write: IOPS=267, BW=1069KiB/s (1095kB/s)(5264KiB/4922msec)
    slat (usec): min=16, max=1100, avg=69.73, stdev=47.19
    clat (usec): min=9845, max=58805, avg=30234.24, stdev=4473.00
     lat (usec): min=9920, max=58853, avg=30305.61, stdev=4477.30
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[22938], 10.00th=[24511], 20.00th=[26608],
     | 30.00th=[28181], 40.00th=[29230], 50.00th=[30278], 60.00th=[31327],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[39584], 99.50th=[41681], 99.90th=[56886], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  984, max= 1168, per=25.51%, avg=1078.78, stdev=63.64, samples=9
   iops        : min=  246, max=  292, avg=269.67, stdev=15.87, samples=9
  lat (msec)   : 10=0.08%, 20=1.72%, 50=98.01%, 100=0.20%
  cpu          : usr=0.85%, sys=6.64%, ctx=2379, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30489: Fri Apr  4 01:16:31 2025
   read: IOPS=252, BW=1010KiB/s (1034kB/s)(4984KiB/4936msec)
    slat (usec): min=860, max=3453, avg=1918.31, stdev=383.36
    clat (usec): min=4853, max=49107, avg=29085.03, stdev=4899.21
     lat (usec): min=7153, max=51070, avg=31006.56, stdev=5027.32
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[21627], 10.00th=[22938], 20.00th=[25035],
     | 30.00th=[26608], 40.00th=[27919], 50.00th=[28967], 60.00th=[30278],
     | 70.00th=[31327], 80.00th=[33162], 90.00th=[35390], 95.00th=[37487],
     | 99.00th=[40633], 99.50th=[42206], 99.90th=[42730], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min=  856, max= 1168, per=24.81%, avg=1007.67, stdev=94.50, samples=9
   iops        : min=  214, max=  292, avg=251.89, stdev=23.61, samples=9
  write: IOPS=266, BW=1065KiB/s (1090kB/s)(5256KiB/4936msec)
    slat (usec): min=17, max=303, avg=67.81, stdev=33.00
    clat (usec): min=18461, max=48945, avg=30305.36, stdev=4294.81
     lat (usec): min=18505, max=48973, avg=30374.75, stdev=4299.29
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[23200], 10.00th=[24511], 20.00th=[26608],
     | 30.00th=[28181], 40.00th=[29230], 50.00th=[30278], 60.00th=[31327],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[40633], 99.90th=[46924], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min=  896, max= 1216, per=25.17%, avg=1064.56, stdev=107.06, samples=9
   iops        : min=  224, max=  304, avg=266.11, stdev=26.74, samples=9
  lat (msec)   : 10=0.04%, 20=1.76%, 50=98.20%
  cpu          : usr=0.89%, sys=6.63%, ctx=2295, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30490: Fri Apr  4 01:16:31 2025
   read: IOPS=256, BW=1024KiB/s (1049kB/s)(5060KiB/4940msec)
    slat (usec): min=943, max=3440, avg=1952.37, stdev=374.57
    clat (usec): min=9266, max=44522, avg=29134.20, stdev=4789.16
     lat (usec): min=10975, max=46730, avg=31089.59, stdev=4910.86
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[21627], 10.00th=[23200], 20.00th=[25035],
     | 30.00th=[26346], 40.00th=[27657], 50.00th=[28967], 60.00th=[30278],
     | 70.00th=[31851], 80.00th=[33424], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[40633], 99.50th=[42206], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min=  856, max= 1208, per=25.17%, avg=1022.22, stdev=101.02, samples=9
   iops        : min=  214, max=  302, avg=255.56, stdev=25.25, samples=9
  write: IOPS=262, BW=1049KiB/s (1074kB/s)(5180KiB/4940msec)
    slat (usec): min=18, max=338, avg=66.79, stdev=32.57
    clat (usec): min=12666, max=43167, avg=30159.41, stdev=4126.86
     lat (usec): min=12807, max=43226, avg=30227.70, stdev=4131.24
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[22938], 10.00th=[24773], 20.00th=[26608],
     | 30.00th=[28181], 40.00th=[29230], 50.00th=[30278], 60.00th=[31327],
     | 70.00th=[32375], 80.00th=[33424], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[39060], 99.50th=[40109], 99.90th=[42206], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min=  880, max= 1176, per=24.57%, avg=1039.11, stdev=89.84, samples=9
   iops        : min=  220, max=  294, avg=259.78, stdev=22.46, samples=9
  lat (msec)   : 10=0.04%, 20=1.25%, 50=98.71%
  cpu          : usr=0.97%, sys=6.66%, ctx=2376, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4062KiB/s (4160kB/s), 1010KiB/s-1024KiB/s (1034kB/s-1049kB/s), io=19.6MiB (20.5MB), run=4922-4940msec
  WRITE: bw=4229KiB/s (4331kB/s), 1049KiB/s-1069KiB/s (1074kB/s-1095kB/s), io=20.4MiB (21.4MB), run=4922-4940msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30497: Fri Apr  4 01:16:32 2025
   read: IOPS=6, BW=245MiB/s (257MB/s)(40.0MiB/163msec)
    slat (nsec): min=5304.1k, max=5304.1k, avg=5304082.00, stdev= 0.00
    clat (nsec): min=151106k, max=151106k, avg=151105578.00, stdev= 0.00
     lat (nsec): min=156419k, max=156419k, avg=156418662.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  153],  5.00th=[  153], 10.00th=[  153], 20.00th=[  153],
     | 30.00th=[  153], 40.00th=[  153], 50.00th=[  153], 60.00th=[  153],
     | 70.00th=[  153], 80.00th=[  153], 90.00th=[  153], 95.00th=[  153],
     | 99.00th=[  153], 99.50th=[  153], 99.90th=[  153], 99.95th=[  153],
     | 99.99th=[  153]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.70%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=40.0MiB (41.9MB), run=163-163msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout