== sanity test 398c: run fio to test AIO ================= 14:01:03 (1747850463)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0985051 s, 426 MB/s
osc.lustre-OST0000-osc-ffff880136b6c000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=8918: Wed May 21 14:01:10 2025
  write: IOPS=445, BW=1781KiB/s (1824kB/s)(10.0MiB/5750msec)
    slat (usec): min=10, max=1637, avg=74.75, stdev=101.57
    clat (usec): min=4037, max=83091, avg=35759.21, stdev=7228.52
     lat (usec): min=4162, max=83146, avg=35835.62, stdev=7232.53
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[25822], 10.00th=[27657], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[40633], 90.00th=[44303], 95.00th=[47973],
     | 99.00th=[55837], 99.50th=[57934], 99.90th=[78119], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1544, max= 2008, per=25.20%, avg=1786.55, stdev=139.85, samples=11
   iops        : min=  386, max=  502, avg=446.64, stdev=34.96, samples=11
  lat (msec)   : 10=0.27%, 20=1.09%, 50=94.80%, 100=3.83%
  cpu          : usr=0.90%, sys=3.69%, ctx=1573, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8919: Wed May 21 14:01:10 2025
  write: IOPS=443, BW=1776KiB/s (1819kB/s)(10.0MiB/5766msec)
    slat (usec): min=11, max=1120, avg=67.24, stdev=76.37
    clat (usec): min=3833, max=68276, avg=35833.74, stdev=6973.58
     lat (usec): min=3933, max=68341, avg=35902.39, stdev=6976.06
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[26346], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[32113], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[44303], 95.00th=[48497],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1552, max= 2016, per=25.09%, avg=1778.91, stdev=126.95, samples=11
   iops        : min=  388, max=  504, avg=444.73, stdev=31.74, samples=11
  lat (msec)   : 4=0.04%, 10=0.12%, 20=1.33%, 50=94.92%, 100=3.59%
  cpu          : usr=0.49%, sys=3.76%, ctx=1406, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8920: Wed May 21 14:01:10 2025
  write: IOPS=443, BW=1773KiB/s (1816kB/s)(10.0MiB/5775msec)
    slat (usec): min=11, max=1155, avg=66.79, stdev=79.40
    clat (usec): min=4171, max=65685, avg=35910.29, stdev=6852.31
     lat (usec): min=4266, max=65711, avg=35978.83, stdev=6853.12
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[26084], 10.00th=[27657], 20.00th=[30278],
     | 30.00th=[32113], 40.00th=[33817], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[44303], 95.00th=[47449],
     | 99.00th=[55313], 99.50th=[55837], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1544, max= 2024, per=25.02%, avg=1774.18, stdev=134.88, samples=11
   iops        : min=  386, max=  506, avg=443.55, stdev=33.72, samples=11
  lat (msec)   : 10=0.08%, 20=1.41%, 50=95.66%, 100=2.85%
  cpu          : usr=0.87%, sys=3.27%, ctx=1389, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8921: Wed May 21 14:01:10 2025
  write: IOPS=443, BW=1773KiB/s (1815kB/s)(10.0MiB/5777msec)
    slat (usec): min=11, max=1348, avg=69.47, stdev=80.43
    clat (usec): min=4917, max=63725, avg=35880.82, stdev=6690.89
     lat (usec): min=4968, max=63776, avg=35951.67, stdev=6693.88
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[26346], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[32113], 40.00th=[33817], 50.00th=[34866], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[44303], 95.00th=[47973],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[57934], 99.95th=[59507],
     | 99.99th=[63701]
   bw (  KiB/s): min= 1536, max= 2032, per=24.98%, avg=1770.91, stdev=143.60, samples=11
   iops        : min=  384, max=  508, avg=442.73, stdev=35.90, samples=11
  lat (msec)   : 10=0.08%, 20=0.55%, 50=96.05%, 100=3.32%
  cpu          : usr=1.04%, sys=3.41%, ctx=1475, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7090KiB/s (7260kB/s), 1773KiB/s-1781KiB/s (1815kB/s-1824kB/s), io=40.0MiB (41.9MB), run=5750-5777msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=8935: Wed May 21 14:01:17 2025
   read: IOPS=245, BW=983KiB/s (1006kB/s)(5048KiB/5137msec)
    slat (usec): min=20, max=5376, avg=1924.34, stdev=607.85
    clat (usec): min=3571, max=76444, avg=30494.97, stdev=6661.04
     lat (usec): min=3615, max=79522, avg=32422.27, stdev=6856.13
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[21627], 10.00th=[23462], 20.00th=[25297],
     | 30.00th=[27132], 40.00th=[28705], 50.00th=[30016], 60.00th=[31327],
     | 70.00th=[32900], 80.00th=[34866], 90.00th=[37487], 95.00th=[41681],
     | 99.00th=[50070], 99.50th=[55837], 99.90th=[73925], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  784, max= 1136, per=25.38%, avg=984.60, stdev=128.22, samples=10
   iops        : min=  196, max=  284, avg=246.10, stdev=32.05, samples=10
  write: IOPS=252, BW=1011KiB/s (1035kB/s)(5192KiB/5137msec)
    slat (usec): min=13, max=1826, avg=65.40, stdev=84.15
    clat (usec): min=5083, max=74390, avg=31598.40, stdev=6205.22
     lat (usec): min=5150, max=74431, avg=31665.32, stdev=6210.25
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[23200], 10.00th=[25035], 20.00th=[27132],
     | 30.00th=[28443], 40.00th=[29754], 50.00th=[31065], 60.00th=[32375],
     | 70.00th=[33817], 80.00th=[35914], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[48497], 99.50th=[52691], 99.90th=[70779], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  832, max= 1128, per=25.18%, avg=1017.30, stdev=98.17, samples=10
   iops        : min=  208, max=  282, avg=254.30, stdev=24.52, samples=10
  lat (msec)   : 4=0.08%, 10=0.20%, 20=1.76%, 50=97.07%, 100=0.90%
  cpu          : usr=1.15%, sys=5.92%, ctx=2543, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8936: Wed May 21 14:01:17 2025
   read: IOPS=241, BW=965KiB/s (988kB/s)(4976KiB/5159msec)
    slat (usec): min=1007, max=6618, avg=1929.41, stdev=573.30
    clat (usec): min=15243, max=80374, avg=30773.29, stdev=6367.10
     lat (usec): min=16636, max=81752, avg=32705.66, stdev=6465.17
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[22152], 10.00th=[23725], 20.00th=[25560],
     | 30.00th=[27132], 40.00th=[28443], 50.00th=[30278], 60.00th=[31589],
     | 70.00th=[33162], 80.00th=[35390], 90.00th=[38011], 95.00th=[41681],
     | 99.00th=[47973], 99.50th=[53216], 99.90th=[72877], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  848, max= 1128, per=24.99%, avg=969.60, stdev=98.18, samples=10
   iops        : min=  212, max=  282, avg=242.40, stdev=24.55, samples=10
  write: IOPS=255, BW=1020KiB/s (1045kB/s)(5264KiB/5159msec)
    slat (usec): min=14, max=1524, avg=64.49, stdev=78.74
    clat (usec): min=12536, max=84747, avg=31405.78, stdev=5971.31
     lat (usec): min=12624, max=84795, avg=31471.91, stdev=5969.19
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[23725], 10.00th=[25035], 20.00th=[26870],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30802], 60.00th=[31851],
     | 70.00th=[33162], 80.00th=[34866], 90.00th=[38536], 95.00th=[41681],
     | 99.00th=[47973], 99.50th=[52167], 99.90th=[81265], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min=  800, max= 1176, per=25.37%, avg=1024.80, stdev=114.42, samples=10
   iops        : min=  200, max=  294, avg=256.20, stdev=28.60, samples=10
  lat (msec)   : 20=0.94%, 50=98.28%, 100=0.78%
  cpu          : usr=1.43%, sys=5.82%, ctx=2407, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8937: Wed May 21 14:01:17 2025
   read: IOPS=240, BW=964KiB/s (987kB/s)(4984KiB/5171msec)
    slat (usec): min=953, max=5081, avg=1925.66, stdev=569.02
    clat (usec): min=4216, max=77077, avg=30529.56, stdev=6070.40
     lat (usec): min=5387, max=78763, avg=32458.53, stdev=6203.94
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[22152], 10.00th=[23462], 20.00th=[25560],
     | 30.00th=[27132], 40.00th=[28443], 50.00th=[30016], 60.00th=[31327],
     | 70.00th=[32900], 80.00th=[34866], 90.00th=[38011], 95.00th=[41681],
     | 99.00th=[47973], 99.50th=[48497], 99.90th=[53216], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  768, max= 1112, per=25.05%, avg=971.80, stdev=121.19, samples=10
   iops        : min=  192, max=  278, avg=242.90, stdev=30.29, samples=10
  write: IOPS=254, BW=1016KiB/s (1041kB/s)(5256KiB/5171msec)
    slat (usec): min=12, max=1253, avg=63.40, stdev=76.15
    clat (usec): min=5435, max=71881, avg=31624.57, stdev=5665.80
     lat (usec): min=5469, max=72015, avg=31689.41, stdev=5670.61
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[23725], 10.00th=[25297], 20.00th=[27395],
     | 30.00th=[28705], 40.00th=[30016], 50.00th=[31065], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[35390], 90.00th=[38536], 95.00th=[41157],
     | 99.00th=[48497], 99.50th=[50594], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  880, max= 1144, per=25.16%, avg=1016.50, stdev=92.79, samples=10
   iops        : min=  220, max=  286, avg=254.10, stdev=23.17, samples=10
  lat (msec)   : 10=0.12%, 20=0.82%, 50=98.63%, 100=0.43%
  cpu          : usr=0.00%, sys=7.08%, ctx=2427, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8938: Wed May 21 14:01:17 2025
   read: IOPS=244, BW=979KiB/s (1002kB/s)(5060KiB/5170msec)
    slat (usec): min=797, max=5955, avg=1957.57, stdev=579.92
    clat (usec): min=7755, max=52813, avg=30745.88, stdev=5815.84
     lat (usec): min=9339, max=55406, avg=32706.27, stdev=5956.16
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[22414], 10.00th=[23725], 20.00th=[25822],
     | 30.00th=[27395], 40.00th=[28967], 50.00th=[30278], 60.00th=[31851],
     | 70.00th=[33424], 80.00th=[35390], 90.00th=[38011], 95.00th=[41157],
     | 99.00th=[47449], 99.50th=[50594], 99.90th=[52167], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min=  808, max= 1064, per=25.30%, avg=981.60, stdev=86.08, samples=10
   iops        : min=  202, max=  266, avg=245.40, stdev=21.52, samples=10
  write: IOPS=250, BW=1002KiB/s (1026kB/s)(5180KiB/5170msec)
    slat (usec): min=12, max=920, avg=64.01, stdev=69.25
    clat (usec): min=19125, max=55655, avg=31443.22, stdev=5605.38
     lat (usec): min=19182, max=55791, avg=31508.93, stdev=5610.90
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23725], 10.00th=[25035], 20.00th=[26870],
     | 30.00th=[28181], 40.00th=[29492], 50.00th=[30802], 60.00th=[31851],
     | 70.00th=[33424], 80.00th=[35914], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[49021], 99.50th=[50594], 99.90th=[53740], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=  808, max= 1168, per=24.83%, avg=1003.20, stdev=98.50, samples=10
   iops        : min=  202, max=  292, avg=250.80, stdev=24.63, samples=10
  lat (msec)   : 10=0.04%, 20=0.66%, 50=98.59%, 100=0.70%
  cpu          : usr=0.00%, sys=7.25%, ctx=2569, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3881KiB/s (3974kB/s), 964KiB/s-983KiB/s (987kB/s-1006kB/s), io=19.6MiB (20.5MB), run=5137-5171msec
  WRITE: bw=4040KiB/s (4137kB/s), 1002KiB/s-1020KiB/s (1026kB/s-1045kB/s), io=20.4MiB (21.4MB), run=5137-5171msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=8946: Wed May 21 14:01:18 2025
   read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=8080.1k, max=8080.1k, avg=8080137.00, stdev= 0.00
    clat (nsec): min=96006k, max=96006k, avg=96005905.00, stdev= 0.00
     lat (nsec): min=104108k, max=104108k, avg=104107609.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[95945],  5.00th=[95945], 10.00th=[95945], 20.00th=[95945],
     | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945],
     | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945],
     | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945],
     | 99.99th=[95945]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.92%, sys=6.42%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout