== sanity test 398c: run fio to test AIO ================= 21:50:15 (1743213015)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.140169 s, 299 MB/s
osc.lustre-OST0000-osc-ffff8800b5c40800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=28653: Fri Mar 28 21:50:23 2025
  write: IOPS=369, BW=1478KiB/s (1514kB/s)(10.0MiB/6927msec)
    slat (usec): min=23, max=778, avg=90.59, stdev=51.38
    clat (usec): min=5544, max=88142, avg=43096.83, stdev=5689.07
     lat (usec): min=5583, max=88193, avg=43189.58, stdev=5689.77
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[32900], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[43779], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[50070],
     | 99.00th=[56361], 99.50th=[56886], 99.90th=[73925], 99.95th=[83362],
     | 99.99th=[88605]
   bw (  KiB/s): min= 1296, max= 1692, per=25.13%, avg=1480.69, stdev=106.38, samples=13
   iops        : min=  324, max=  423, avg=370.15, stdev=26.59, samples=13
  lat (msec)   : 10=0.31%, 20=0.23%, 50=94.77%, 100=4.69%
  cpu          : usr=0.62%, sys=4.01%, ctx=1564, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28654: Fri Mar 28 21:50:23 2025
  write: IOPS=368, BW=1473KiB/s (1509kB/s)(10.0MiB/6950msec)
    slat (usec): min=24, max=1171, avg=81.49, stdev=54.17
    clat (usec): min=7803, max=62076, avg=43175.81, stdev=5040.56
     lat (usec): min=7921, max=62122, avg=43259.44, stdev=5042.09
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[32900], 10.00th=[36439], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[43779], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[50070],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[58983], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min= 1296, max= 1680, per=24.95%, avg=1470.15, stdev=92.10, samples=13
   iops        : min=  324, max=  420, avg=367.54, stdev=23.02, samples=13
  lat (msec)   : 10=0.04%, 20=0.16%, 50=95.04%, 100=4.77%
  cpu          : usr=0.55%, sys=3.48%, ctx=1085, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28655: Fri Mar 28 21:50:23 2025
  write: IOPS=368, BW=1474KiB/s (1510kB/s)(10.0MiB/6945msec)
    slat (usec): min=18, max=795, avg=86.02, stdev=50.04
    clat (usec): min=5051, max=75329, avg=43207.73, stdev=5222.67
     lat (usec): min=5203, max=75464, avg=43296.03, stdev=5226.33
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[33162], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[43779], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[50070],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1336, max= 1664, per=25.07%, avg=1477.23, stdev=87.81, samples=13
   iops        : min=  334, max=  416, avg=369.31, stdev=21.95, samples=13
  lat (msec)   : 10=0.04%, 20=0.62%, 50=94.57%, 100=4.77%
  cpu          : usr=0.69%, sys=3.66%, ctx=1265, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28656: Fri Mar 28 21:50:23 2025
  write: IOPS=368, BW=1474KiB/s (1509kB/s)(10.0MiB/6949msec)
    slat (usec): min=21, max=585, avg=96.09, stdev=54.92
    clat (usec): min=9522, max=71703, avg=43184.10, stdev=5462.78
     lat (usec): min=9577, max=71821, avg=43282.66, stdev=5460.82
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[33162], 10.00th=[36963], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[43779], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[48497], 95.00th=[50070],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[66847], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1336, max= 1640, per=25.04%, avg=1475.69, stdev=89.48, samples=13
   iops        : min=  334, max=  410, avg=368.92, stdev=22.37, samples=13
  lat (msec)   : 10=0.04%, 20=0.62%, 50=94.30%, 100=5.04%
  cpu          : usr=0.73%, sys=3.99%, ctx=1658, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5894KiB/s (6035kB/s), 1473KiB/s-1478KiB/s (1509kB/s-1514kB/s), io=40.0MiB (41.9MB), run=6927-6950msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=28671: Fri Mar 28 21:50:28 2025
   read: IOPS=312, BW=1249KiB/s (1279kB/s)(5048KiB/4041msec)
    slat (usec): min=616, max=4299, avg=1617.11, stdev=489.82
    clat (usec): min=3860, max=39577, avg=23657.88, stdev=6150.86
     lat (usec): min=5044, max=42209, avg=25278.06, stdev=6471.58
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14746], 10.00th=[15926], 20.00th=[17695],
     | 30.00th=[19530], 40.00th=[21365], 50.00th=[23200], 60.00th=[25560],
     | 70.00th=[27395], 80.00th=[29492], 90.00th=[31589], 95.00th=[33817],
     | 99.00th=[38011], 99.50th=[38536], 99.90th=[39060], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min=  872, max= 1816, per=24.97%, avg=1239.00, stdev=324.35, samples=8
   iops        : min=  218, max=  454, avg=309.75, stdev=81.09, samples=8
  write: IOPS=321, BW=1285KiB/s (1316kB/s)(5192KiB/4041msec)
    slat (usec): min=13, max=2102, avg=76.30, stdev=71.12
    clat (usec): min=13196, max=39990, avg=24795.17, stdev=5786.48
     lat (usec): min=13280, max=40031, avg=24873.13, stdev=5794.24
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[15926], 10.00th=[17171], 20.00th=[18744],
     | 30.00th=[20579], 40.00th=[22938], 50.00th=[25297], 60.00th=[26870],
     | 70.00th=[28443], 80.00th=[30278], 90.00th=[32375], 95.00th=[34341],
     | 99.00th=[36963], 99.50th=[37487], 99.90th=[39584], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min=  904, max= 1704, per=24.80%, avg=1281.00, stdev=282.37, samples=8
   iops        : min=  226, max=  426, avg=320.25, stdev=70.59, samples=8
  lat (msec)   : 4=0.04%, 10=0.04%, 20=29.34%, 50=70.59%
  cpu          : usr=0.97%, sys=8.71%, ctx=2775, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28672: Fri Mar 28 21:50:28 2025
   read: IOPS=309, BW=1238KiB/s (1267kB/s)(4976KiB/4021msec)
    slat (usec): min=27, max=4914, avg=1609.34, stdev=499.91
    clat (usec): min=1975, max=45215, avg=23637.34, stdev=6200.32
     lat (usec): min=3140, max=47450, avg=25249.77, stdev=6515.62
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14746], 10.00th=[16057], 20.00th=[17695],
     | 30.00th=[19530], 40.00th=[21365], 50.00th=[23200], 60.00th=[25035],
     | 70.00th=[27132], 80.00th=[29230], 90.00th=[31851], 95.00th=[33817],
     | 99.00th=[38011], 99.50th=[41157], 99.90th=[43254], 99.95th=[45351],
     | 99.99th=[45351]
   bw (  KiB/s): min=  872, max= 1752, per=24.83%, avg=1232.25, stdev=315.42, samples=8
   iops        : min=  218, max=  438, avg=308.00, stdev=78.88, samples=8
  write: IOPS=327, BW=1309KiB/s (1341kB/s)(5264KiB/4021msec)
    slat (usec): min=13, max=490, avg=73.00, stdev=42.94
    clat (usec): min=8922, max=40317, avg=24823.45, stdev=5806.32
     lat (usec): min=8993, max=40471, avg=24898.14, stdev=5817.67
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[16188], 10.00th=[17171], 20.00th=[18744],
     | 30.00th=[20579], 40.00th=[22938], 50.00th=[25297], 60.00th=[26870],
     | 70.00th=[28443], 80.00th=[30016], 90.00th=[32375], 95.00th=[33817],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 1008, max= 1792, per=25.30%, avg=1307.25, stdev=269.73, samples=8
   iops        : min=  252, max=  448, avg=326.75, stdev=67.44, samples=8
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.16%, 20=28.83%, 50=70.94%
  cpu          : usr=1.12%, sys=8.51%, ctx=2711, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28673: Fri Mar 28 21:50:28 2025
   read: IOPS=308, BW=1234KiB/s (1264kB/s)(4984KiB/4038msec)
    slat (usec): min=692, max=4212, avg=1619.45, stdev=480.09
    clat (usec): min=3248, max=41913, avg=23779.78, stdev=6117.30
     lat (usec): min=4313, max=43976, avg=25402.46, stdev=6429.02
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14746], 10.00th=[15926], 20.00th=[17957],
     | 30.00th=[19530], 40.00th=[21627], 50.00th=[23462], 60.00th=[25560],
     | 70.00th=[27657], 80.00th=[28967], 90.00th=[31589], 95.00th=[34341],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min=  840, max= 1720, per=24.83%, avg=1232.00, stdev=269.84, samples=8
   iops        : min=  210, max=  430, avg=308.00, stdev=67.46, samples=8
  write: IOPS=325, BW=1302KiB/s (1333kB/s)(5256KiB/4038msec)
    slat (usec): min=12, max=430, avg=75.79, stdev=41.45
    clat (usec): min=13049, max=41078, avg=24602.77, stdev=5731.52
     lat (usec): min=13091, max=41173, avg=24680.22, stdev=5740.90
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[16188], 10.00th=[17171], 20.00th=[18744],
     | 30.00th=[20317], 40.00th=[22414], 50.00th=[25035], 60.00th=[26608],
     | 70.00th=[28181], 80.00th=[29754], 90.00th=[32113], 95.00th=[34341],
     | 99.00th=[36963], 99.50th=[37487], 99.90th=[39584], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  896, max= 1832, per=25.03%, avg=1293.00, stdev=355.38, samples=8
   iops        : min=  224, max=  458, avg=323.25, stdev=88.85, samples=8
  lat (msec)   : 4=0.04%, 20=30.16%, 50=69.80%
  cpu          : usr=1.02%, sys=8.72%, ctx=2779, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28674: Fri Mar 28 21:50:28 2025
   read: IOPS=312, BW=1251KiB/s (1281kB/s)(5060KiB/4044msec)
    slat (usec): min=661, max=4535, avg=1601.58, stdev=483.39
    clat (usec): min=11661, max=41259, avg=23618.45, stdev=6236.64
     lat (usec): min=12936, max=43096, avg=25222.96, stdev=6560.36
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14877], 10.00th=[16057], 20.00th=[17433],
     | 30.00th=[19268], 40.00th=[20841], 50.00th=[23200], 60.00th=[25297],
     | 70.00th=[27132], 80.00th=[29230], 90.00th=[32113], 95.00th=[34341],
     | 99.00th=[39060], 99.50th=[40109], 99.90th=[41157], 99.95th=[41157],
     | 99.99th=[41157]
   bw (  KiB/s): min=  816, max= 1784, per=25.05%, avg=1243.00, stdev=321.28, samples=8
   iops        : min=  204, max=  446, avg=310.75, stdev=80.32, samples=8
  write: IOPS=320, BW=1281KiB/s (1312kB/s)(5180KiB/4044msec)
    slat (usec): min=13, max=998, avg=76.19, stdev=53.96
    clat (usec): min=6577, max=39940, avg=24758.58, stdev=5819.53
     lat (usec): min=6647, max=39970, avg=24836.53, stdev=5827.31
    clat percentiles (usec):
     |  1.00th=[14091],  5.00th=[15926], 10.00th=[17171], 20.00th=[18744],
     | 30.00th=[20317], 40.00th=[23200], 50.00th=[25297], 60.00th=[26870],
     | 70.00th=[28443], 80.00th=[30016], 90.00th=[32637], 95.00th=[34341],
     | 99.00th=[36963], 99.50th=[38536], 99.90th=[39584], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min=  872, max= 1736, per=24.60%, avg=1271.00, stdev=303.23, samples=8
   iops        : min=  218, max=  434, avg=317.75, stdev=75.81, samples=8
  lat (msec)   : 10=0.04%, 20=31.21%, 50=68.75%
  cpu          : usr=0.77%, sys=8.76%, ctx=2796, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4962KiB/s (5082kB/s), 1234KiB/s-1251KiB/s (1264kB/s-1281kB/s), io=19.6MiB (20.5MB), run=4021-4044msec
  WRITE: bw=5166KiB/s (5290kB/s), 1281KiB/s-1309KiB/s (1312kB/s-1341kB/s), io=20.4MiB (21.4MB), run=4021-4044msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=28681: Fri Mar 28 21:50:30 2025
   read: IOPS=6, BW=263MiB/s (276MB/s)(40.0MiB/152msec)
    slat (nsec): min=5226.6k, max=5226.6k, avg=5226562.00, stdev= 0.00
    clat (nsec): min=140069k, max=140069k, avg=140069379.00, stdev= 0.00
     lat (nsec): min=145306k, max=145306k, avg=145305982.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  140],  5.00th=[  140], 10.00th=[  140], 20.00th=[  140],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  140]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.97%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=263MiB/s (276MB/s), 263MiB/s-263MiB/s (276MB/s-276MB/s), io=40.0MiB (41.9MB), run=152-152msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout