== sanity test 398c: run fio to test AIO ================= 03:20:02 (1789370402)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.11042 s, 380 MB/s
osc.lustre-OST0000-osc-ffff928d079c1800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182739: Mon Sep 14 03:20:10 2026
  write: IOPS=499, BW=1996KiB/s (2044kB/s)(10.0MiB/5129msec); 0 zone resets
    slat (usec): min=27, max=668, avg=111.19, stdev=74.25
    clat (usec): min=8747, max=57266, avg=31884.46, stdev=3826.48
     lat (usec): min=8808, max=57305, avg=31996.79, stdev=3826.92
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26084], 10.00th=[27657], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33424], 80.00th=[34341], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[45351], 99.50th=[52167], 99.90th=[57410], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min= 1920, max= 2184, per=25.15%, avg=1993.67, stdev=82.72, samples=9
   iops        : min=  480, max=  546, avg=498.33, stdev=20.60, samples=9
  lat (msec)   : 10=0.08%, 20=0.16%, 50=99.26%, 100=0.51%
  cpu          : usr=0.41%, sys=4.45%, ctx=2845, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182740: Mon Sep 14 03:20:10 2026
  write: IOPS=496, BW=1986KiB/s (2033kB/s)(10.0MiB/5157msec); 0 zone resets
    slat (usec): min=31, max=27307, avg=119.88, stdev=542.42
    clat (usec): min=4472, max=63667, avg=31777.06, stdev=3513.50
     lat (usec): min=4547, max=63735, avg=31898.07, stdev=3508.32
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[26346], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33424], 80.00th=[34341], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[39060], 99.50th=[43254], 99.90th=[52691], 99.95th=[55837],
     | 99.99th=[63701]
   bw (  KiB/s): min= 1912, max= 2192, per=25.06%, avg=1986.40, stdev=79.47, samples=10
   iops        : min=  478, max=  548, avg=496.60, stdev=19.87, samples=10
  lat (msec)   : 10=0.08%, 20=0.35%, 50=99.34%, 100=0.23%
  cpu          : usr=0.52%, sys=4.77%, ctx=2953, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182741: Mon Sep 14 03:20:10 2026
  write: IOPS=495, BW=1982KiB/s (2030kB/s)(10.0MiB/5166msec); 0 zone resets
    slat (usec): min=23, max=556, avg=94.32, stdev=64.79
    clat (usec): min=17001, max=47198, avg=31802.87, stdev=3294.92
     lat (usec): min=17042, max=47248, avg=31898.18, stdev=3297.17
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26084], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33817], 80.00th=[34341], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[38536], 99.50th=[44303], 99.90th=[46400], 99.95th=[46924],
     | 99.99th=[47449]
   bw (  KiB/s): min= 1920, max= 2112, per=24.97%, avg=1979.80, stdev=64.71, samples=10
   iops        : min=  480, max=  528, avg=494.90, stdev=16.21, samples=10
  lat (msec)   : 20=0.23%, 50=99.77%
  cpu          : usr=0.58%, sys=3.70%, ctx=2470, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182742: Mon Sep 14 03:20:10 2026
  write: IOPS=495, BW=1982KiB/s (2030kB/s)(10.0MiB/5166msec); 0 zone resets
    slat (usec): min=29, max=711, avg=106.75, stdev=74.61
    clat (usec): min=18460, max=54684, avg=31819.75, stdev=3274.63
     lat (usec): min=18538, max=54786, avg=31927.55, stdev=3278.37
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26084], 10.00th=[27919], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637],
     | 70.00th=[33424], 80.00th=[34341], 90.00th=[35390], 95.00th=[36439],
     | 99.00th=[39060], 99.50th=[41157], 99.90th=[51119], 99.95th=[51119],
     | 99.99th=[54789]
   bw (  KiB/s): min= 1880, max= 2176, per=25.01%, avg=1983.00, stdev=79.44, samples=10
   iops        : min=  470, max=  544, avg=495.70, stdev=19.89, samples=10
  lat (msec)   : 20=0.12%, 50=99.69%, 100=0.20%
  cpu          : usr=0.29%, sys=4.78%, ctx=2910, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7929KiB/s (8119kB/s), 1982KiB/s-1996KiB/s (2030kB/s-2044kB/s), io=40.0MiB (41.9MB), run=5129-5166msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182752: Mon Sep 14 03:20:20 2026
  read: IOPS=197, BW=789KiB/s (808kB/s)(5048KiB/6394msec)
    slat (usec): min=22, max=5870, avg=2635.37, stdev=845.38
    clat (usec): min=16450, max=77149, avg=38344.08, stdev=6927.90
     lat (usec): min=16544, max=81204, avg=40981.03, stdev=7204.61
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[29230], 10.00th=[31065], 20.00th=[33162],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[47449], 95.00th=[50594],
     | 99.00th=[61080], 99.50th=[63701], 99.90th=[74974], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  568, max=  888, per=25.18%, avg=784.67, stdev=98.60, samples=12
   iops        : min=  142, max=  222, avg=196.17, stdev=24.65, samples=12
  write: IOPS=203, BW=812KiB/s (831kB/s)(5192KiB/6394msec); 0 zone resets
    slat (usec): min=25, max=603, avg=98.81, stdev=61.97
    clat (usec): min=9888, max=74112, avg=38743.50, stdev=6702.47
     lat (usec): min=9953, max=74176, avg=38843.27, stdev=6704.30
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[30802], 10.00th=[32375], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[46924], 95.00th=[51119],
     | 99.00th=[61604], 99.50th=[62653], 99.90th=[71828], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  648, max=  952, per=24.98%, avg=810.50, stdev=93.35, samples=12
   iops        : min=  162, max=  238, avg=202.58, stdev=23.33, samples=12
  lat (msec)   : 10=0.12%, 20=0.35%, 50=93.75%, 100=5.78%
  cpu          : usr=0.45%, sys=4.30%, ctx=4839, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182753: Mon Sep 14 03:20:20 2026
  read: IOPS=193, BW=773KiB/s (791kB/s)(4976KiB/6439msec)
    slat (usec): min=1018, max=8848, avg=2642.55, stdev=853.45
    clat (usec): min=15473, max=76450, avg=38290.34, stdev=6979.50
     lat (usec): min=17834, max=80221, avg=40934.55, stdev=7254.40
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[46924], 95.00th=[51119],
     | 99.00th=[60556], 99.50th=[66323], 99.90th=[73925], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  600, max=  912, per=24.48%, avg=762.67, stdev=88.65, samples=12
   iops        : min=  150, max=  228, avg=190.67, stdev=22.16, samples=12
  write: IOPS=204, BW=818KiB/s (837kB/s)(5264KiB/6439msec); 0 zone resets
    slat (usec): min=24, max=755, avg=95.84, stdev=57.02
    clat (usec): min=15313, max=69262, avg=38948.80, stdev=6217.33
     lat (usec): min=15461, max=69315, avg=39045.60, stdev=6219.26
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[31065], 10.00th=[32375], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[43254], 90.00th=[46924], 95.00th=[50594],
     | 99.00th=[60031], 99.50th=[63177], 99.90th=[67634], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  704, max=  928, per=25.19%, avg=817.08, stdev=72.18, samples=12
   iops        : min=  176, max=  232, avg=204.25, stdev=18.05, samples=12
  lat (msec)   : 20=0.12%, 50=94.10%, 100=5.78%
  cpu          : usr=0.51%, sys=4.24%, ctx=4801, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182754: Mon Sep 14 03:20:20 2026
  read: IOPS=193, BW=774KiB/s (793kB/s)(4984KiB/6437msec)
    slat (usec): min=1120, max=6685, avg=2648.93, stdev=811.21
    clat (usec): min=17177, max=71868, avg=38572.73, stdev=7106.95
     lat (usec): min=18990, max=74815, avg=41223.35, stdev=7331.56
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29230], 10.00th=[31065], 20.00th=[33162],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[47973], 95.00th=[52691],
     | 99.00th=[61604], 99.50th=[64750], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  568, max=  904, per=24.77%, avg=771.92, stdev=98.76, samples=12
   iops        : min=  142, max=  226, avg=192.92, stdev=24.72, samples=12
  write: IOPS=204, BW=817KiB/s (836kB/s)(5256KiB/6437msec); 0 zone resets
    slat (usec): min=34, max=632, avg=102.74, stdev=62.67
    clat (usec): min=15658, max=69553, avg=38674.84, stdev=6350.56
     lat (usec): min=15765, max=69648, avg=38778.63, stdev=6351.30
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30540], 10.00th=[32113], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[47973], 95.00th=[51119],
     | 99.00th=[57934], 99.50th=[62653], 99.90th=[66847], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  632, max=  960, per=24.81%, avg=804.83, stdev=95.37, samples=12
   iops        : min=  158, max=  240, avg=201.17, stdev=23.85, samples=12
  lat (msec)   : 20=0.12%, 50=92.70%, 100=7.19%
  cpu          : usr=0.54%, sys=4.26%, ctx=4897, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182755: Mon Sep 14 03:20:20 2026
  read: IOPS=196, BW=786KiB/s (805kB/s)(5060KiB/6436msec)
    slat (usec): min=1090, max=7171, avg=2662.59, stdev=837.85
    clat (usec): min=22141, max=73379, avg=38673.35, stdev=7085.52
     lat (usec): min=24734, max=77462, avg=41337.59, stdev=7291.45
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[31065], 20.00th=[33162],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[47973], 95.00th=[52167],
     | 99.00th=[62129], 99.50th=[63701], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  648, max=  880, per=25.00%, avg=778.92, stdev=65.96, samples=12
   iops        : min=  162, max=  220, avg=194.67, stdev=16.50, samples=12
  write: IOPS=201, BW=805KiB/s (824kB/s)(5180KiB/6436msec); 0 zone resets
    slat (usec): min=32, max=2979, avg=102.78, stdev=99.92
    clat (usec): min=7176, max=70335, avg=38608.99, stdev=6227.27
     lat (usec): min=7247, max=70413, avg=38712.84, stdev=6228.67
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[30802], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[46924], 95.00th=[50594],
     | 99.00th=[58459], 99.50th=[62129], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  664, max=  944, per=24.70%, avg=801.25, stdev=79.63, samples=12
   iops        : min=  166, max=  236, avg=200.25, stdev=19.92, samples=12
  lat (msec)   : 10=0.04%, 50=93.98%, 100=5.98%
  cpu          : usr=0.44%, sys=4.34%, ctx=4885, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3117KiB/s (3191kB/s), 773KiB/s-789KiB/s (791kB/s-808kB/s), io=19.6MiB (20.5MB), run=6394-6439msec
  WRITE: bw=3245KiB/s (3322kB/s), 805KiB/s-818KiB/s (824kB/s-837kB/s), io=20.4MiB (21.4MB), run=6394-6439msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182759: Mon Sep 14 03:20:23 2026
  read: IOPS=9, BW=381MiB/s (399MB/s)(40.0MiB/105msec)
    slat (nsec): min=14628k, max=14628k, avg=14628403.00, stdev= 0.00
    clat (nsec): min=85061k, max=85061k, avg=85060508.00, stdev= 0.00
     lat (nsec): min=99693k, max=99693k, avg=99693170.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[85459],  5.00th=[85459], 10.00th=[85459], 20.00th=[85459],
     | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459],
     | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459],
     | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=12.50%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=381MiB/s (399MB/s), 381MiB/s-381MiB/s (399MB/s-399MB/s), io=40.0MiB (41.9MB), run=105-105msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout