== sanity test 398c: run fio to test AIO ================= 07:11:48 (1778843508)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.135024 s, 311 MB/s
osc.lustre-OST0000-osc-ffff93785d9c4000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183911: Fri May 15 07:11:58 2026
  write: IOPS=448, BW=1793KiB/s (1836kB/s)(10.0MiB/5712msec); 0 zone resets
    slat (usec): min=40, max=5016, avg=202.37, stdev=149.05
    clat (usec): min=6647, max=66301, avg=35425.45, stdev=5071.87
     lat (usec): min=6917, max=66423, avg=35629.51, stdev=5077.87
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[28181], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[40633], 95.00th=[41681],
     | 99.00th=[48497], 99.50th=[49021], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[66323]
   bw (  KiB/s): min= 1576, max= 2146, per=25.08%, avg=1785.64, stdev=144.14, samples=11
   iops        : min=  394, max=  536, avg=446.36, stdev=35.91, samples=11
  lat (msec)   : 10=0.04%, 20=1.13%, 50=98.48%, 100=0.35%
  cpu          : usr=0.53%, sys=6.90%, ctx=2904, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183912: Fri May 15 07:11:58 2026
  write: IOPS=446, BW=1786KiB/s (1829kB/s)(10.0MiB/5733msec); 0 zone resets
    slat (usec): min=57, max=29358, avg=212.15, stdev=589.68
    clat (usec): min=7117, max=50578, avg=35309.06, stdev=4850.31
     lat (usec): min=8388, max=50688, avg=35522.35, stdev=4818.35
    clat percentiles (usec):
     |  1.00th=[16712],  5.00th=[27919], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[47973], 99.50th=[48497], 99.90th=[49021], 99.95th=[50070],
     | 99.99th=[50594]
   bw (  KiB/s): min= 1616, max= 2025, per=24.95%, avg=1776.09, stdev=121.48, samples=11
   iops        : min=  404, max=  506, avg=444.00, stdev=30.32, samples=11
  lat (msec)   : 10=0.08%, 20=1.13%, 50=98.71%, 100=0.08%
  cpu          : usr=0.47%, sys=7.26%, ctx=2888, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183913: Fri May 15 07:11:58 2026
  write: IOPS=444, BW=1780KiB/s (1823kB/s)(10.0MiB/5753msec); 0 zone resets
    slat (usec): min=46, max=1816, avg=202.47, stdev=122.22
    clat (usec): min=8060, max=50436, avg=35406.74, stdev=4504.85
     lat (usec): min=8406, max=50690, avg=35610.48, stdev=4510.54
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[28443], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[47449], 99.50th=[47973], 99.90th=[49546], 99.95th=[49546],
     | 99.99th=[50594]
   bw (  KiB/s): min= 1616, max= 1955, per=24.86%, avg=1769.73, stdev=102.23, samples=11
   iops        : min=  404, max=  488, avg=442.36, stdev=25.42, samples=11
  lat (msec)   : 10=0.04%, 20=0.82%, 50=99.10%, 100=0.04%
  cpu          : usr=0.71%, sys=7.11%, ctx=2978, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183914: Fri May 15 07:11:58 2026
  write: IOPS=444, BW=1780KiB/s (1823kB/s)(10.0MiB/5753msec); 0 zone resets
    slat (usec): min=40, max=807, avg=190.80, stdev=105.47
    clat (usec): min=19120, max=50530, avg=35393.97, stdev=4516.74
     lat (usec): min=19237, max=50686, avg=35586.02, stdev=4527.53
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[28181], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[48497], 99.50th=[48497], 99.90th=[49021], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min= 1584, max= 1942, per=24.82%, avg=1767.09, stdev=98.76, samples=11
   iops        : min=  396, max=  485, avg=441.73, stdev=24.60, samples=11
  lat (msec)   : 20=0.62%, 50=99.30%, 100=0.08%
  cpu          : usr=0.47%, sys=6.52%, ctx=2750, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7120KiB/s (7291kB/s), 1780KiB/s-1793KiB/s (1823kB/s-1836kB/s), io=40.0MiB (41.9MB), run=5712-5753msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183924: Fri May 15 07:12:08 2026
  read: IOPS=184, BW=737KiB/s (754kB/s)(5048KiB/6853msec)
    slat (usec): min=99, max=16083, avg=2887.86, stdev=1131.90
    clat (usec): min=3930, max=72050, avg=40820.82, stdev=9240.00
     lat (usec): min=6513, max=78162, avg=43710.80, stdev=9675.44
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[29492], 20.00th=[32375],
     | 30.00th=[34866], 40.00th=[37487], 50.00th=[40109], 60.00th=[43254],
     | 70.00th=[46400], 80.00th=[49021], 90.00th=[53216], 95.00th=[55837],
     | 99.00th=[61080], 99.50th=[65799], 99.90th=[70779], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  577, max= 1000, per=25.25%, avg=737.92, stdev=146.30, samples=13
   iops        : min=  144, max=  250, avg=184.46, stdev=36.60, samples=13
  write: IOPS=189, BW=758KiB/s (776kB/s)(5192KiB/6853msec); 0 zone resets
    slat (usec): min=55, max=1519, avg=171.70, stdev=97.24
    clat (usec): min=19199, max=65495, avg=41331.88, stdev=8331.93
     lat (usec): min=19527, max=65607, avg=41504.71, stdev=8341.09
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[35390], 40.00th=[38011], 50.00th=[41681], 60.00th=[44303],
     | 70.00th=[46924], 80.00th=[49021], 90.00th=[52167], 95.00th=[54264],
     | 99.00th=[60031], 99.50th=[61604], 99.90th=[65274], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  544, max= 1008, per=24.83%, avg=755.54, stdev=145.17, samples=13
   iops        : min=  136, max=  252, avg=188.85, stdev=36.33, samples=13
  lat (msec)   : 4=0.04%, 10=0.12%, 20=0.12%, 50=82.85%, 100=16.88%
  cpu          : usr=0.44%, sys=5.85%, ctx=4594, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183925: Fri May 15 07:12:08 2026
  read: IOPS=181, BW=725KiB/s (742kB/s)(4976KiB/6865msec)
    slat (usec): min=1011, max=16317, avg=2902.03, stdev=1097.99
    clat (usec): min=18537, max=74031, avg=40594.52, stdev=8734.55
     lat (usec): min=21411, max=76823, avg=43498.28, stdev=9184.85
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27657], 10.00th=[29754], 20.00th=[32637],
     | 30.00th=[34866], 40.00th=[36963], 50.00th=[39584], 60.00th=[42730],
     | 70.00th=[45351], 80.00th=[49021], 90.00th=[52691], 95.00th=[54789],
     | 99.00th=[61604], 99.50th=[63701], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  532, max=  968, per=24.64%, avg=720.31, stdev=148.81, samples=13
   iops        : min=  133, max=  242, avg=180.08, stdev=37.20, samples=13
  write: IOPS=191, BW=767KiB/s (785kB/s)(5264KiB/6865msec); 0 zone resets
    slat (usec): min=53, max=2603, avg=166.95, stdev=105.44
    clat (usec): min=18413, max=66735, avg=41347.88, stdev=8152.48
     lat (usec): min=18530, max=66981, avg=41515.86, stdev=8167.50
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29230], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[35390], 40.00th=[38536], 50.00th=[41681], 60.00th=[44827],
     | 70.00th=[46924], 80.00th=[49021], 90.00th=[51643], 95.00th=[54264],
     | 99.00th=[58983], 99.50th=[60556], 99.90th=[64226], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  571, max= 1008, per=25.16%, avg=765.77, stdev=143.30, samples=13
   iops        : min=  142, max=  252, avg=191.38, stdev=35.91, samples=13
  lat (msec)   : 20=0.08%, 50=83.83%, 100=16.09%
  cpu          : usr=0.45%, sys=5.77%, ctx=4672, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183926: Fri May 15 07:12:08 2026
  read: IOPS=182, BW=731KiB/s (749kB/s)(4984KiB/6816msec)
    slat (usec): min=40, max=8916, avg=2915.36, stdev=1105.81
    clat (usec): min=5837, max=70272, avg=40851.38, stdev=9493.09
     lat (usec): min=8353, max=72637, avg=43768.38, stdev=9939.40
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[26870], 10.00th=[28967], 20.00th=[32113],
     | 30.00th=[34866], 40.00th=[38011], 50.00th=[40633], 60.00th=[43779],
     | 70.00th=[46400], 80.00th=[49021], 90.00th=[53216], 95.00th=[56361],
     | 99.00th=[60556], 99.50th=[62653], 99.90th=[65274], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  600, max=  928, per=25.10%, avg=733.54, stdev=114.56, samples=13
   iops        : min=  150, max=  232, avg=183.38, stdev=28.64, samples=13
  write: IOPS=192, BW=771KiB/s (790kB/s)(5256KiB/6816msec); 0 zone resets
    slat (usec): min=40, max=2107, avg=171.38, stdev=106.20
    clat (usec): min=12222, max=67442, avg=41167.35, stdev=8675.09
     lat (usec): min=12458, max=67609, avg=41339.75, stdev=8686.93
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28443], 10.00th=[30278], 20.00th=[32637],
     | 30.00th=[34866], 40.00th=[38011], 50.00th=[41157], 60.00th=[44303],
     | 70.00th=[46924], 80.00th=[49021], 90.00th=[52167], 95.00th=[54789],
     | 99.00th=[60556], 99.50th=[62129], 99.90th=[66847], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  600, max= 1056, per=25.16%, avg=765.77, stdev=160.70, samples=13
   iops        : min=  150, max=  264, avg=191.38, stdev=40.20, samples=13
  lat (msec)   : 10=0.20%, 20=0.43%, 50=81.56%, 100=17.81%
  cpu          : usr=0.40%, sys=5.85%, ctx=4596, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183927: Fri May 15 07:12:08 2026
  read: IOPS=184, BW=737KiB/s (755kB/s)(5060KiB/6862msec)
    slat (usec): min=990, max=8115, avg=2917.81, stdev=1061.58
    clat (usec): min=7051, max=65121, avg=40519.63, stdev=9033.50
     lat (usec): min=9704, max=68121, avg=43439.16, stdev=9489.20
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[27132], 10.00th=[29492], 20.00th=[32113],
     | 30.00th=[34866], 40.00th=[36963], 50.00th=[39584], 60.00th=[42730],
     | 70.00th=[45876], 80.00th=[49021], 90.00th=[52691], 95.00th=[55313],
     | 99.00th=[60031], 99.50th=[61604], 99.90th=[64750], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  552, max=  944, per=24.99%, avg=730.46, stdev=141.51, samples=13
   iops        : min=  138, max=  236, avg=182.62, stdev=35.38, samples=13
  write: IOPS=188, BW=755KiB/s (773kB/s)(5180KiB/6862msec); 0 zone resets
    slat (usec): min=54, max=784, avg=169.75, stdev=88.07
    clat (usec): min=11711, max=69144, avg=41385.20, stdev=8596.76
     lat (usec): min=11854, max=69261, avg=41556.04, stdev=8607.79
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[35390], 40.00th=[38011], 50.00th=[41681], 60.00th=[44827],
     | 70.00th=[46924], 80.00th=[49021], 90.00th=[52691], 95.00th=[54789],
     | 99.00th=[60556], 99.50th=[61080], 99.90th=[66323], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  558, max=  976, per=24.75%, avg=753.08, stdev=142.21, samples=13
   iops        : min=  139, max=  244, avg=188.23, stdev=35.61, samples=13
  lat (msec)   : 10=0.12%, 20=0.47%, 50=82.34%, 100=17.07%
  cpu          : usr=0.50%, sys=5.61%, ctx=4675, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2923KiB/s (2993kB/s), 725KiB/s-737KiB/s (742kB/s-755kB/s), io=19.6MiB (20.5MB), run=6816-6865msec
  WRITE: bw=3043KiB/s (3116kB/s), 755KiB/s-771KiB/s (773kB/s-790kB/s), io=20.4MiB (21.4MB), run=6816-6865msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183932: Fri May 15 07:12:11 2026
  read: IOPS=8, BW=336MiB/s (352MB/s)(40.0MiB/119msec)
    slat (nsec): min=9428.2k, max=9428.2k, avg=9428187.00, stdev= 0.00
    clat (nsec): min=103244k, max=103244k, avg=103243545.00, stdev= 0.00
     lat (nsec): min=112676k, max=112676k, avg=112676107.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  104],  5.00th=[  104], 10.00th=[  104], 20.00th=[  104],
     | 30.00th=[  104], 40.00th=[  104], 50.00th=[  104], 60.00th=[  104],
     | 70.00th=[  104], 80.00th=[  104], 90.00th=[  104], 95.00th=[  104],
     | 99.00th=[  104], 99.50th=[  104], 99.90th=[  104], 99.95th=[  104],
     | 99.99th=[  104]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.63%, ctx=9, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=336MiB/s (352MB/s), 336MiB/s-336MiB/s (352MB/s-352MB/s), io=40.0MiB (41.9MB), run=119-119msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout