== sanity test 398c: run fio to test AIO ================= 23:22:11 (1784172131)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.162113 s, 259 MB/s
osc.lustre-OST0000-osc-ffff8c0caa21e800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=194131: Wed Jul 15 23:22:23 2026
  write: IOPS=332, BW=1330KiB/s (1362kB/s)(10.0MiB/7701msec); 0 zone resets
    slat (usec): min=41, max=24127, avg=360.04, stdev=507.28
    clat (usec): min=10102, max=84934, avg=47625.03, stdev=9287.61
     lat (usec): min=10474, max=85505, avg=47986.97, stdev=9341.88
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[26870], 10.00th=[29492], 20.00th=[44303],
     | 30.00th=[47449], 40.00th=[49546], 50.00th=[50594], 60.00th=[51119],
     | 70.00th=[52167], 80.00th=[53740], 90.00th=[55313], 95.00th=[56886],
     | 99.00th=[60556], 99.50th=[64750], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min= 1160, max= 1912, per=24.97%, avg=1324.33, stdev=222.07, samples=15
   iops        : min=  290, max=  478, avg=331.07, stdev=55.52, samples=15
  lat (msec)   : 20=0.78%, 50=42.30%, 100=56.91%
  cpu          : usr=0.49%, sys=9.16%, ctx=2756, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194132: Wed Jul 15 23:22:23 2026
  write: IOPS=331, BW=1326KiB/s (1358kB/s)(10.0MiB/7723msec); 0 zone resets
    slat (usec): min=60, max=1485, avg=323.57, stdev=184.56
    clat (usec): min=10060, max=65025, avg=47657.57, stdev=9122.70
     lat (usec): min=10314, max=65346, avg=47982.85, stdev=9182.10
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[27132], 10.00th=[29492], 20.00th=[45351],
     | 30.00th=[47449], 40.00th=[49546], 50.00th=[50594], 60.00th=[51643],
     | 70.00th=[52167], 80.00th=[53740], 90.00th=[55313], 95.00th=[56886],
     | 99.00th=[59507], 99.50th=[62129], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[65274]
   bw (  KiB/s): min= 1168, max= 1912, per=24.85%, avg=1317.73, stdev=211.18, samples=15
   iops        : min=  292, max=  478, avg=329.40, stdev=52.77, samples=15
  lat (msec)   : 20=1.17%, 50=41.02%, 100=57.81%
  cpu          : usr=0.51%, sys=8.42%, ctx=2722, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194133: Wed Jul 15 23:22:23 2026
  write: IOPS=333, BW=1335KiB/s (1367kB/s)(10.0MiB/7668msec); 0 zone resets
    slat (usec): min=39, max=1150, avg=351.92, stdev=176.05
    clat (msec): min=5, max=101, avg=47.51, stdev= 9.61
     lat (msec): min=5, max=101, avg=47.86, stdev= 9.68
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   27], 10.00th=[   30], 20.00th=[   45],
     | 30.00th=[   48], 40.00th=[   50], 50.00th=[   51], 60.00th=[   52],
     | 70.00th=[   53], 80.00th=[   54], 90.00th=[   56], 95.00th=[   57],
     | 99.00th=[   62], 99.50th=[   65], 99.90th=[   96], 99.95th=[   96],
     | 99.99th=[  102]
   bw (  KiB/s): min= 1144, max= 1912, per=24.99%, avg=1325.27, stdev=230.22, samples=15
   iops        : min=  286, max=  478, avg=331.27, stdev=57.48, samples=15
  lat (msec)   : 10=0.12%, 20=1.37%, 50=41.45%, 100=57.03%, 250=0.04%
  cpu          : usr=0.55%, sys=9.09%, ctx=2579, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194134: Wed Jul 15 23:22:23 2026
  write: IOPS=331, BW=1327KiB/s (1359kB/s)(10.0MiB/7714msec); 0 zone resets
    slat (usec): min=56, max=22017, avg=362.56, stdev=466.38
    clat (msec): min=7, max=103, avg=47.57, stdev= 9.60
     lat (msec): min=10, max=104, avg=47.94, stdev= 9.64
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   27], 10.00th=[   30], 20.00th=[   45],
     | 30.00th=[   48], 40.00th=[   50], 50.00th=[   51], 60.00th=[   52],
     | 70.00th=[   53], 80.00th=[   54], 90.00th=[   56], 95.00th=[   57],
     | 99.00th=[   61], 99.50th=[   64], 99.90th=[   99], 99.95th=[   99],
     | 99.99th=[  105]
   bw (  KiB/s): min= 1160, max= 1896, per=24.94%, avg=1322.73, stdev=209.30, samples=15
   iops        : min=  290, max=  474, avg=330.67, stdev=52.33, samples=15
  lat (msec)   : 10=0.04%, 20=1.21%, 50=42.66%, 100=56.05%, 250=0.04%
  cpu          : usr=0.57%, sys=9.17%, ctx=2718, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5304KiB/s (5431kB/s), 1326KiB/s-1335KiB/s (1358kB/s-1367kB/s), io=40.0MiB (41.9MB), run=7668-7723msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=194145: Wed Jul 15 23:22:32 2026
  read: IOPS=186, BW=745KiB/s (763kB/s)(5048KiB/6772msec)
    slat (usec): min=1107, max=11277, avg=3140.54, stdev=1065.53
    clat (usec): min=20794, max=62290, avg=40032.24, stdev=6488.67
     lat (usec): min=23298, max=65685, avg=43174.25, stdev=6699.89
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[29230], 10.00th=[31589], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[38536], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[47973], 95.00th=[50070],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[59507], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  656, max=  840, per=25.02%, avg=741.38, stdev=51.60, samples=13
   iops        : min=  164, max=  210, avg=185.31, stdev=12.88, samples=13
  write: IOPS=191, BW=767KiB/s (785kB/s)(5192KiB/6772msec); 0 zone resets
    slat (usec): min=50, max=1771, avg=164.07, stdev=93.58
    clat (usec): min=9318, max=61298, avg=40182.76, stdev=5919.46
     lat (usec): min=9473, max=61467, avg=40347.71, stdev=5916.45
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30278], 10.00th=[32113], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[45351], 90.00th=[47449], 95.00th=[49021],
     | 99.00th=[53740], 99.50th=[55313], 99.90th=[58983], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  672, max= 1072, per=24.91%, avg=768.46, stdev=107.56, samples=13
   iops        : min=  168, max=  268, avg=192.08, stdev=26.90, samples=13
  lat (msec)   : 10=0.04%, 50=95.51%, 100=4.45%
  cpu          : usr=0.40%, sys=5.61%, ctx=4568, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194146: Wed Jul 15 23:22:32 2026
  read: IOPS=183, BW=735KiB/s (753kB/s)(4976KiB/6769msec)
    slat (usec): min=1110, max=16974, avg=3148.07, stdev=1181.18
    clat (usec): min=22570, max=65124, avg=40163.68, stdev=6745.11
     lat (usec): min=24731, max=67043, avg=43313.12, stdev=7017.38
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[30016], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[63177], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  640, max=  896, per=24.79%, avg=734.46, stdev=67.60, samples=13
   iops        : min=  160, max=  224, avg=183.62, stdev=16.90, samples=13
  write: IOPS=194, BW=778KiB/s (796kB/s)(5264KiB/6769msec); 0 zone resets
    slat (usec): min=53, max=864, avg=166.79, stdev=86.52
    clat (usec): min=13308, max=62941, avg=40578.44, stdev=5885.95
     lat (usec): min=13406, max=63005, avg=40746.39, stdev=5888.12
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30802], 10.00th=[32637], 20.00th=[35390],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[45351], 90.00th=[47449], 95.00th=[49546],
     | 99.00th=[53740], 99.50th=[59507], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  680, max= 1000, per=25.14%, avg=775.54, stdev=93.17, samples=13
   iops        : min=  170, max=  250, avg=193.85, stdev=23.26, samples=13
  lat (msec)   : 20=0.12%, 50=94.18%, 100=5.70%
  cpu          : usr=0.34%, sys=5.66%, ctx=4495, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194147: Wed Jul 15 23:22:32 2026
  read: IOPS=184, BW=737KiB/s (755kB/s)(4984KiB/6761msec)
    slat (usec): min=76, max=17110, avg=3134.03, stdev=1219.02
    clat (usec): min=13313, max=75324, avg=40508.51, stdev=7341.67
     lat (usec): min=16789, max=75402, avg=43643.96, stdev=7494.71
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28967], 10.00th=[31327], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[38536], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49546], 95.00th=[52167],
     | 99.00th=[60031], 99.50th=[65799], 99.90th=[70779], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  640, max= 1000, per=24.85%, avg=736.23, stdev=100.42, samples=13
   iops        : min=  160, max=  250, avg=184.00, stdev=25.06, samples=13
  write: IOPS=194, BW=777KiB/s (796kB/s)(5256KiB/6761msec); 0 zone resets
    slat (usec): min=43, max=1003, avg=162.15, stdev=84.34
    clat (usec): min=18437, max=67213, avg=40596.87, stdev=6203.90
     lat (usec): min=18882, max=67280, avg=40760.00, stdev=6203.61
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30802], 10.00th=[32637], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[45351], 90.00th=[47449], 95.00th=[50070],
     | 99.00th=[58459], 99.50th=[63177], 99.90th=[65799], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  688, max=  912, per=25.10%, avg=774.31, stdev=63.71, samples=13
   iops        : min=  172, max=  228, avg=193.54, stdev=15.90, samples=13
  lat (msec)   : 20=0.12%, 50=92.77%, 100=7.11%
  cpu          : usr=0.33%, sys=5.59%, ctx=4463, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194148: Wed Jul 15 23:22:32 2026
  read: IOPS=186, BW=747KiB/s (765kB/s)(5060KiB/6772msec)
    slat (usec): min=998, max=16324, avg=3120.12, stdev=1167.54
    clat (usec): min=8913, max=61930, avg=40019.79, stdev=6861.23
     lat (usec): min=11668, max=65657, avg=43141.28, stdev=7049.59
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[30802], 20.00th=[33817],
     | 30.00th=[36439], 40.00th=[38536], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[48497], 95.00th=[50594],
     | 99.00th=[56361], 99.50th=[58459], 99.90th=[60556], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  616, max=  920, per=25.15%, avg=745.31, stdev=74.40, samples=13
   iops        : min=  154, max=  230, avg=186.31, stdev=18.60, samples=13
  write: IOPS=191, BW=765KiB/s (783kB/s)(5180KiB/6772msec); 0 zone resets
    slat (usec): min=51, max=768, avg=161.24, stdev=83.91
    clat (usec): min=14605, max=60071, avg=40515.21, stdev=5939.28
     lat (usec): min=14727, max=60312, avg=40677.41, stdev=5939.97
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30540], 10.00th=[32375], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[45351], 90.00th=[47449], 95.00th=[49546],
     | 99.00th=[54789], 99.50th=[55837], 99.90th=[59507], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  656, max=  976, per=24.66%, avg=760.85, stdev=93.37, samples=13
   iops        : min=  164, max=  244, avg=190.15, stdev=23.33, samples=13
  lat (msec)   : 10=0.04%, 20=0.27%, 50=94.69%, 100=5.00%
  cpu          : usr=0.49%, sys=5.42%, ctx=4536, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2963KiB/s (3034kB/s), 735KiB/s-747KiB/s (753kB/s-765kB/s), io=19.6MiB (20.5MB), run=6761-6772msec
  WRITE: bw=3085KiB/s (3159kB/s), 765KiB/s-778KiB/s (783kB/s-796kB/s), io=20.4MiB (21.4MB), run=6761-6772msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=194152: Wed Jul 15 23:22:36 2026
  read: IOPS=7, BW=286MiB/s (300MB/s)(40.0MiB/140msec)
    slat (nsec): min=8849.2k, max=8849.2k, avg=8849154.00, stdev= 0.00
    clat (nsec): min=123132k, max=123132k, avg=123131759.00, stdev= 0.00
     lat (nsec): min=131984k, max=131984k, avg=131983568.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  124],  5.00th=[  124], 10.00th=[  124], 20.00th=[  124],
     | 30.00th=[  124], 40.00th=[  124], 50.00th=[  124], 60.00th=[  124],
     | 70.00th=[  124], 80.00th=[  124], 90.00th=[  124], 95.00th=[  124],
     | 99.00th=[  124], 99.50th=[  124], 99.90th=[  124], 99.95th=[  124],
     | 99.99th=[  124]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.72%, sys=5.76%, ctx=5, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=286MiB/s (300MB/s), 286MiB/s-286MiB/s (300MB/s-300MB/s), io=40.0MiB (41.9MB), run=140-140msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout