== sanity test 398c: run fio to test AIO ================= 00:18:43 (1789359523)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.124707 s, 336 MB/s
osc.lustre-OST0000-osc-ffff994d498ab800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=195344: Mon Sep 14 00:18:51 2026
  write: IOPS=554, BW=2216KiB/s (2270kB/s)(10.0MiB/4620msec); 0 zone resets
    slat (usec): min=20, max=485, avg=88.18, stdev=55.17
    clat (usec): min=8021, max=47831, avg=28683.56, stdev=3310.38
     lat (usec): min=8084, max=47910, avg=28772.75, stdev=3314.17
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[24249], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28705], 60.00th=[29492],
     | 70.00th=[30016], 80.00th=[30802], 90.00th=[32375], 95.00th=[33424],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[47973]
   bw (  KiB/s): min= 2080, max= 2424, per=25.26%, avg=2217.00, stdev=104.96, samples=8
   iops        : min=  520, max=  606, avg=554.25, stdev=26.24, samples=8
  lat (msec)   : 10=0.47%, 20=1.56%, 50=97.97%
  cpu          : usr=0.61%, sys=3.90%, ctx=2666, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195345: Mon Sep 14 00:18:51 2026
  write: IOPS=548, BW=2195KiB/s (2247kB/s)(10.0MiB/4666msec); 0 zone resets
    slat (usec): min=26, max=646, avg=92.30, stdev=58.65
    clat (usec): min=5864, max=42305, avg=28780.80, stdev=2987.23
     lat (usec): min=5967, max=42389, avg=28874.09, stdev=2986.12
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[24511], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28705], 60.00th=[29492],
     | 70.00th=[30016], 80.00th=[30802], 90.00th=[32113], 95.00th=[33162],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[42206], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2080, max= 2264, per=24.98%, avg=2192.33, stdev=66.78, samples=9
   iops        : min=  520, max=  566, avg=548.00, stdev=16.61, samples=9
  lat (msec)   : 10=0.04%, 20=1.33%, 50=98.63%
  cpu          : usr=0.43%, sys=4.72%, ctx=2756, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195346: Mon Sep 14 00:18:51 2026
  write: IOPS=553, BW=2216KiB/s (2269kB/s)(10.0MiB/4621msec); 0 zone resets
    slat (usec): min=23, max=984, avg=87.41, stdev=58.63
    clat (usec): min=3299, max=44024, avg=28744.37, stdev=3425.97
     lat (usec): min=3347, max=44085, avg=28832.71, stdev=3427.54
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[24511], 10.00th=[25297], 20.00th=[27132],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28705], 60.00th=[29492],
     | 70.00th=[30016], 80.00th=[30802], 90.00th=[32113], 95.00th=[33424],
     | 99.00th=[36439], 99.50th=[36963], 99.90th=[43779], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 2112, max= 2385, per=25.25%, avg=2216.12, stdev=86.12, samples=8
   iops        : min=  528, max=  596, avg=554.00, stdev=21.46, samples=8
  lat (msec)   : 4=0.04%, 10=0.62%, 20=1.45%, 50=97.89%
  cpu          : usr=0.58%, sys=3.81%, ctx=2699, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195347: Mon Sep 14 00:18:51 2026
  write: IOPS=549, BW=2196KiB/s (2249kB/s)(10.0MiB/4663msec); 0 zone resets
    slat (usec): min=20, max=640, avg=89.11, stdev=55.20
    clat (usec): min=15276, max=37309, avg=28741.55, stdev=2938.29
     lat (usec): min=15331, max=37407, avg=28831.67, stdev=2940.54
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[24249], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28705], 60.00th=[29492],
     | 70.00th=[30016], 80.00th=[30802], 90.00th=[32113], 95.00th=[33162],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[36963], 99.95th=[36963],
     | 99.99th=[37487]
   bw (  KiB/s): min= 2112, max= 2304, per=24.94%, avg=2189.00, stdev=66.61, samples=9
   iops        : min=  528, max=  576, avg=547.22, stdev=16.63, samples=9
  lat (msec)   : 20=1.80%, 50=98.20%
  cpu          : usr=0.64%, sys=3.90%, ctx=2687, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8778KiB/s (8989kB/s), 2195KiB/s-2216KiB/s (2247kB/s-2270kB/s), io=40.0MiB (41.9MB), run=4620-4666msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=195358: Mon Sep 14 00:19:00 2026
  read: IOPS=183, BW=733KiB/s (751kB/s)(5048KiB/6883msec)
    slat (usec): min=18, max=10299, avg=2874.21, stdev=1072.28
    clat (usec): min=15266, max=63945, avg=41524.03, stdev=6903.07
     lat (usec): min=15363, max=66903, avg=44399.73, stdev=7215.09
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[30016], 10.00th=[32900], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[46924], 90.00th=[50070], 95.00th=[52691],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[62653], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  648, max=  840, per=25.25%, avg=735.92, stdev=63.34, samples=13
   iops        : min=  162, max=  210, avg=183.92, stdev=15.91, samples=13
  write: IOPS=188, BW=754KiB/s (772kB/s)(5192KiB/6883msec); 0 zone resets
    slat (usec): min=21, max=634, avg=86.61, stdev=54.26
    clat (usec): min=21945, max=63013, avg=41334.68, stdev=5963.04
     lat (usec): min=21972, max=63059, avg=41422.14, stdev=5966.54
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[31065], 10.00th=[33817], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[60556], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  592, max=  936, per=24.84%, avg=753.77, stdev=79.33, samples=13
   iops        : min=  148, max=  234, avg=188.38, stdev=19.88, samples=13
  lat (msec)   : 20=0.35%, 50=91.52%, 100=8.12%
  cpu          : usr=0.38%, sys=3.62%, ctx=4667, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195359: Mon Sep 14 00:19:00 2026
  read: IOPS=180, BW=723KiB/s (741kB/s)(4976KiB/6878msec)
    slat (usec): min=952, max=7070, avg=2891.23, stdev=995.12
    clat (usec): min=22992, max=64638, avg=41167.44, stdev=6587.31
     lat (usec): min=24858, max=68922, avg=44060.38, stdev=6907.70
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[30016], 10.00th=[32900], 20.00th=[35390],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49546], 95.00th=[52167],
     | 99.00th=[57934], 99.50th=[58983], 99.90th=[62129], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  605, max=  880, per=24.86%, avg=724.69, stdev=84.29, samples=13
   iops        : min=  151, max=  220, avg=181.15, stdev=21.10, samples=13
  write: IOPS=191, BW=765KiB/s (784kB/s)(5264KiB/6878msec); 0 zone resets
    slat (usec): min=30, max=1941, avg=93.76, stdev=96.90
    clat (usec): min=25370, max=64451, avg=41526.55, stdev=6183.70
     lat (usec): min=25421, max=64614, avg=41621.49, stdev=6188.89
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[31589], 10.00th=[33817], 20.00th=[36439],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49546], 95.00th=[52691],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[61080], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  664, max=  904, per=25.31%, avg=768.31, stdev=73.11, samples=13
   iops        : min=  166, max=  226, avg=192.08, stdev=18.28, samples=13
  lat (msec)   : 50=90.94%, 100=9.06%
  cpu          : usr=0.49%, sys=3.52%, ctx=4749, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195360: Mon Sep 14 00:19:00 2026
  read: IOPS=182, BW=729KiB/s (747kB/s)(4984KiB/6836msec)
    slat (usec): min=21, max=10369, avg=2884.44, stdev=1057.55
    clat (usec): min=12452, max=64139, avg=41310.91, stdev=7124.86
     lat (usec): min=12496, max=66958, avg=44196.79, stdev=7466.57
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[29492], 10.00th=[32375], 20.00th=[35390],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[49546], 95.00th=[52691],
     | 99.00th=[57934], 99.50th=[59507], 99.90th=[62653], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  672, max=  776, per=25.17%, avg=733.85, stdev=31.77, samples=13
   iops        : min=  168, max=  194, avg=183.46, stdev= 7.94, samples=13
  write: IOPS=192, BW=769KiB/s (787kB/s)(5256KiB/6836msec); 0 zone resets
    slat (usec): min=20, max=483, avg=85.82, stdev=52.98
    clat (usec): min=9477, max=86867, avg=41120.19, stdev=7211.81
     lat (usec): min=9608, max=86924, avg=41206.90, stdev=7213.69
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[30016], 10.00th=[32900], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[49021], 95.00th=[52167],
     | 99.00th=[58459], 99.50th=[61604], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  632, max= 1040, per=25.31%, avg=768.08, stdev=112.13, samples=13
   iops        : min=  158, max=  260, avg=192.00, stdev=28.04, samples=13
  lat (msec)   : 10=0.08%, 20=0.62%, 50=90.20%, 100=9.10%
  cpu          : usr=0.34%, sys=3.63%, ctx=4664, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195361: Mon Sep 14 00:19:00 2026
  read: IOPS=183, BW=735KiB/s (753kB/s)(5060KiB/6883msec)
    slat (usec): min=882, max=10764, avg=2878.87, stdev=1060.32
    clat (usec): min=10161, max=64313, avg=41215.91, stdev=7037.16
     lat (usec): min=12800, max=67861, avg=44096.52, stdev=7303.54
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[30016], 10.00th=[32375], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41157], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[50070], 95.00th=[53740],
     | 99.00th=[59507], 99.50th=[60031], 99.90th=[63701], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  608, max=  880, per=25.21%, avg=734.77, stdev=78.98, samples=13
   iops        : min=  152, max=  220, avg=183.69, stdev=19.75, samples=13
  write: IOPS=188, BW=753KiB/s (771kB/s)(5180KiB/6883msec); 0 zone resets
    slat (usec): min=24, max=751, avg=85.16, stdev=55.05
    clat (usec): min=12807, max=63518, avg=41427.26, stdev=6095.39
     lat (usec): min=12879, max=63694, avg=41513.39, stdev=6098.25
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[31065], 10.00th=[33817], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[42730],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[55837], 99.50th=[56886], 99.90th=[58459], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  608, max=  928, per=24.80%, avg=752.69, stdev=87.14, samples=13
   iops        : min=  152, max=  232, avg=188.15, stdev=21.81, samples=13
  lat (msec)   : 20=0.31%, 50=90.59%, 100=9.10%
  cpu          : usr=0.46%, sys=3.56%, ctx=4631, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2916KiB/s (2986kB/s), 723KiB/s-735KiB/s (741kB/s-753kB/s), io=19.6MiB (20.5MB), run=6836-6883msec
  WRITE: bw=3035KiB/s (3108kB/s), 753KiB/s-769KiB/s (771kB/s-787kB/s), io=20.4MiB (21.4MB), run=6836-6883msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=195365: Mon Sep 14 00:19:04 2026
  read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=8842.0k, max=8842.0k, avg=8842990.00, stdev= 0.00
    clat (nsec): min=96578k, max=96578k, avg=96578059.00, stdev= 0.00
     lat (nsec): min=105425k, max=105425k, avg=105424980.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[96994],  5.00th=[96994], 10.00th=[96994], 20.00th=[96994],
     | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994],
     | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994],
     | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.34%, ctx=9, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout