== sanity test 398c: run fio to test AIO ================= 04:32:39 (1781685159)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.114234 s, 367 MB/s
osc.lustre-OST0000-osc-ffff90ed093df000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190656: Wed Jun 17 04:32:47 2026
  write: IOPS=536, BW=2146KiB/s (2198kB/s)(10.0MiB/4771msec); 0 zone resets
    slat (usec): min=48, max=21825, avg=181.05, stdev=437.96
    clat (usec): min=13795, max=65093, avg=29376.87, stdev=4609.20
     lat (usec): min=18496, max=65189, avg=29559.05, stdev=4619.77
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[23462], 10.00th=[24249], 20.00th=[25560],
     | 30.00th=[26346], 40.00th=[27395], 50.00th=[28443], 60.00th=[30016],
     | 70.00th=[31589], 80.00th=[33162], 90.00th=[35390], 95.00th=[38011],
     | 99.00th=[41681], 99.50th=[43779], 99.90th=[51643], 99.95th=[61604],
     | 99.99th=[65274]
   bw (  KiB/s): min= 1776, max= 2416, per=25.11%, avg=2154.56, stdev=251.48, samples=9
   iops        : min=  444, max=  604, avg=538.56, stdev=62.84, samples=9
  lat (msec)   : 20=0.35%, 50=99.53%, 100=0.12%
  cpu          : usr=0.44%, sys=7.53%, ctx=2849, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190657: Wed Jun 17 04:32:47 2026
  write: IOPS=536, BW=2145KiB/s (2196kB/s)(10.0MiB/4774msec); 0 zone resets
    slat (usec): min=44, max=701, avg=167.39, stdev=87.48
    clat (usec): min=17889, max=61578, avg=29357.00, stdev=4636.12
     lat (usec): min=18026, max=61750, avg=29525.50, stdev=4650.73
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[23200], 10.00th=[24249], 20.00th=[25560],
     | 30.00th=[26346], 40.00th=[27395], 50.00th=[28443], 60.00th=[30016],
     | 70.00th=[31589], 80.00th=[33162], 90.00th=[35390], 95.00th=[37487],
     | 99.00th=[41681], 99.50th=[43779], 99.90th=[57410], 99.95th=[57934],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1752, max= 2424, per=25.07%, avg=2150.44, stdev=253.66, samples=9
   iops        : min=  438, max=  606, avg=537.56, stdev=63.39, samples=9
  lat (msec)   : 20=0.16%, 50=99.65%, 100=0.20%
  cpu          : usr=0.63%, sys=6.75%, ctx=2808, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190658: Wed Jun 17 04:32:47 2026
  write: IOPS=536, BW=2145KiB/s (2196kB/s)(10.0MiB/4774msec); 0 zone resets
    slat (usec): min=41, max=826, avg=162.90, stdev=85.96
    clat (usec): min=2911, max=57628, avg=29376.03, stdev=4683.47
     lat (usec): min=3154, max=57773, avg=29539.97, stdev=4692.38
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[23462], 10.00th=[24249], 20.00th=[25560],
     | 30.00th=[26346], 40.00th=[27395], 50.00th=[28443], 60.00th=[30016],
     | 70.00th=[31589], 80.00th=[33162], 90.00th=[35390], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[44303], 99.90th=[54264], 99.95th=[54789],
     | 99.99th=[57410]
   bw (  KiB/s): min= 1800, max= 2432, per=25.09%, avg=2152.56, stdev=252.28, samples=9
   iops        : min=  450, max=  608, avg=538.11, stdev=63.06, samples=9
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.08%, 50=99.57%, 100=0.27%
  cpu          : usr=0.52%, sys=7.10%, ctx=2801, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190659: Wed Jun 17 04:32:47 2026
  write: IOPS=540, BW=2162KiB/s (2214kB/s)(10.0MiB/4737msec); 0 zone resets
    slat (usec): min=41, max=789, avg=165.14, stdev=92.77
    clat (usec): min=5044, max=66299, avg=29385.55, stdev=4780.47
     lat (usec): min=5134, max=66448, avg=29551.77, stdev=4798.84
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[23462], 10.00th=[24249], 20.00th=[25560],
     | 30.00th=[26084], 40.00th=[27395], 50.00th=[28443], 60.00th=[30016],
     | 70.00th=[31589], 80.00th=[33162], 90.00th=[35390], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[44303], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[66323]
   bw (  KiB/s): min= 1768, max= 2432, per=25.24%, avg=2165.11, stdev=257.20, samples=9
   iops        : min=  442, max=  608, avg=541.22, stdev=64.25, samples=9
  lat (msec)   : 10=0.08%, 20=0.27%, 50=99.49%, 100=0.16%
  cpu          : usr=0.46%, sys=6.76%, ctx=2983, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8580KiB/s (8786kB/s), 2145KiB/s-2162KiB/s (2196kB/s-2214kB/s), io=40.0MiB (41.9MB), run=4737-4774msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190670: Wed Jun 17 04:32:57 2026
  read: IOPS=198, BW=793KiB/s (812kB/s)(5048KiB/6365msec)
    slat (usec): min=867, max=7793, avg=2683.80, stdev=958.32
    clat (usec): min=18758, max=69607, avg=37617.38, stdev=7569.09
     lat (usec): min=20351, max=72863, avg=40303.31, stdev=7958.46
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27132], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[44303], 90.00th=[47973], 95.00th=[52167],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[66847], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  595, max=  984, per=25.01%, avg=788.25, stdev=134.75, samples=12
   iops        : min=  148, max=  246, avg=197.00, stdev=33.79, samples=12
  write: IOPS=203, BW=816KiB/s (835kB/s)(5192KiB/6365msec); 0 zone resets
    slat (usec): min=54, max=657, avg=163.40, stdev=82.47
    clat (usec): min=18652, max=66217, avg=38369.58, stdev=7216.61
     lat (usec): min=18748, max=66283, avg=38534.06, stdev=7221.06
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28705], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[39060],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[56361], 99.50th=[60031], 99.90th=[62129], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  633, max= 1008, per=24.74%, avg=812.08, stdev=108.77, samples=12
   iops        : min=  158, max=  252, avg=203.00, stdev=27.23, samples=12
  lat (msec)   : 20=0.08%, 50=92.97%, 100=6.95%
  cpu          : usr=0.46%, sys=6.03%, ctx=4577, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190671: Wed Jun 17 04:32:57 2026
  read: IOPS=195, BW=782KiB/s (801kB/s)(4976KiB/6365msec)
    slat (usec): min=1048, max=11651, avg=2710.94, stdev=974.74
    clat (usec): min=19375, max=67511, avg=37952.28, stdev=7694.72
     lat (usec): min=22061, max=72026, avg=40665.08, stdev=8052.17
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27395], 10.00th=[28967], 20.00th=[31589],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[36439], 60.00th=[39060],
     | 70.00th=[41681], 80.00th=[44827], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[58459], 99.50th=[60556], 99.90th=[65799], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  588, max=  976, per=24.59%, avg=775.00, stdev=139.32, samples=12
   iops        : min=  147, max=  244, avg=193.75, stdev=34.83, samples=12
  write: IOPS=206, BW=827KiB/s (847kB/s)(5264KiB/6365msec); 0 zone resets
    slat (usec): min=44, max=2705, avg=162.30, stdev=108.13
    clat (usec): min=14814, max=64041, avg=38317.81, stdev=7261.77
     lat (usec): min=14955, max=64122, avg=38481.20, stdev=7268.30
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28443], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[42206], 80.00th=[45351], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[60031], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  624, max=  992, per=25.14%, avg=825.00, stdev=107.41, samples=12
   iops        : min=  156, max=  248, avg=206.25, stdev=26.85, samples=12
  lat (msec)   : 20=0.16%, 50=92.70%, 100=7.15%
  cpu          : usr=0.47%, sys=6.05%, ctx=4520, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190672: Wed Jun 17 04:32:57 2026
  read: IOPS=195, BW=783KiB/s (802kB/s)(4984KiB/6363msec)
    slat (usec): min=1054, max=7663, avg=2757.45, stdev=938.12
    clat (usec): min=17167, max=71144, avg=37913.70, stdev=7641.81
     lat (usec): min=20023, max=74594, avg=40672.85, stdev=7995.80
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27395], 10.00th=[29230], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[41157], 80.00th=[44827], 90.00th=[48497], 95.00th=[52167],
     | 99.00th=[58459], 99.50th=[58983], 99.90th=[67634], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  601, max= 1024, per=24.88%, avg=784.08, stdev=141.52, samples=12
   iops        : min=  150, max=  256, avg=196.00, stdev=35.41, samples=12
  write: IOPS=206, BW=826KiB/s (846kB/s)(5256KiB/6363msec); 0 zone resets
    slat (usec): min=51, max=736, avg=166.94, stdev=84.91
    clat (usec): min=16337, max=64561, avg=38263.02, stdev=7392.77
     lat (usec): min=16430, max=64724, avg=38431.18, stdev=7398.23
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28443], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[36439], 60.00th=[39060],
     | 70.00th=[42206], 80.00th=[45351], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[56886], 99.50th=[60031], 99.90th=[63701], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  656, max= 1072, per=24.87%, avg=816.33, stdev=132.41, samples=12
   iops        : min=  164, max=  268, avg=204.08, stdev=33.10, samples=12
  lat (msec)   : 20=0.08%, 50=92.62%, 100=7.30%
  cpu          : usr=0.44%, sys=6.13%, ctx=4612, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190673: Wed Jun 17 04:32:57 2026
  read: IOPS=200, BW=800KiB/s (820kB/s)(5060KiB/6322msec)
    slat (usec): min=107, max=6623, avg=2677.94, stdev=960.84
    clat (usec): min=3416, max=64319, avg=37592.83, stdev=7870.87
     lat (usec): min=4697, max=68425, avg=40272.61, stdev=8260.66
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27395], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[41157], 80.00th=[44303], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[60556], 99.50th=[61604], 99.90th=[62653], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  680, max=  952, per=25.32%, avg=798.08, stdev=95.06, samples=12
   iops        : min=  170, max=  238, avg=199.50, stdev=23.79, samples=12
  write: IOPS=204, BW=819KiB/s (839kB/s)(5180KiB/6322msec); 0 zone resets
    slat (usec): min=55, max=772, avg=164.88, stdev=85.19
    clat (usec): min=13975, max=62598, avg=38460.19, stdev=7272.79
     lat (usec): min=14194, max=62761, avg=38626.06, stdev=7275.85
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36963], 60.00th=[39584],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[56361], 99.50th=[57410], 99.90th=[60031], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  648, max= 1000, per=24.85%, avg=815.42, stdev=124.97, samples=12
   iops        : min=  162, max=  250, avg=203.83, stdev=31.26, samples=12
  lat (msec)   : 4=0.04%, 10=0.23%, 20=0.12%, 50=92.81%, 100=6.80%
  cpu          : usr=0.43%, sys=6.06%, ctx=4604, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3153KiB/s (3229kB/s), 782KiB/s-800KiB/s (801kB/s-820kB/s), io=19.6MiB (20.5MB), run=6322-6365msec
  WRITE: bw=3282KiB/s (3361kB/s), 816KiB/s-827KiB/s (835kB/s-847kB/s), io=20.4MiB (21.4MB), run=6322-6365msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190678: Wed Jun 17 04:33:00 2026
  read: IOPS=11, BW=455MiB/s (477MB/s)(40.0MiB/88msec)
    slat (nsec): min=14297k, max=14297k, avg=14296978.00, stdev= 0.00
    clat (nsec): min=68837k, max=68837k, avg=68836588.00, stdev= 0.00
     lat (nsec): min=83138k, max=83138k, avg=83138308.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[68682],  5.00th=[68682], 10.00th=[68682], 20.00th=[68682],
     | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682],
     | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682],
     | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[68682]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=14.94%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=40.0MiB (41.9MB), run=88-88msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout