== sanity test 398c: run fio to test AIO ================= 14:26:41 (1782930401)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.141604 s, 296 MB/s
osc.lustre-OST0000-osc-ffff9d6d5140a000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=187385: Wed Jul  1 14:26:50 2026
  write: IOPS=476, BW=1905KiB/s (1950kB/s)(10.0MiB/5376msec); 0 zone resets
    slat (usec): min=43, max=1160, avg=189.33, stdev=102.91
    clat (usec): min=13284, max=47503, avg=33084.05, stdev=3563.94
     lat (usec): min=13577, max=47728, avg=33274.65, stdev=3566.03
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[27395], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[43254], 99.50th=[44303], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min= 1752, max= 2035, per=25.00%, avg=1904.30, stdev=86.91, samples=10
   iops        : min=  438, max=  508, avg=476.00, stdev=21.60, samples=10
  lat (msec)   : 20=0.51%, 50=99.49%
  cpu          : usr=0.67%, sys=6.72%, ctx=2821, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187386: Wed Jul  1 14:26:50 2026
  write: IOPS=477, BW=1909KiB/s (1955kB/s)(10.0MiB/5363msec); 0 zone resets
    slat (usec): min=48, max=25960, avg=200.46, stdev=519.87
    clat (usec): min=4017, max=60468, avg=33058.80, stdev=4168.64
     lat (usec): min=4169, max=60581, avg=33260.51, stdev=4154.63
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[27132], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34341], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[44303], 99.50th=[46924], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1792, max= 2150, per=25.12%, avg=1913.40, stdev=107.22, samples=10
   iops        : min=  448, max=  537, avg=478.30, stdev=26.68, samples=10
  lat (msec)   : 10=0.08%, 20=1.33%, 50=98.32%, 100=0.27%
  cpu          : usr=0.41%, sys=7.52%, ctx=2829, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187387: Wed Jul  1 14:26:50 2026
  write: IOPS=479, BW=1918KiB/s (1964kB/s)(10.0MiB/5338msec); 0 zone resets
    slat (usec): min=40, max=1979, avg=194.89, stdev=108.32
    clat (usec): min=13625, max=66204, avg=33108.83, stdev=4236.80
     lat (usec): min=13727, max=66457, avg=33304.89, stdev=4241.74
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[27132], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[44303], 99.50th=[47449], 99.90th=[62653], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min= 1800, max= 2202, per=25.23%, avg=1921.80, stdev=121.36, samples=10
   iops        : min=  450, max=  550, avg=480.40, stdev=30.21, samples=10
  lat (msec)   : 20=1.33%, 50=98.24%, 100=0.43%
  cpu          : usr=0.37%, sys=7.31%, ctx=2871, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187388: Wed Jul  1 14:26:50 2026
  write: IOPS=476, BW=1904KiB/s (1950kB/s)(10.0MiB/5377msec); 0 zone resets
    slat (usec): min=47, max=846, avg=189.31, stdev=101.71
    clat (usec): min=4867, max=52996, avg=33102.56, stdev=3803.12
     lat (usec): min=4965, max=53232, avg=33293.21, stdev=3802.88
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[27132], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[43254], 99.50th=[45351], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1792, max= 2086, per=25.02%, avg=1905.40, stdev=98.06, samples=10
   iops        : min=  448, max=  521, avg=476.30, stdev=24.41, samples=10
  lat (msec)   : 10=0.04%, 20=0.66%, 50=99.18%, 100=0.12%
  cpu          : usr=0.54%, sys=7.16%, ctx=2937, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7618KiB/s (7800kB/s), 1904KiB/s-1918KiB/s (1950kB/s-1964kB/s), io=40.0MiB (41.9MB), run=5338-5377msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=187398: Wed Jul  1 14:27:00 2026
  read: IOPS=197, BW=790KiB/s (809kB/s)(5048KiB/6389msec)
    slat (usec): min=1132, max=9172, avg=2724.17, stdev=814.06
    clat (usec): min=23170, max=67494, avg=38253.52, stdev=5629.76
     lat (usec): min=25287, max=70244, avg=40979.61, stdev=5774.75
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29754], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[42206], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[53740], 99.50th=[54789], 99.90th=[61604], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  592, max=  872, per=25.00%, avg=784.92, stdev=72.10, samples=12
   iops        : min=  148, max=  218, avg=196.17, stdev=18.04, samples=12
  write: IOPS=203, BW=813KiB/s (832kB/s)(5192KiB/6389msec); 0 zone resets
    slat (usec): min=66, max=2356, avg=193.47, stdev=132.06
    clat (usec): min=9615, max=59046, avg=38173.10, stdev=4898.21
     lat (usec): min=9745, max=59185, avg=38367.75, stdev=4894.15
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[31327], 10.00th=[32637], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[39584], 80.00th=[41157], 90.00th=[44303], 95.00th=[47973],
     | 99.00th=[53740], 99.50th=[55837], 99.90th=[58459], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  680, max=  936, per=24.84%, avg=811.75, stdev=73.01, samples=12
   iops        : min=  170, max=  234, avg=202.92, stdev=18.28, samples=12
  lat (msec)   : 10=0.04%, 20=0.04%, 50=96.84%, 100=3.09%
  cpu          : usr=0.56%, sys=6.73%, ctx=4592, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187399: Wed Jul  1 14:27:00 2026
  read: IOPS=194, BW=779KiB/s (798kB/s)(4976KiB/6385msec)
    slat (usec): min=1229, max=9881, avg=2759.09, stdev=845.57
    clat (usec): min=18447, max=62731, avg=38223.22, stdev=5680.30
     lat (usec): min=21019, max=66252, avg=40984.29, stdev=5838.75
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29492], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[41681], 90.00th=[44827], 95.00th=[48497],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[61604], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  648, max=  856, per=24.70%, avg=775.58, stdev=66.25, samples=12
   iops        : min=  162, max=  214, avg=193.83, stdev=16.63, samples=12
  write: IOPS=206, BW=824KiB/s (844kB/s)(5264KiB/6385msec); 0 zone resets
    slat (usec): min=63, max=3398, avg=192.13, stdev=144.11
    clat (usec): min=12503, max=62548, avg=38302.75, stdev=4990.07
     lat (usec): min=12631, max=62720, avg=38496.07, stdev=5002.08
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[31327], 10.00th=[32375], 20.00th=[34341],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[41681], 90.00th=[44303], 95.00th=[47449],
     | 99.00th=[53216], 99.50th=[55837], 99.90th=[58983], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  640, max=  912, per=25.24%, avg=824.83, stdev=75.58, samples=12
   iops        : min=  160, max=  228, avg=206.17, stdev=18.91, samples=12
  lat (msec)   : 20=0.08%, 50=96.72%, 100=3.20%
  cpu          : usr=0.53%, sys=6.85%, ctx=4491, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187400: Wed Jul  1 14:27:00 2026
  read: IOPS=194, BW=780KiB/s (799kB/s)(4984KiB/6391msec)
    slat (usec): min=1186, max=9516, avg=2769.90, stdev=844.04
    clat (usec): min=1326, max=60399, avg=38117.08, stdev=5884.21
     lat (usec): min=3811, max=64639, avg=40888.88, stdev=6013.56
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[31065], 20.00th=[33424],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[42206], 90.00th=[44827], 95.00th=[48497],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[60031], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  616, max=  888, per=24.89%, avg=781.42, stdev=78.99, samples=12
   iops        : min=  154, max=  222, avg=195.33, stdev=19.77, samples=12
  write: IOPS=205, BW=822KiB/s (842kB/s)(5256KiB/6391msec); 0 zone resets
    slat (usec): min=47, max=1128, avg=186.38, stdev=103.04
    clat (usec): min=17794, max=60806, avg=38313.98, stdev=4784.88
     lat (usec): min=17898, max=60922, avg=38501.66, stdev=4781.20
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[31589], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[41681], 90.00th=[44303], 95.00th=[47449],
     | 99.00th=[51643], 99.50th=[54789], 99.90th=[58983], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  672, max=  928, per=24.95%, avg=815.33, stdev=81.83, samples=12
   iops        : min=  168, max=  232, avg=203.83, stdev=20.46, samples=12
  lat (msec)   : 2=0.04%, 20=0.08%, 50=96.88%, 100=3.01%
  cpu          : usr=0.56%, sys=6.74%, ctx=4529, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187401: Wed Jul  1 14:27:00 2026
  read: IOPS=199, BW=800KiB/s (819kB/s)(5060KiB/6328msec)
    slat (usec): min=70, max=6698, avg=2742.22, stdev=825.66
    clat (usec): min=4862, max=63406, avg=37769.58, stdev=5905.60
     lat (usec): min=5057, max=68065, avg=40513.66, stdev=6115.58
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[29230], 10.00th=[31327], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[41681], 90.00th=[44303], 95.00th=[47449],
     | 99.00th=[55837], 99.50th=[57934], 99.90th=[61080], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  584, max=  880, per=25.45%, avg=799.25, stdev=77.94, samples=12
   iops        : min=  146, max=  220, avg=199.75, stdev=19.47, samples=12
  write: IOPS=204, BW=819KiB/s (838kB/s)(5180KiB/6328msec); 0 zone resets
    slat (usec): min=65, max=3047, avg=192.92, stdev=137.95
    clat (usec): min=12904, max=72112, avg=38258.49, stdev=5051.04
     lat (usec): min=13144, max=72323, avg=38452.60, stdev=5053.18
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[31065], 10.00th=[32637], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[36963], 50.00th=[38011], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[41681], 90.00th=[44303], 95.00th=[46924],
     | 99.00th=[53740], 99.50th=[56361], 99.90th=[66847], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  736, max=  888, per=24.98%, avg=816.33, stdev=44.62, samples=12
   iops        : min=  184, max=  222, avg=204.08, stdev=11.16, samples=12
  lat (msec)   : 10=0.20%, 20=0.12%, 50=96.84%, 100=2.85%
  cpu          : usr=0.41%, sys=7.10%, ctx=4568, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3140KiB/s (3215kB/s), 779KiB/s-800KiB/s (798kB/s-819kB/s), io=19.6MiB (20.5MB), run=6328-6391msec
  WRITE: bw=3269KiB/s (3347kB/s), 813KiB/s-824KiB/s (832kB/s-844kB/s), io=20.4MiB (21.4MB), run=6328-6391msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=187405: Wed Jul  1 14:27:04 2026
  read: IOPS=8, BW=339MiB/s (355MB/s)(40.0MiB/118msec)
    slat (nsec): min=13719k, max=13719k, avg=13719101.00, stdev= 0.00
    clat (nsec): min=91751k, max=91751k, avg=91751457.00, stdev= 0.00
     lat (nsec): min=105476k, max=105476k, avg=105476236.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=13.68%, ctx=11, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=40.0MiB (41.9MB), run=118-118msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout