== sanity test 398c: run fio to test AIO ================= 23:57:33 (1790395053)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.114713 s, 366 MB/s
osc.lustre-OST0000-osc-ffff93ced2046800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183331: Fri Sep 25 23:57:41 2026
  write: IOPS=563, BW=2253KiB/s (2307kB/s)(10.0MiB/4545msec); 0 zone resets
    slat (usec): min=24, max=24602, avg=99.59, stdev=487.41
    clat (usec): min=3664, max=48205, avg=28014.59, stdev=3937.14
     lat (usec): min=3819, max=48277, avg=28115.32, stdev=3936.82
    clat percentiles (usec):
     |  1.00th=[15008],  5.00th=[21103], 10.00th=[23987], 20.00th=[25560],
     | 30.00th=[26608], 40.00th=[27657], 50.00th=[28443], 60.00th=[29230],
     | 70.00th=[30016], 80.00th=[30802], 90.00th=[32113], 95.00th=[33424],
     | 99.00th=[35390], 99.50th=[40109], 99.90th=[42730], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min= 2088, max= 2517, per=24.96%, avg=2241.62, stdev=134.74, samples=8
   iops        : min=  522, max=  629, avg=560.38, stdev=33.61, samples=8
  lat (msec)   : 4=0.04%, 10=0.35%, 20=3.44%, 50=96.17%
  cpu          : usr=0.44%, sys=4.73%, ctx=2792, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183332: Fri Sep 25 23:57:41 2026
  write: IOPS=561, BW=2245KiB/s (2299kB/s)(10.0MiB/4561msec); 0 zone resets
    slat (usec): min=21, max=520, avg=86.08, stdev=52.60
    clat (usec): min=9444, max=42661, avg=28060.34, stdev=3657.13
     lat (usec): min=9552, max=42731, avg=28147.54, stdev=3664.61
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[21627], 10.00th=[23987], 20.00th=[25822],
     | 30.00th=[26608], 40.00th=[27395], 50.00th=[28443], 60.00th=[28967],
     | 70.00th=[30016], 80.00th=[30802], 90.00th=[32113], 95.00th=[33424],
     | 99.00th=[35390], 99.50th=[36963], 99.90th=[42206], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 2112, max= 2419, per=24.87%, avg=2233.38, stdev=103.79, samples=8
   iops        : min=  528, max=  604, avg=558.25, stdev=25.76, samples=8
  lat (msec)   : 10=0.12%, 20=3.05%, 50=96.84%
  cpu          : usr=0.53%, sys=3.95%, ctx=2653, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183333: Fri Sep 25 23:57:41 2026
  write: IOPS=561, BW=2245KiB/s (2299kB/s)(10.0MiB/4561msec); 0 zone resets
    slat (usec): min=26, max=465, avg=90.27, stdev=55.78
    clat (usec): min=9727, max=42416, avg=28074.78, stdev=3631.65
     lat (usec): min=9799, max=42507, avg=28166.13, stdev=3638.46
    clat percentiles (usec):
     |  1.00th=[16909],  5.00th=[21627], 10.00th=[23987], 20.00th=[25822],
     | 30.00th=[26608], 40.00th=[27657], 50.00th=[28181], 60.00th=[28967],
     | 70.00th=[29754], 80.00th=[30802], 90.00th=[32113], 95.00th=[33817],
     | 99.00th=[35390], 99.50th=[36439], 99.90th=[41681], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2096, max= 2408, per=24.83%, avg=2230.00, stdev=107.31, samples=8
   iops        : min=  524, max=  602, avg=557.50, stdev=26.83, samples=8
  lat (msec)   : 10=0.12%, 20=2.85%, 50=97.03%
  cpu          : usr=0.55%, sys=4.43%, ctx=2940, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183334: Fri Sep 25 23:57:41 2026
  write: IOPS=563, BW=2255KiB/s (2309kB/s)(10.0MiB/4542msec); 0 zone resets
    slat (usec): min=22, max=448, avg=85.41, stdev=50.05
    clat (usec): min=3187, max=63089, avg=28250.53, stdev=4258.65
     lat (usec): min=3295, max=63139, avg=28336.90, stdev=4262.46
    clat percentiles (usec):
     |  1.00th=[16188],  5.00th=[21365], 10.00th=[24249], 20.00th=[25822],
     | 30.00th=[26608], 40.00th=[27657], 50.00th=[28443], 60.00th=[29230],
     | 70.00th=[30016], 80.00th=[30802], 90.00th=[32375], 95.00th=[33817],
     | 99.00th=[40109], 99.50th=[42730], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min= 2112, max= 2536, per=25.00%, avg=2245.00, stdev=134.54, samples=8
   iops        : min=  528, max=  634, avg=561.25, stdev=33.64, samples=8
  lat (msec)   : 4=0.04%, 10=0.08%, 20=3.20%, 50=96.41%, 100=0.27%
  cpu          : usr=0.75%, sys=3.74%, ctx=2703, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8980KiB/s (9196kB/s), 2245KiB/s-2255KiB/s (2299kB/s-2309kB/s), io=40.0MiB (41.9MB), run=4542-4561msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183345: Fri Sep 25 23:57:50 2026
  read: IOPS=206, BW=828KiB/s (848kB/s)(5048KiB/6099msec)
    slat (usec): min=952, max=13474, avg=2533.99, stdev=907.93
    clat (usec): min=19620, max=57767, avg=36475.14, stdev=6742.08
     lat (usec): min=22156, max=60596, avg=39010.58, stdev=7077.68
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26870], 10.00th=[28181], 20.00th=[30540],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[53740], 99.50th=[54264], 99.90th=[57410], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  664, max=  960, per=25.11%, avg=826.27, stdev=99.68, samples=11
   iops        : min=  166, max=  240, avg=206.55, stdev=24.92, samples=11
  write: IOPS=212, BW=851KiB/s (872kB/s)(5192KiB/6099msec); 0 zone resets
    slat (usec): min=27, max=697, avg=85.02, stdev=50.66
    clat (usec): min=20688, max=57584, avg=36620.54, stdev=5985.10
     lat (usec): min=20764, max=57655, avg=36706.45, stdev=5986.26
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28181], 10.00th=[29754], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[45351], 95.00th=[46924],
     | 99.00th=[52691], 99.50th=[54264], 99.90th=[56886], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  672, max= 1024, per=25.15%, avg=861.45, stdev=125.25, samples=11
   iops        : min=  168, max=  256, avg=215.36, stdev=31.31, samples=11
  lat (msec)   : 20=0.04%, 50=97.34%, 100=2.62%
  cpu          : usr=0.61%, sys=3.85%, ctx=4748, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183346: Fri Sep 25 23:57:50 2026
  read: IOPS=204, BW=819KiB/s (838kB/s)(4976KiB/6078msec)
    slat (usec): min=984, max=13578, avg=2548.18, stdev=918.18
    clat (usec): min=6289, max=72345, avg=36506.51, stdev=7015.97
     lat (usec): min=9059, max=75393, avg=39056.46, stdev=7318.52
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26346], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[32113], 40.00th=[33817], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46400], 95.00th=[48497],
     | 99.00th=[52167], 99.50th=[57410], 99.90th=[69731], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  688, max=  968, per=24.96%, avg=821.18, stdev=107.80, samples=11
   iops        : min=  172, max=  242, avg=205.27, stdev=26.97, samples=11
  write: IOPS=216, BW=866KiB/s (887kB/s)(5264KiB/6078msec); 0 zone resets
    slat (usec): min=24, max=476, avg=85.12, stdev=47.45
    clat (usec): min=24432, max=72258, avg=36488.16, stdev=6266.46
     lat (usec): min=24516, max=72334, avg=36574.29, stdev=6268.52
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[27919], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[53216], 99.50th=[56886], 99.90th=[63177], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  680, max= 1032, per=25.71%, avg=880.45, stdev=118.61, samples=11
   iops        : min=  170, max=  258, avg=220.09, stdev=29.65, samples=11
  lat (msec)   : 10=0.04%, 20=0.04%, 50=97.23%, 100=2.70%
  cpu          : usr=0.51%, sys=3.97%, ctx=4618, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183347: Fri Sep 25 23:57:50 2026
  read: IOPS=204, BW=818KiB/s (837kB/s)(4984KiB/6096msec)
    slat (usec): min=947, max=26623, avg=2544.08, stdev=1151.33
    clat (usec): min=16912, max=58449, avg=36534.71, stdev=6620.54
     lat (usec): min=23203, max=61181, avg=39080.30, stdev=6891.67
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[28443], 20.00th=[30802],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[53216], 99.50th=[55313], 99.90th=[57410], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  640, max=  976, per=25.12%, avg=826.55, stdev=120.51, samples=11
   iops        : min=  160, max=  244, avg=206.64, stdev=30.13, samples=11
  write: IOPS=215, BW=862KiB/s (883kB/s)(5256KiB/6096msec); 0 zone resets
    slat (usec): min=30, max=2052, avg=87.82, stdev=75.00
    clat (usec): min=21189, max=62456, avg=36658.67, stdev=6219.21
     lat (usec): min=21266, max=62503, avg=36747.41, stdev=6220.10
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[27919], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[54789], 99.50th=[56886], 99.90th=[61080], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  608, max= 1088, per=25.23%, avg=864.18, stdev=128.99, samples=11
   iops        : min=  152, max=  272, avg=216.00, stdev=32.26, samples=11
  lat (msec)   : 20=0.04%, 50=96.68%, 100=3.28%
  cpu          : usr=0.51%, sys=3.89%, ctx=4695, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183348: Fri Sep 25 23:57:50 2026
  read: IOPS=208, BW=835KiB/s (855kB/s)(5060KiB/6059msec)
    slat (usec): min=40, max=35869, avg=2540.42, stdev=1359.49
    clat (usec): min=3273, max=62486, avg=36255.48, stdev=7243.93
     lat (usec): min=3437, max=64733, avg=38797.58, stdev=7505.57
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26346], 10.00th=[27919], 20.00th=[30016],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[57410], 99.50th=[59507], 99.90th=[61604], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  648, max=  976, per=25.51%, avg=839.18, stdev=104.27, samples=11
   iops        : min=  162, max=  244, avg=209.73, stdev=26.02, samples=11
  write: IOPS=213, BW=855KiB/s (875kB/s)(5180KiB/6059msec); 0 zone resets
    slat (usec): min=30, max=907, avg=85.33, stdev=57.16
    clat (usec): min=19085, max=77509, avg=36737.46, stdev=6399.80
     lat (usec): min=19221, max=77576, avg=36823.86, stdev=6398.46
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[29492], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[55313], 99.50th=[59507], 99.90th=[64750], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  680, max= 1008, per=25.07%, avg=858.64, stdev=113.21, samples=11
   iops        : min=  170, max=  252, avg=214.64, stdev=28.30, samples=11
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.16%, 50=96.33%, 100=3.40%
  cpu          : usr=0.66%, sys=3.73%, ctx=4663, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3290KiB/s (3369kB/s), 818KiB/s-835KiB/s (837kB/s-855kB/s), io=19.6MiB (20.5MB), run=6059-6099msec
  WRITE: bw=3425KiB/s (3508kB/s), 851KiB/s-866KiB/s (872kB/s-887kB/s), io=20.4MiB (21.4MB), run=6059-6099msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183353: Fri Sep 25 23:57:53 2026
  read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=11576k, max=11576k, avg=11576365.00, stdev= 0.00
    clat (nsec): min=93096k, max=93096k, avg=93095692.00, stdev= 0.00
     lat (nsec): min=104676k, max=104676k, avg=104676142.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[92799],  5.00th=[92799], 10.00th=[92799], 20.00th=[92799],
     | 30.00th=[92799], 40.00th=[92799], 50.00th=[92799], 60.00th=[92799],
     | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799],
     | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799],
     | 99.99th=[92799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.17%, ctx=10, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout