== sanity test 398c: run fio to test AIO ================= 12:55:43 (1777913743)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.140251 s, 299 MB/s
osc.lustre-OST0000-osc-ffff9c8082b48800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183814: Mon May  4 12:55:52 2026
  write: IOPS=459, BW=1838KiB/s (1882kB/s)(10.0MiB/5572msec); 0 zone resets
    slat (usec): min=39, max=646, avg=181.77, stdev=78.99
    clat (usec): min=20470, max=55340, avg=34276.79, stdev=4138.67
     lat (usec): min=20644, max=55466, avg=34459.67, stdev=4156.10
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[26870], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[40109],
     | 99.00th=[44827], 99.50th=[49546], 99.90th=[52691], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1688, max= 1964, per=24.67%, avg=1813.20, stdev=85.63, samples=10
   iops        : min=  422, max=  491, avg=453.30, stdev=21.41, samples=10
  lat (msec)   : 50=99.61%, 100=0.39%
  cpu          : usr=0.70%, sys=6.53%, ctx=2441, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183815: Mon May  4 12:55:52 2026
  write: IOPS=460, BW=1841KiB/s (1886kB/s)(10.0MiB/5561msec); 0 zone resets
    slat (usec): min=41, max=30613, avg=186.42, stdev=606.49
    clat (usec): min=6551, max=58142, avg=34377.14, stdev=4243.26
     lat (usec): min=6668, max=58276, avg=34564.64, stdev=4233.70
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[27395], 10.00th=[29492], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[37487], 90.00th=[39060], 95.00th=[40633],
     | 99.00th=[45351], 99.50th=[51119], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1680, max= 2042, per=24.73%, avg=1817.80, stdev=109.16, samples=10
   iops        : min=  420, max=  510, avg=454.40, stdev=27.18, samples=10
  lat (msec)   : 10=0.04%, 20=0.31%, 50=99.10%, 100=0.55%
  cpu          : usr=0.40%, sys=6.92%, ctx=2537, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183816: Mon May  4 12:55:52 2026
  write: IOPS=462, BW=1851KiB/s (1895kB/s)(10.0MiB/5533msec); 0 zone resets
    slat (usec): min=40, max=988, avg=173.74, stdev=85.73
    clat (usec): min=7363, max=59188, avg=34356.09, stdev=4269.13
     lat (usec): min=7522, max=59310, avg=34530.89, stdev=4282.23
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[27132], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[39584], 95.00th=[40109],
     | 99.00th=[43779], 99.50th=[49546], 99.90th=[53740], 99.95th=[56361],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1704, max= 2080, per=24.85%, avg=1826.40, stdev=117.88, samples=10
   iops        : min=  426, max=  520, avg=456.60, stdev=29.47, samples=10
  lat (msec)   : 10=0.08%, 20=0.70%, 50=98.83%, 100=0.39%
  cpu          : usr=0.34%, sys=6.33%, ctx=2509, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183817: Mon May  4 12:55:52 2026
  write: IOPS=459, BW=1838KiB/s (1883kB/s)(10.0MiB/5570msec); 0 zone resets
    slat (usec): min=45, max=633, avg=177.48, stdev=77.67
    clat (usec): min=16113, max=56803, avg=34290.52, stdev=4168.66
     lat (usec): min=16325, max=56899, avg=34469.23, stdev=4173.62
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[27132], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[39060], 95.00th=[40633],
     | 99.00th=[44303], 99.50th=[49546], 99.90th=[53740], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min= 1672, max= 1970, per=24.67%, avg=1813.80, stdev=84.91, samples=10
   iops        : min=  418, max=  492, avg=453.40, stdev=21.13, samples=10
  lat (msec)   : 20=0.27%, 50=99.49%, 100=0.23%
  cpu          : usr=0.54%, sys=7.06%, ctx=2560, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7351KiB/s (7527kB/s), 1838KiB/s-1851KiB/s (1882kB/s-1895kB/s), io=40.0MiB (41.9MB), run=5533-5572msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183827: Mon May  4 12:56:01 2026
  read: IOPS=216, BW=864KiB/s (885kB/s)(5048KiB/5840msec)
    slat (usec): min=1097, max=5905, avg=2442.74, stdev=664.37
    clat (usec): min=5307, max=53815, avg=34661.60, stdev=4919.40
     lat (usec): min=7627, max=57184, avg=37105.79, stdev=4994.62
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40633], 95.00th=[42730],
     | 99.00th=[46400], 99.50th=[49546], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  752, max=  944, per=24.94%, avg=856.91, stdev=62.48, samples=11
   iops        : min=  188, max=  236, avg=214.18, stdev=15.61, samples=11
  write: IOPS=222, BW=889KiB/s (910kB/s)(5192KiB/5840msec); 0 zone resets
    slat (usec): min=57, max=1127, avg=172.52, stdev=85.25
    clat (usec): min=18164, max=53773, avg=35131.73, stdev=4111.89
     lat (usec): min=18369, max=54060, avg=35305.45, stdev=4118.08
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[40109], 95.00th=[42206],
     | 99.00th=[46400], 99.50th=[49021], 99.90th=[50594], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  784, max= 1080, per=24.71%, avg=883.91, stdev=85.08, samples=11
   iops        : min=  196, max=  270, avg=220.91, stdev=21.34, samples=11
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.53%, 100=0.35%
  cpu          : usr=0.51%, sys=6.80%, ctx=4693, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183828: Mon May  4 12:56:01 2026
  read: IOPS=214, BW=857KiB/s (878kB/s)(4976KiB/5804msec)
    slat (usec): min=60, max=8439, avg=2448.82, stdev=678.33
    clat (usec): min=3206, max=52019, avg=34795.45, stdev=5005.70
     lat (usec): min=3495, max=53797, avg=37245.80, stdev=5061.16
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[46924], 99.50th=[47973], 99.90th=[49546], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min=  760, max=  936, per=24.81%, avg=852.36, stdev=58.72, samples=11
   iops        : min=  190, max=  234, avg=213.09, stdev=14.68, samples=11
  write: IOPS=226, BW=907KiB/s (929kB/s)(5264KiB/5804msec); 0 zone resets
    slat (usec): min=48, max=710, avg=173.63, stdev=80.11
    clat (usec): min=13731, max=51899, avg=35049.49, stdev=4091.87
     lat (usec): min=13934, max=52013, avg=35224.30, stdev=4092.53
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28967], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[40109], 95.00th=[42206],
     | 99.00th=[45351], 99.50th=[46400], 99.90th=[50594], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min=  768, max= 1007, per=25.27%, avg=903.91, stdev=61.07, samples=11
   iops        : min=  192, max=  251, avg=225.91, stdev=15.14, samples=11
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.27%, 50=99.45%, 100=0.16%
  cpu          : usr=0.45%, sys=6.84%, ctx=4604, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183829: Mon May  4 12:56:01 2026
  read: IOPS=213, BW=854KiB/s (875kB/s)(4984KiB/5833msec)
    slat (usec): min=995, max=7990, avg=2435.69, stdev=653.69
    clat (usec): min=4400, max=50986, avg=34684.12, stdev=5040.25
     lat (usec): min=6346, max=54739, avg=37121.59, stdev=5120.98
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40633], 95.00th=[42730],
     | 99.00th=[47973], 99.50th=[48497], 99.90th=[50070], 99.95th=[51119],
     | 99.99th=[51119]
   bw (  KiB/s): min=  680, max=  976, per=24.79%, avg=851.91, stdev=91.14, samples=11
   iops        : min=  170, max=  244, avg=212.91, stdev=22.77, samples=11
  write: IOPS=225, BW=901KiB/s (923kB/s)(5256KiB/5833msec); 0 zone resets
    slat (usec): min=51, max=1776, avg=170.45, stdev=92.53
    clat (usec): min=7151, max=52031, avg=35172.10, stdev=4143.64
     lat (usec): min=7253, max=52138, avg=35343.69, stdev=4147.81
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[45876], 99.50th=[47449], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min=  816, max= 1024, per=24.91%, avg=890.91, stdev=71.40, samples=11
   iops        : min=  204, max=  256, avg=222.73, stdev=17.85, samples=11
  lat (msec)   : 10=0.12%, 20=0.12%, 50=99.65%, 100=0.12%
  cpu          : usr=0.43%, sys=6.84%, ctx=4615, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183830: Mon May  4 12:56:01 2026
  read: IOPS=216, BW=866KiB/s (887kB/s)(5060KiB/5840msec)
    slat (usec): min=1013, max=6677, avg=2411.89, stdev=642.63
    clat (usec): min=13231, max=52619, avg=34725.18, stdev=4771.96
     lat (usec): min=14654, max=55157, avg=37138.87, stdev=4834.55
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27395], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40633], 95.00th=[42730],
     | 99.00th=[46924], 99.50th=[47973], 99.90th=[49546], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min=  736, max=  920, per=24.93%, avg=856.45, stdev=50.93, samples=11
   iops        : min=  184, max=  230, avg=214.09, stdev=12.75, samples=11
  write: IOPS=221, BW=887KiB/s (908kB/s)(5180KiB/5840msec); 0 zone resets
    slat (usec): min=49, max=1162, avg=173.13, stdev=89.44
    clat (usec): min=11886, max=48914, avg=35076.40, stdev=4349.05
     lat (usec): min=12041, max=49017, avg=35250.66, stdev=4352.25
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28443], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[45876], 99.50th=[47449], 99.90th=[48497], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min=  792, max=  976, per=24.70%, avg=883.36, stdev=63.18, samples=11
   iops        : min=  198, max=  244, avg=220.82, stdev=15.82, samples=11
  lat (msec)   : 20=0.27%, 50=99.69%, 100=0.04%
  cpu          : usr=0.45%, sys=6.82%, ctx=4604, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3436KiB/s (3519kB/s), 854KiB/s-866KiB/s (875kB/s-887kB/s), io=19.6MiB (20.5MB), run=5804-5840msec
  WRITE: bw=3577KiB/s (3663kB/s), 887KiB/s-907KiB/s (908kB/s-929kB/s), io=20.4MiB (21.4MB), run=5804-5840msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183835: Mon May  4 12:56:05 2026
  read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=9438.7k, max=9438.7k, avg=9438726.00, stdev= 0.00
    clat (nsec): min=75735k, max=75735k, avg=75735151.00, stdev= 0.00
     lat (nsec): min=85177k, max=85177k, avg=85177347.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[76022],  5.00th=[76022], 10.00th=[76022], 20.00th=[76022],
     | 30.00th=[76022], 40.00th=[76022], 50.00th=[76022], 60.00th=[76022],
     | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022],
     | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.11%, ctx=8, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout