== sanity test 398c: run fio to test AIO ================= 13:36:16 (1772908576)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.153809 s, 273 MB/s
osc.lustre-OST0000-osc-ffff9c4f04a65000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=192098: Sat Mar  7 13:36:25 2026
  write: IOPS=422, BW=1691KiB/s (1732kB/s)(10.0MiB/6055msec); 0 zone resets
    slat (usec): min=40, max=1295, avg=239.33, stdev=145.64
    clat (usec): min=11624, max=67325, avg=37550.85, stdev=9935.21
     lat (usec): min=11789, max=67878, avg=37791.46, stdev=9978.53
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[26870], 10.00th=[28181], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32900], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[39584], 80.00th=[45351], 90.00th=[53740], 95.00th=[60556],
     | 99.00th=[65274], 99.50th=[66323], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1088, max= 2160, per=24.99%, avg=1681.73, stdev=348.47, samples=11
   iops        : min=  272, max=  540, avg=420.36, stdev=87.03, samples=11
  lat (msec)   : 20=0.23%, 50=86.91%, 100=12.85%
  cpu          : usr=0.48%, sys=7.55%, ctx=2730, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192099: Sat Mar  7 13:36:25 2026
  write: IOPS=420, BW=1683KiB/s (1723kB/s)(10.0MiB/6086msec); 0 zone resets
    slat (usec): min=43, max=21832, avg=250.67, stdev=453.10
    clat (usec): min=8493, max=68431, avg=37515.50, stdev=9995.89
     lat (usec): min=8597, max=68664, avg=37767.36, stdev=10025.46
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26870], 10.00th=[28181], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32900], 50.00th=[33817], 60.00th=[35914],
     | 70.00th=[39584], 80.00th=[45876], 90.00th=[53740], 95.00th=[60031],
     | 99.00th=[65799], 99.50th=[66323], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1096, max= 2176, per=24.81%, avg=1669.64, stdev=343.60, samples=11
   iops        : min=  274, max=  544, avg=417.36, stdev=85.85, samples=11
  lat (msec)   : 10=0.04%, 20=0.31%, 50=86.21%, 100=13.44%
  cpu          : usr=0.46%, sys=7.76%, ctx=2762, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192100: Sat Mar  7 13:36:25 2026
  write: IOPS=421, BW=1685KiB/s (1725kB/s)(10.0MiB/6077msec); 0 zone resets
    slat (usec): min=47, max=23717, avg=248.23, stdev=486.63
    clat (usec): min=8094, max=67366, avg=37483.60, stdev=9985.90
     lat (usec): min=8648, max=68431, avg=37733.06, stdev=10016.38
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26608], 10.00th=[28181], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[35914],
     | 70.00th=[39584], 80.00th=[45876], 90.00th=[53740], 95.00th=[60031],
     | 99.00th=[65274], 99.50th=[66323], 99.90th=[66847], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1088, max= 2144, per=24.92%, avg=1676.82, stdev=346.02, samples=11
   iops        : min=  272, max=  536, avg=419.18, stdev=86.48, samples=11
  lat (msec)   : 10=0.04%, 20=0.35%, 50=86.33%, 100=13.28%
  cpu          : usr=0.41%, sys=7.90%, ctx=2764, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192101: Sat Mar  7 13:36:25 2026
  write: IOPS=420, BW=1683KiB/s (1723kB/s)(10.0MiB/6086msec); 0 zone resets
    slat (usec): min=40, max=1206, avg=233.99, stdev=141.17
    clat (usec): min=5338, max=67848, avg=37510.84, stdev=9960.85
     lat (usec): min=5791, max=68115, avg=37746.06, stdev=10002.45
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26870], 10.00th=[28181], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[35914],
     | 70.00th=[39584], 80.00th=[45351], 90.00th=[53740], 95.00th=[60556],
     | 99.00th=[65274], 99.50th=[65799], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1072, max= 2152, per=24.85%, avg=1672.73, stdev=342.76, samples=11
   iops        : min=  268, max=  538, avg=418.18, stdev=85.69, samples=11
  lat (msec)   : 10=0.04%, 20=0.39%, 50=86.76%, 100=12.81%
  cpu          : usr=0.43%, sys=7.66%, ctx=2736, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6730KiB/s (6892kB/s), 1683KiB/s-1691KiB/s (1723kB/s-1732kB/s), io=40.0MiB (41.9MB), run=6055-6086msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=192112: Sat Mar  7 13:36:34 2026
  read: IOPS=223, BW=893KiB/s (915kB/s)(5048KiB/5652msec)
    slat (usec): min=972, max=7775, avg=2454.74, stdev=695.78
    clat (usec): min=16760, max=44928, avg=33529.86, stdev=4059.57
     lat (usec): min=18690, max=47675, avg=35985.81, stdev=4081.83
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[43779], 99.90th=[44827], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min=  784, max=  984, per=25.12%, avg=891.82, stdev=70.00, samples=11
   iops        : min=  196, max=  246, avg=222.91, stdev=17.56, samples=11
  write: IOPS=229, BW=919KiB/s (941kB/s)(5192KiB/5652msec); 0 zone resets
    slat (usec): min=50, max=652, avg=165.40, stdev=81.92
    clat (usec): min=16593, max=47647, avg=33856.61, stdev=3368.88
     lat (usec): min=16751, max=47729, avg=34023.01, stdev=3369.78
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28443], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[47449]
   bw (  KiB/s): min=  771, max= 1048, per=24.92%, avg=921.00, stdev=76.68, samples=11
   iops        : min=  192, max=  262, avg=230.18, stdev=19.32, samples=11
  lat (msec)   : 20=0.20%, 50=99.80%
  cpu          : usr=0.48%, sys=6.55%, ctx=4919, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192113: Sat Mar  7 13:36:34 2026
  read: IOPS=220, BW=880KiB/s (902kB/s)(4976KiB/5652msec)
    slat (usec): min=1014, max=7296, avg=2442.11, stdev=702.08
    clat (usec): min=14963, max=46606, avg=33667.95, stdev=4158.58
     lat (usec): min=16777, max=48712, avg=36111.40, stdev=4190.16
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26870], 10.00th=[28443], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[40109],
     | 99.00th=[42730], 99.50th=[43779], 99.90th=[45351], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  740, max=  976, per=24.76%, avg=878.91, stdev=66.93, samples=11
   iops        : min=  185, max=  244, avg=219.73, stdev=16.73, samples=11
  write: IOPS=232, BW=931KiB/s (954kB/s)(5264KiB/5652msec); 0 zone resets
    slat (usec): min=41, max=679, avg=161.34, stdev=81.41
    clat (usec): min=14876, max=44587, avg=33804.91, stdev=3446.42
     lat (usec): min=14954, max=44726, avg=33967.34, stdev=3444.97
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28443], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[44303], 99.95th=[44827],
     | 99.99th=[44827]
   bw (  KiB/s): min=  848, max= 1008, per=25.25%, avg=933.09, stdev=52.61, samples=11
   iops        : min=  212, max=  252, avg=233.27, stdev=13.15, samples=11
  lat (msec)   : 20=0.20%, 50=99.80%
  cpu          : usr=0.51%, sys=6.49%, ctx=4803, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192114: Sat Mar  7 13:36:34 2026
  read: IOPS=221, BW=888KiB/s (909kB/s)(4984KiB/5615msec)
    slat (usec): min=40, max=5349, avg=2437.89, stdev=698.86
    clat (usec): min=4640, max=65824, avg=33556.13, stdev=4837.05
     lat (usec): min=4694, max=67416, avg=35995.34, stdev=4908.90
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[26608], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[37487], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[44303], 99.50th=[46400], 99.90th=[57934], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  832, max=  944, per=24.99%, avg=887.20, stdev=36.84, samples=10
   iops        : min=  208, max=  236, avg=221.80, stdev= 9.21, samples=10
  write: IOPS=234, BW=936KiB/s (959kB/s)(5256KiB/5615msec); 0 zone resets
    slat (usec): min=40, max=1126, avg=162.44, stdev=82.26
    clat (usec): min=21481, max=65540, avg=33940.51, stdev=3719.43
     lat (usec): min=21660, max=65606, avg=34104.16, stdev=3717.08
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28705], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[42730], 99.50th=[46400], 99.90th=[62129], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  792, max= 1064, per=25.42%, avg=939.70, stdev=91.28, samples=10
   iops        : min=  198, max=  266, avg=234.90, stdev=22.84, samples=10
  lat (msec)   : 10=0.23%, 20=0.12%, 50=99.30%, 100=0.35%
  cpu          : usr=0.48%, sys=6.57%, ctx=4768, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192115: Sat Mar  7 13:36:34 2026
  read: IOPS=224, BW=897KiB/s (918kB/s)(5060KiB/5644msec)
    slat (usec): min=1071, max=6934, avg=2444.20, stdev=687.19
    clat (usec): min=22143, max=54132, avg=33522.91, stdev=4050.00
     lat (usec): min=24360, max=57453, avg=35968.33, stdev=4078.86
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26870], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31327], 40.00th=[32375], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[43779], 99.50th=[44827], 99.90th=[51119], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  816, max= 1000, per=25.22%, avg=895.27, stdev=59.81, samples=11
   iops        : min=  204, max=  250, avg=223.82, stdev=14.95, samples=11
  write: IOPS=229, BW=918KiB/s (940kB/s)(5180KiB/5644msec); 0 zone resets
    slat (usec): min=61, max=3152, avg=165.29, stdev=116.80
    clat (usec): min=23558, max=57760, avg=33974.47, stdev=3491.36
     lat (usec): min=23639, max=57936, avg=34140.83, stdev=3495.46
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28443], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38011], 95.00th=[39060],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[57410], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  829, max=  976, per=24.83%, avg=917.55, stdev=53.09, samples=11
   iops        : min=  207, max=  244, avg=229.36, stdev=13.31, samples=11
  lat (msec)   : 50=99.80%, 100=0.20%
  cpu          : usr=0.41%, sys=6.66%, ctx=4846, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3551KiB/s (3636kB/s), 880KiB/s-897KiB/s (902kB/s-918kB/s), io=19.6MiB (20.5MB), run=5615-5652msec
  WRITE: bw=3696KiB/s (3785kB/s), 918KiB/s-936KiB/s (940kB/s-959kB/s), io=20.4MiB (21.4MB), run=5615-5652msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=192119: Sat Mar  7 13:36:38 2026
  read: IOPS=9, BW=392MiB/s (411MB/s)(40.0MiB/102msec)
    slat (nsec): min=7897.5k, max=7897.5k, avg=7897457.00, stdev= 0.00
    clat (nsec): min=88498k, max=88498k, avg=88498178.00, stdev= 0.00
     lat (nsec): min=96486k, max=96486k, avg=96486077.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[88605],  5.00th=[88605], 10.00th=[88605], 20.00th=[88605],
     | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[88605],
     | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605],
     | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605],
     | 99.99th=[88605]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.92%, ctx=6, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=392MiB/s (411MB/s), 392MiB/s-392MiB/s (411MB/s-411MB/s), io=40.0MiB (41.9MB), run=102-102msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout