== sanity test 398c: run fio to test AIO ================= 02:13:17 (1783145597)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.130626 s, 321 MB/s
osc.lustre-OST0000-osc-ffff8fa6974ae800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182726: Sat Jul  4 02:13:26 2026
  write: IOPS=448, BW=1795KiB/s (1838kB/s)(10.0MiB/5706msec); 0 zone resets
    slat (usec): min=41, max=1005, avg=196.74, stdev=108.60
    clat (usec): min=3202, max=56340, avg=35013.97, stdev=4227.92
     lat (usec): min=3596, max=56409, avg=35211.82, stdev=4236.74
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[27657], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[45876], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[56361]
   bw (  KiB/s): min= 1632, max= 1960, per=24.96%, avg=1791.45, stdev=88.51, samples=11
   iops        : min=  408, max=  490, avg=447.82, stdev=22.15, samples=11
  lat (msec)   : 4=0.04%, 50=99.92%, 100=0.04%
  cpu          : usr=0.44%, sys=6.89%, ctx=2640, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182727: Sat Jul  4 02:13:26 2026
  write: IOPS=449, BW=1797KiB/s (1840kB/s)(10.0MiB/5698msec); 0 zone resets
    slat (usec): min=44, max=25401, avg=203.48, stdev=509.61
    clat (usec): min=3959, max=56670, avg=35041.71, stdev=4495.98
     lat (usec): min=4143, max=56831, avg=35246.39, stdev=4473.81
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[27919], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40109], 95.00th=[42206],
     | 99.00th=[45351], 99.50th=[46400], 99.90th=[56361], 99.95th=[56361],
     | 99.99th=[56886]
   bw (  KiB/s): min= 1624, max= 1976, per=25.01%, avg=1795.27, stdev=102.87, samples=11
   iops        : min=  406, max=  494, avg=448.82, stdev=25.72, samples=11
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.59%, 50=98.98%, 100=0.35%
  cpu          : usr=0.63%, sys=7.02%, ctx=2640, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182728: Sat Jul  4 02:13:26 2026
  write: IOPS=452, BW=1809KiB/s (1853kB/s)(10.0MiB/5660msec); 0 zone resets
    slat (usec): min=46, max=771, avg=187.76, stdev=99.73
    clat (usec): min=16972, max=59116, avg=35115.85, stdev=4500.82
     lat (usec): min=17248, max=59234, avg=35304.77, stdev=4499.73
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27657], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[38536], 90.00th=[40109], 95.00th=[42206],
     | 99.00th=[46400], 99.50th=[49021], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1656, max= 1968, per=25.18%, avg=1807.45, stdev=94.29, samples=11
   iops        : min=  414, max=  492, avg=451.82, stdev=23.52, samples=11
  lat (msec)   : 20=0.35%, 50=99.26%, 100=0.39%
  cpu          : usr=0.46%, sys=6.63%, ctx=2569, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182729: Sat Jul  4 02:13:26 2026
  write: IOPS=449, BW=1797KiB/s (1840kB/s)(10.0MiB/5698msec); 0 zone resets
    slat (usec): min=49, max=870, avg=202.75, stdev=112.09
    clat (usec): min=17883, max=49306, avg=35006.46, stdev=4099.64
     lat (usec): min=18201, max=49561, avg=35210.47, stdev=4108.11
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28181], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[40109], 95.00th=[41681],
     | 99.00th=[44827], 99.50th=[45876], 99.90th=[49021], 99.95th=[49021],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1648, max= 1992, per=25.01%, avg=1795.36, stdev=97.44, samples=11
   iops        : min=  412, max=  498, avg=448.82, stdev=24.37, samples=11
  lat (msec)   : 20=0.08%, 50=99.92%
  cpu          : usr=0.39%, sys=7.41%, ctx=2681, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7178KiB/s (7351kB/s), 1795KiB/s-1809KiB/s (1838kB/s-1853kB/s), io=40.0MiB (41.9MB), run=5660-5706msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182739: Sat Jul  4 02:13:36 2026
  read: IOPS=204, BW=817KiB/s (837kB/s)(5048KiB/6179msec)
    slat (usec): min=65, max=11876, avg=2696.06, stdev=923.83
    clat (usec): min=20457, max=59328, avg=36970.08, stdev=5713.23
     lat (usec): min=21339, max=62500, avg=39668.20, stdev=5970.97
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27919], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41157], 90.00th=[44303], 95.00th=[46924],
     | 99.00th=[51643], 99.50th=[52691], 99.90th=[56361], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  656, max= 1016, per=25.25%, avg=814.92, stdev=90.41, samples=12
   iops        : min=  164, max=  254, avg=203.67, stdev=22.58, samples=12
  write: IOPS=210, BW=840KiB/s (860kB/s)(5192KiB/6179msec); 0 zone resets
    slat (usec): min=54, max=738, avg=182.05, stdev=88.04
    clat (usec): min=11004, max=60261, avg=37268.21, stdev=5495.87
     lat (usec): min=11115, max=60439, avg=37451.50, stdev=5503.18
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29492], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41157], 90.00th=[44303], 95.00th=[46924],
     | 99.00th=[52167], 99.50th=[52691], 99.90th=[56886], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  712, max= 1000, per=25.05%, avg=841.83, stdev=87.38, samples=12
   iops        : min=  178, max=  250, avg=210.42, stdev=21.87, samples=12
  lat (msec)   : 20=0.20%, 50=97.73%, 100=2.07%
  cpu          : usr=0.50%, sys=6.91%, ctx=4684, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182740: Sat Jul  4 02:13:36 2026
  read: IOPS=200, BW=801KiB/s (820kB/s)(4976KiB/6215msec)
    slat (usec): min=1034, max=6490, avg=2692.02, stdev=841.38
    clat (usec): min=5837, max=59366, avg=36881.40, stdev=6075.75
     lat (usec): min=8496, max=63083, avg=39575.29, stdev=6308.47
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27657], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[44303], 95.00th=[47449],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[57410], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  678, max=  984, per=24.72%, avg=797.83, stdev=83.41, samples=12
   iops        : min=  169, max=  246, avg=199.42, stdev=20.92, samples=12
  write: IOPS=211, BW=847KiB/s (867kB/s)(5264KiB/6215msec); 0 zone resets
    slat (usec): min=42, max=744, avg=190.11, stdev=101.04
    clat (usec): min=8996, max=58822, avg=37355.77, stdev=5421.15
     lat (usec): min=9142, max=59249, avg=37547.03, stdev=5419.11
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30540], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[44827], 95.00th=[46924],
     | 99.00th=[50594], 99.50th=[52167], 99.90th=[57934], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  648, max= 1008, per=25.11%, avg=844.00, stdev=97.53, samples=12
   iops        : min=  162, max=  252, avg=211.00, stdev=24.38, samples=12
  lat (msec)   : 10=0.08%, 20=0.16%, 50=97.81%, 100=1.95%
  cpu          : usr=0.56%, sys=6.89%, ctx=4654, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182741: Sat Jul  4 02:13:36 2026
  read: IOPS=200, BW=803KiB/s (823kB/s)(4984KiB/6203msec)
    slat (usec): min=1051, max=11958, avg=2700.69, stdev=881.46
    clat (usec): min=6149, max=59869, avg=36907.87, stdev=5917.03
     lat (usec): min=8772, max=62434, avg=39610.57, stdev=6121.18
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27395], 10.00th=[29230], 20.00th=[31851],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[44827], 95.00th=[46924],
     | 99.00th=[51119], 99.50th=[52167], 99.90th=[56361], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  584, max=  968, per=24.91%, avg=804.17, stdev=103.18, samples=12
   iops        : min=  146, max=  242, avg=201.00, stdev=25.83, samples=12
  write: IOPS=211, BW=847KiB/s (868kB/s)(5256KiB/6203msec); 0 zone resets
    slat (usec): min=56, max=853, avg=188.24, stdev=99.84
    clat (usec): min=23260, max=58590, avg=37281.93, stdev=5563.34
     lat (usec): min=23402, max=58783, avg=37471.35, stdev=5564.46
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[44827], 95.00th=[47449],
     | 99.00th=[51643], 99.50th=[53740], 99.90th=[55837], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  736, max= 1016, per=25.16%, avg=845.58, stdev=81.88, samples=12
   iops        : min=  184, max=  254, avg=211.33, stdev=20.53, samples=12
  lat (msec)   : 10=0.04%, 50=97.93%, 100=2.03%
  cpu          : usr=0.52%, sys=6.84%, ctx=4745, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182742: Sat Jul  4 02:13:36 2026
  read: IOPS=203, BW=814KiB/s (834kB/s)(5060KiB/6214msec)
    slat (usec): min=1132, max=12351, avg=2710.78, stdev=900.70
    clat (usec): min=9686, max=59915, avg=36997.88, stdev=6059.61
     lat (usec): min=11717, max=63762, avg=39710.57, stdev=6296.47
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[28181], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[44827], 95.00th=[47973],
     | 99.00th=[52691], 99.50th=[54264], 99.90th=[58983], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  664, max=  928, per=25.20%, avg=813.33, stdev=74.68, samples=12
   iops        : min=  166, max=  232, avg=203.33, stdev=18.67, samples=12
  write: IOPS=208, BW=834KiB/s (854kB/s)(5180KiB/6214msec); 0 zone resets
    slat (usec): min=49, max=923, avg=185.21, stdev=99.38
    clat (usec): min=23475, max=59316, avg=37363.87, stdev=5625.43
     lat (usec): min=23566, max=59685, avg=37550.31, stdev=5623.31
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29230], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[51643], 99.50th=[54264], 99.90th=[56886], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  640, max= 1008, per=24.73%, avg=831.08, stdev=93.99, samples=12
   iops        : min=  160, max=  252, avg=207.75, stdev=23.51, samples=12
  lat (msec)   : 10=0.04%, 20=0.04%, 50=97.42%, 100=2.50%
  cpu          : usr=0.42%, sys=6.92%, ctx=4727, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3229KiB/s (3306kB/s), 801KiB/s-817KiB/s (820kB/s-837kB/s), io=19.6MiB (20.5MB), run=6179-6215msec
  WRITE: bw=3362KiB/s (3442kB/s), 834KiB/s-847KiB/s (854kB/s-868kB/s), io=20.4MiB (21.4MB), run=6179-6215msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182747: Sat Jul  4 02:13:39 2026
  read: IOPS=9, BW=396MiB/s (415MB/s)(40.0MiB/101msec)
    slat (nsec): min=10119k, max=10119k, avg=10119145.00, stdev= 0.00
    clat (nsec): min=85732k, max=85732k, avg=85732373.00, stdev= 0.00
     lat (nsec): min=95855k, max=95855k, avg=95855381.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[85459],  5.00th=[85459], 10.00th=[85459], 20.00th=[85459],
     | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459],
     | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459],
     | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.00%, ctx=9, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=396MiB/s (415MB/s), 396MiB/s-396MiB/s (415MB/s-415MB/s), io=40.0MiB (41.9MB), run=101-101msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout