== sanity test 398c: run fio to test AIO ================= 17:04:06 (1763157846)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.165776 s, 253 MB/s
osc.lustre-OST0000-osc-ffff985310827000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183538: Fri Nov 14 17:04:18 2025
  write: IOPS=315, BW=1261KiB/s (1292kB/s)(10.0MiB/8118msec); 0 zone resets
    slat (usec): min=22, max=1282, avg=119.13, stdev=75.78
    clat (usec): min=18938, max=86748, avg=50469.99, stdev=11683.98
     lat (usec): min=19076, max=86921, avg=50590.38, stdev=11694.13
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[32637], 10.00th=[36963], 20.00th=[41157],
     | 30.00th=[43779], 40.00th=[46924], 50.00th=[49546], 60.00th=[52691],
     | 70.00th=[55837], 80.00th=[60031], 90.00th=[65274], 95.00th=[72877],
     | 99.00th=[81265], 99.50th=[82314], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  976, max= 1659, per=24.65%, avg=1239.13, stdev=208.95, samples=15
   iops        : min=  244, max=  414, avg=309.73, stdev=52.13, samples=15
  lat (msec)   : 20=0.62%, 50=50.66%, 100=48.71%
  cpu          : usr=0.43%, sys=3.38%, ctx=2467, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183539: Fri Nov 14 17:04:18 2025
  write: IOPS=314, BW=1257KiB/s (1287kB/s)(10.0MiB/8149msec); 0 zone resets
    slat (usec): min=26, max=987, avg=97.03, stdev=61.63
    clat (usec): min=16864, max=86539, avg=50414.13, stdev=11706.60
     lat (usec): min=16920, max=86609, avg=50512.26, stdev=11713.46
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[32637], 10.00th=[36439], 20.00th=[41157],
     | 30.00th=[43779], 40.00th=[46924], 50.00th=[49546], 60.00th=[52691],
     | 70.00th=[55837], 80.00th=[60031], 90.00th=[65799], 95.00th=[70779],
     | 99.00th=[80217], 99.50th=[83362], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  936, max= 1543, per=24.84%, avg=1248.44, stdev=200.77, samples=16
   iops        : min=  234, max=  385, avg=312.06, stdev=50.12, samples=16
  lat (msec)   : 20=0.74%, 50=51.60%, 100=47.66%
  cpu          : usr=0.60%, sys=2.43%, ctx=2470, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183540: Fri Nov 14 17:04:18 2025
  write: IOPS=315, BW=1261KiB/s (1292kB/s)(10.0MiB/8119msec); 0 zone resets
    slat (usec): min=30, max=1213, avg=113.58, stdev=68.64
    clat (usec): min=18884, max=89921, avg=50567.28, stdev=11601.46
     lat (usec): min=18943, max=90050, avg=50682.15, stdev=11612.92
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[32900], 10.00th=[36963], 20.00th=[41157],
     | 30.00th=[43779], 40.00th=[46924], 50.00th=[49546], 60.00th=[52167],
     | 70.00th=[55837], 80.00th=[60031], 90.00th=[65799], 95.00th=[71828],
     | 99.00th=[81265], 99.50th=[83362], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min=  992, max= 1659, per=24.63%, avg=1238.07, stdev=205.15, samples=15
   iops        : min=  248, max=  414, avg=309.47, stdev=51.18, samples=15
  lat (msec)   : 20=0.62%, 50=51.17%, 100=48.20%
  cpu          : usr=0.54%, sys=3.01%, ctx=2413, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183541: Fri Nov 14 17:04:18 2025
  write: IOPS=314, BW=1257KiB/s (1287kB/s)(10.0MiB/8146msec); 0 zone resets
    slat (usec): min=26, max=1237, avg=117.76, stdev=70.33
    clat (usec): min=13960, max=90186, avg=50407.34, stdev=11760.60
     lat (usec): min=14052, max=90247, avg=50526.42, stdev=11761.60
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[32637], 10.00th=[36439], 20.00th=[41157],
     | 30.00th=[43779], 40.00th=[46924], 50.00th=[49546], 60.00th=[52691],
     | 70.00th=[55837], 80.00th=[60031], 90.00th=[65799], 95.00th=[71828],
     | 99.00th=[81265], 99.50th=[82314], 99.90th=[86508], 99.95th=[89654],
     | 99.99th=[90702]
   bw (  KiB/s): min=  984, max= 1600, per=24.85%, avg=1248.75, stdev=197.17, samples=16
   iops        : min=  246, max=  400, avg=312.19, stdev=49.29, samples=16
  lat (msec)   : 20=0.51%, 50=50.98%, 100=48.52%
  cpu          : usr=0.53%, sys=3.43%, ctx=2580, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5026KiB/s (5147kB/s), 1257KiB/s-1261KiB/s (1287kB/s-1292kB/s), io=40.0MiB (41.9MB), run=8118-8149msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183552: Fri Nov 14 17:04:28 2025
  read: IOPS=173, BW=695KiB/s (711kB/s)(5048KiB/7266msec)
    slat (usec): min=33, max=10382, avg=3166.39, stdev=1069.85
    clat (usec): min=7610, max=85166, avg=43195.28, stdev=8963.85
     lat (usec): min=7923, max=87966, avg=46364.94, stdev=9316.52
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[30016], 10.00th=[32900], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[42730], 60.00th=[44827],
     | 70.00th=[47449], 80.00th=[50594], 90.00th=[53740], 95.00th=[57934],
     | 99.00th=[66847], 99.50th=[68682], 99.90th=[83362], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min=  528, max=  848, per=25.32%, avg=693.36, stdev=86.27, samples=14
   iops        : min=  132, max=  212, avg=173.29, stdev=21.57, samples=14
  write: IOPS=178, BW=715KiB/s (732kB/s)(5192KiB/7266msec); 0 zone resets
    slat (usec): min=33, max=2282, avg=108.72, stdev=84.58
    clat (usec): min=26612, max=83450, avg=44192.71, stdev=8180.89
     lat (usec): min=26666, max=83529, avg=44302.85, stdev=8183.04
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[32113], 10.00th=[33817], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[41681], 50.00th=[43779], 60.00th=[45876],
     | 70.00th=[47973], 80.00th=[50070], 90.00th=[54789], 95.00th=[58459],
     | 99.00th=[67634], 99.50th=[70779], 99.90th=[80217], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  600, max=  880, per=25.10%, avg=715.36, stdev=73.93, samples=14
   iops        : min=  150, max=  220, avg=178.79, stdev=18.48, samples=14
  lat (msec)   : 10=0.12%, 20=0.31%, 50=78.16%, 100=21.41%
  cpu          : usr=0.47%, sys=4.80%, ctx=4520, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183553: Fri Nov 14 17:04:28 2025
  read: IOPS=169, BW=680KiB/s (696kB/s)(4976KiB/7320msec)
    slat (usec): min=1163, max=9507, avg=3139.13, stdev=1059.77
    clat (usec): min=14803, max=72493, avg=43187.53, stdev=8286.71
     lat (usec): min=18036, max=76083, avg=46329.71, stdev=8575.88
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30802], 10.00th=[32900], 20.00th=[35914],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[42730], 60.00th=[44827],
     | 70.00th=[46924], 80.00th=[50070], 90.00th=[53740], 95.00th=[57934],
     | 99.00th=[65274], 99.50th=[66847], 99.90th=[71828], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  488, max=  848, per=24.66%, avg=675.29, stdev=104.87, samples=14
   iops        : min=  122, max=  212, avg=168.79, stdev=26.21, samples=14
  write: IOPS=179, BW=719KiB/s (736kB/s)(5264KiB/7320msec); 0 zone resets
    slat (usec): min=35, max=952, avg=109.47, stdev=71.29
    clat (usec): min=23975, max=72379, avg=44637.36, stdev=8115.28
     lat (usec): min=24065, max=72558, avg=44748.16, stdev=8121.46
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[32637], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44303], 60.00th=[46400],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[55313], 95.00th=[58983],
     | 99.00th=[65799], 99.50th=[67634], 99.90th=[71828], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  640, max=  832, per=25.21%, avg=718.36, stdev=60.79, samples=14
   iops        : min=  160, max=  208, avg=179.50, stdev=15.26, samples=14
  lat (msec)   : 20=0.04%, 50=76.80%, 100=23.16%
  cpu          : usr=0.57%, sys=4.67%, ctx=4435, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183554: Fri Nov 14 17:04:28 2025
  read: IOPS=170, BW=680KiB/s (697kB/s)(4984KiB/7325msec)
    slat (usec): min=1334, max=12452, avg=3175.64, stdev=1061.00
    clat (usec): min=7242, max=74485, avg=43378.46, stdev=8172.62
     lat (usec): min=10320, max=80488, avg=46557.29, stdev=8513.20
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[31065], 10.00th=[33817], 20.00th=[36439],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[42730], 60.00th=[44827],
     | 70.00th=[47449], 80.00th=[50070], 90.00th=[54264], 95.00th=[57410],
     | 99.00th=[65799], 99.50th=[66323], 99.90th=[71828], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  560, max=  776, per=24.86%, avg=680.71, stdev=67.93, samples=14
   iops        : min=  140, max=  194, avg=170.14, stdev=16.97, samples=14
  write: IOPS=179, BW=718KiB/s (735kB/s)(5256KiB/7325msec); 0 zone resets
    slat (usec): min=35, max=1010, avg=109.38, stdev=73.31
    clat (usec): min=22527, max=79266, avg=44068.59, stdev=8342.93
     lat (usec): min=22603, max=79349, avg=44178.96, stdev=8347.69
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[32113], 10.00th=[33817], 20.00th=[36963],
     | 30.00th=[39060], 40.00th=[41157], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[50594], 90.00th=[54789], 95.00th=[59507],
     | 99.00th=[66323], 99.50th=[69731], 99.90th=[76022], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  552, max=  920, per=24.94%, avg=710.93, stdev=103.13, samples=14
   iops        : min=  138, max=  230, avg=177.64, stdev=25.81, samples=14
  lat (msec)   : 10=0.04%, 50=79.57%, 100=20.39%
  cpu          : usr=0.60%, sys=4.55%, ctx=4522, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183555: Fri Nov 14 17:04:28 2025
  read: IOPS=172, BW=691KiB/s (707kB/s)(5060KiB/7328msec)
    slat (usec): min=1077, max=9005, avg=3146.14, stdev=1035.80
    clat (usec): min=18471, max=82467, avg=43458.41, stdev=8237.99
     lat (usec): min=20705, max=90526, avg=46608.26, stdev=8562.57
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[31327], 10.00th=[33424], 20.00th=[36439],
     | 30.00th=[39060], 40.00th=[40633], 50.00th=[42730], 60.00th=[44827],
     | 70.00th=[46924], 80.00th=[50070], 90.00th=[54789], 95.00th=[57934],
     | 99.00th=[64750], 99.50th=[68682], 99.90th=[72877], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min=  552, max=  872, per=25.15%, avg=688.57, stdev=86.08, samples=14
   iops        : min=  138, max=  218, avg=172.07, stdev=21.61, samples=14
  write: IOPS=176, BW=707KiB/s (724kB/s)(5180KiB/7328msec); 0 zone resets
    slat (usec): min=33, max=2471, avg=108.45, stdev=88.56
    clat (usec): min=20713, max=74853, avg=44158.30, stdev=8084.93
     lat (usec): min=20835, max=74932, avg=44267.85, stdev=8085.74
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[32375], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[39584], 40.00th=[41681], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[50594], 90.00th=[54789], 95.00th=[58459],
     | 99.00th=[65799], 99.50th=[68682], 99.90th=[72877], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  576, max=  880, per=24.64%, avg=702.29, stdev=87.16, samples=14
   iops        : min=  144, max=  220, avg=175.50, stdev=21.78, samples=14
  lat (msec)   : 20=0.04%, 50=78.63%, 100=21.33%
  cpu          : usr=0.67%, sys=4.63%, ctx=4552, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2739KiB/s (2804kB/s), 680KiB/s-695KiB/s (696kB/s-711kB/s), io=19.6MiB (20.5MB), run=7266-7328msec
  WRITE: bw=2851KiB/s (2919kB/s), 707KiB/s-719KiB/s (724kB/s-736kB/s), io=20.4MiB (21.4MB), run=7266-7328msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183559: Fri Nov 14 17:04:32 2025
  read: IOPS=8, BW=328MiB/s (344MB/s)(40.0MiB/122msec)
    slat (nsec): min=11415k, max=11415k, avg=11415023.00, stdev= 0.00
    clat (nsec): min=103973k, max=103973k, avg=103972516.00, stdev= 0.00
     lat (nsec): min=115391k, max=115391k, avg=115391488.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  105],  5.00th=[  105], 10.00th=[  105], 20.00th=[  105],
     | 30.00th=[  105], 40.00th=[  105], 50.00th=[  105], 60.00th=[  105],
     | 70.00th=[  105], 80.00th=[  105], 90.00th=[  105], 95.00th=[  105],
     | 99.00th=[  105], 99.50th=[  105], 99.90th=[  105], 99.95th=[  105],
     | 99.99th=[  105]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=9.92%, ctx=3, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=40.0MiB (41.9MB), run=122-122msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout