== sanity test 398c: run fio to test AIO ================= 09:24:06 (1774531446)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.13591 s, 309 MB/s
osc.lustre-OST0000-osc-ffff8c1229986800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=148384: Thu Mar 26 09:24:14 2026
  write: IOPS=655, BW=2624KiB/s (2687kB/s)(10.0MiB/3903msec); 0 zone resets
    slat (usec): min=24, max=1271, avg=84.70, stdev=57.81
    clat (usec): min=2852, max=47099, avg=24238.31, stdev=4663.96
     lat (usec): min=2901, max=47163, avg=24324.20, stdev=4667.65
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[18220], 10.00th=[19268], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23462], 60.00th=[24773],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[29492], 95.00th=[33424],
     | 99.00th=[40109], 99.50th=[44827], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 2279, max= 2904, per=24.93%, avg=2615.86, stdev=245.98, samples=7
   iops        : min=  569, max=  726, avg=653.86, stdev=61.67, samples=7
  lat (msec)   : 4=0.08%, 10=0.04%, 20=14.53%, 50=85.35%
  cpu          : usr=0.85%, sys=5.05%, ctx=1422, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=148385: Thu Mar 26 09:24:14 2026
  write: IOPS=655, BW=2624KiB/s (2687kB/s)(10.0MiB/3903msec); 0 zone resets
    slat (usec): min=20, max=1950, avg=85.38, stdev=64.13
    clat (usec): min=3602, max=49053, avg=24243.23, stdev=4768.93
     lat (usec): min=3736, max=49137, avg=24329.74, stdev=4769.67
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[18220], 10.00th=[19268], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23462], 60.00th=[24773],
     | 70.00th=[25822], 80.00th=[27657], 90.00th=[29754], 95.00th=[33817],
     | 99.00th=[40109], 99.50th=[44827], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[49021]
   bw (  KiB/s): min= 2279, max= 2920, per=24.94%, avg=2617.00, stdev=252.55, samples=7
   iops        : min=  569, max=  730, avg=654.14, stdev=63.30, samples=7
  lat (msec)   : 4=0.04%, 10=0.12%, 20=15.12%, 50=84.73%
  cpu          : usr=0.90%, sys=5.10%, ctx=1457, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=148386: Thu Mar 26 09:24:14 2026
  write: IOPS=657, BW=2630KiB/s (2693kB/s)(10.0MiB/3894msec); 0 zone resets
    slat (usec): min=21, max=752, avg=84.51, stdev=49.26
    clat (usec): min=7228, max=46880, avg=24218.90, stdev=4888.98
     lat (usec): min=7357, max=46960, avg=24304.37, stdev=4888.83
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[18220], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21627], 40.00th=[22414], 50.00th=[23462], 60.00th=[24511],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[29754], 95.00th=[33424],
     | 99.00th=[41681], 99.50th=[44303], 99.90th=[45876], 99.95th=[46400],
     | 99.99th=[46924]
   bw (  KiB/s): min= 2298, max= 2944, per=25.04%, avg=2627.71, stdev=265.11, samples=7
   iops        : min=  574, max=  736, avg=656.86, stdev=66.38, samples=7
  lat (msec)   : 10=0.20%, 20=15.82%, 50=83.98%
  cpu          : usr=0.54%, sys=5.47%, ctx=1446, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=148387: Thu Mar 26 09:24:14 2026
  write: IOPS=655, BW=2624KiB/s (2687kB/s)(10.0MiB/3903msec); 0 zone resets
    slat (usec): min=23, max=1824, avg=84.51, stdev=69.55
    clat (usec): min=10958, max=46560, avg=24244.50, stdev=4652.26
     lat (usec): min=11027, max=46790, avg=24329.99, stdev=4653.07
    clat percentiles (usec):
     |  1.00th=[16581],  5.00th=[18220], 10.00th=[19268], 20.00th=[20579],
     | 30.00th=[21627], 40.00th=[22414], 50.00th=[23200], 60.00th=[24511],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[30016], 95.00th=[33817],
     | 99.00th=[40109], 99.50th=[44303], 99.90th=[46400], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min= 2227, max= 2888, per=24.89%, avg=2611.86, stdev=264.23, samples=7
   iops        : min=  556, max=  722, avg=652.86, stdev=66.24, samples=7
  lat (msec)   : 20=14.41%, 50=85.59%
  cpu          : usr=1.00%, sys=4.89%, ctx=1423, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.2MiB/s (10.7MB/s), 2624KiB/s-2630KiB/s (2687kB/s-2693kB/s), io=40.0MiB (41.9MB), run=3894-3903msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=148397: Thu Mar 26 09:24:21 2026
  read: IOPS=319, BW=1277KiB/s (1308kB/s)(5048KiB/3953msec)
    slat (usec): min=700, max=9475, avg=1469.14, stdev=718.61
    clat (usec): min=1726, max=72265, avg=23795.45, stdev=6471.98
     lat (usec): min=3197, max=73336, avg=25266.14, stdev=6736.00
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[16188], 10.00th=[17171], 20.00th=[19006],
     | 30.00th=[20055], 40.00th=[21365], 50.00th=[22414], 60.00th=[23987],
     | 70.00th=[25297], 80.00th=[27919], 90.00th=[32375], 95.00th=[36439],
     | 99.00th=[43779], 99.50th=[46924], 99.90th=[64750], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  990, max= 1512, per=25.68%, avg=1301.43, stdev=184.12, samples=7
   iops        : min=  247, max=  378, avg=325.14, stdev=46.28, samples=7
  write: IOPS=328, BW=1313KiB/s (1345kB/s)(5192KiB/3953msec); 0 zone resets
    slat (usec): min=22, max=803, avg=75.07, stdev=45.91
    clat (usec): min=9082, max=59870, avg=23838.16, stdev=6140.27
     lat (usec): min=9160, max=59993, avg=23914.11, stdev=6144.98
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[17171], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20317], 40.00th=[21365], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[25035], 80.00th=[27132], 90.00th=[32113], 95.00th=[38011],
     | 99.00th=[44827], 99.50th=[46924], 99.90th=[53740], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min= 1021, max= 1632, per=25.44%, avg=1342.57, stdev=208.29, samples=7
   iops        : min=  255, max=  408, avg=335.43, stdev=52.06, samples=7
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.08%, 20=27.81%, 50=71.80%
  lat (msec)   : 100=0.23%
  cpu          : usr=0.76%, sys=6.60%, ctx=2356, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148398: Thu Mar 26 09:24:21 2026
  read: IOPS=314, BW=1257KiB/s (1287kB/s)(4976KiB/3959msec)
    slat (usec): min=705, max=6915, avg=1437.04, stdev=662.23
    clat (usec): min=1915, max=60693, avg=23832.96, stdev=6395.21
     lat (usec): min=3380, max=62483, avg=25271.97, stdev=6623.26
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[16581], 10.00th=[17433], 20.00th=[19006],
     | 30.00th=[20317], 40.00th=[21365], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[25035], 80.00th=[27919], 90.00th=[32113], 95.00th=[36963],
     | 99.00th=[46924], 99.50th=[50070], 99.90th=[58459], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1053, max= 1584, per=25.14%, avg=1273.86, stdev=178.98, samples=7
   iops        : min=  263, max=  396, avg=318.29, stdev=44.92, samples=7
  write: IOPS=332, BW=1330KiB/s (1362kB/s)(5264KiB/3959msec); 0 zone resets
    slat (usec): min=24, max=1705, avg=76.47, stdev=64.76
    clat (usec): min=2891, max=60626, avg=23982.99, stdev=6214.64
     lat (usec): min=2944, max=60683, avg=24060.48, stdev=6221.97
    clat percentiles (usec):
     |  1.00th=[15008],  5.00th=[16909], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20579], 40.00th=[21627], 50.00th=[22414], 60.00th=[23725],
     | 70.00th=[25297], 80.00th=[27657], 90.00th=[32375], 95.00th=[36963],
     | 99.00th=[45876], 99.50th=[47973], 99.90th=[56886], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  982, max= 1600, per=25.94%, avg=1368.86, stdev=197.51, samples=7
   iops        : min=  245, max=  400, avg=342.00, stdev=49.51, samples=7
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.08%, 20=26.25%, 50=73.16%
  lat (msec)   : 100=0.43%
  cpu          : usr=0.91%, sys=6.42%, ctx=2308, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148399: Thu Mar 26 09:24:21 2026
  read: IOPS=318, BW=1273KiB/s (1304kB/s)(4984KiB/3915msec)
    slat (usec): min=30, max=6676, avg=1436.81, stdev=681.01
    clat (usec): min=4751, max=58088, avg=23213.47, stdev=5722.71
     lat (usec): min=5941, max=59134, avg=24652.02, stdev=5996.41
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[16188], 10.00th=[17171], 20.00th=[18744],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[22152], 60.00th=[23462],
     | 70.00th=[25035], 80.00th=[26870], 90.00th=[30802], 95.00th=[33817],
     | 99.00th=[41157], 99.50th=[45351], 99.90th=[51643], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  910, max= 1552, per=25.68%, avg=1301.71, stdev=228.87, samples=7
   iops        : min=  227, max=  388, avg=325.14, stdev=57.40, samples=7
  write: IOPS=335, BW=1343KiB/s (1375kB/s)(5256KiB/3915msec); 0 zone resets
    slat (usec): min=26, max=1002, avg=76.72, stdev=56.16
    clat (usec): min=6309, max=70336, avg=24115.84, stdev=6844.06
     lat (usec): min=6542, max=70423, avg=24193.80, stdev=6847.62
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[16712], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20579], 40.00th=[21365], 50.00th=[22414], 60.00th=[23725],
     | 70.00th=[25035], 80.00th=[27657], 90.00th=[32900], 95.00th=[38011],
     | 99.00th=[46924], 99.50th=[50070], 99.90th=[68682], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min= 1125, max= 1600, per=25.82%, avg=1362.43, stdev=150.99, samples=7
   iops        : min=  281, max=  400, avg=340.43, stdev=37.80, samples=7
  lat (msec)   : 10=0.31%, 20=27.70%, 50=71.56%, 100=0.43%
  cpu          : usr=0.69%, sys=6.75%, ctx=2376, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148400: Thu Mar 26 09:24:21 2026
  read: IOPS=319, BW=1278KiB/s (1309kB/s)(5060KiB/3959msec)
    slat (usec): min=686, max=5766, avg=1464.60, stdev=659.07
    clat (usec): min=12430, max=59469, avg=23668.75, stdev=6260.31
     lat (usec): min=13738, max=60953, avg=25135.09, stdev=6498.52
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[16057], 10.00th=[17433], 20.00th=[18744],
     | 30.00th=[20055], 40.00th=[21103], 50.00th=[22414], 60.00th=[23462],
     | 70.00th=[25297], 80.00th=[27657], 90.00th=[32637], 95.00th=[36439],
     | 99.00th=[45351], 99.50th=[47973], 99.90th=[51643], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  950, max= 1512, per=25.74%, avg=1304.43, stdev=187.13, samples=7
   iops        : min=  237, max=  378, avg=325.86, stdev=46.98, samples=7
  write: IOPS=327, BW=1308KiB/s (1340kB/s)(5180KiB/3959msec); 0 zone resets
    slat (usec): min=24, max=2953, avg=78.41, stdev=94.26
    clat (usec): min=10862, max=61056, avg=24064.77, stdev=6541.16
     lat (usec): min=10939, max=61165, avg=24144.16, stdev=6543.05
    clat percentiles (usec):
     |  1.00th=[14484],  5.00th=[16712], 10.00th=[17695], 20.00th=[19268],
     | 30.00th=[20317], 40.00th=[21365], 50.00th=[22414], 60.00th=[23725],
     | 70.00th=[25297], 80.00th=[27395], 90.00th=[32900], 95.00th=[36963],
     | 99.00th=[47449], 99.50th=[48497], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min= 1037, max= 1640, per=25.29%, avg=1334.71, stdev=209.15, samples=7
   iops        : min=  259, max=  410, avg=333.43, stdev=52.41, samples=7
  lat (msec)   : 20=28.40%, 50=71.33%, 100=0.27%
  cpu          : usr=0.63%, sys=6.70%, ctx=2317, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5069KiB/s (5191kB/s), 1257KiB/s-1278KiB/s (1287kB/s-1309kB/s), io=19.6MiB (20.5MB), run=3915-3959msec
  WRITE: bw=5277KiB/s (5404kB/s), 1308KiB/s-1343KiB/s (1340kB/s-1375kB/s), io=20.4MiB (21.4MB), run=3915-3959msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=148404: Thu Mar 26 09:24:25 2026
  read: IOPS=9, BW=367MiB/s (385MB/s)(40.0MiB/109msec)
    slat (nsec): min=14527k, max=14527k, avg=14527389.00, stdev= 0.00
    clat (nsec): min=90355k, max=90355k, avg=90355113.00, stdev= 0.00
     lat (nsec): min=104910k, max=104910k, avg=104910183.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.93%, sys=11.11%, ctx=4, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=367MiB/s (385MB/s), 367MiB/s-367MiB/s (385MB/s-385MB/s), io=40.0MiB (41.9MB), run=109-109msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck