== sanity test 398c: run fio to test AIO ================= 13:01:06 (1752512466)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.233846 s, 179 MB/s
osc.lustre-OST0000-osc-ffff891d44d2d000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190804: Mon Jul 14 13:01:23 2025
  write: IOPS=220, BW=883KiB/s (904kB/s)(10.0MiB/11602msec); 0 zone resets
    slat (usec): min=23, max=4550, avg=108.67, stdev=203.76
    clat (msec): min=16, max=168, avg=71.61, stdev=31.29
     lat (msec): min=16, max=168, avg=71.72, stdev=31.30
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   30], 10.00th=[   33], 20.00th=[   38],
     | 30.00th=[   48], 40.00th=[   62], 50.00th=[   70], 60.00th=[   82],
     | 70.00th=[   92], 80.00th=[  101], 90.00th=[  111], 95.00th=[  123],
     | 99.00th=[  157], 99.50th=[  165], 99.90th=[  169], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  510, max= 1888, per=24.18%, avg=853.59, stdev=400.14, samples=22
   iops        : min=  127, max=  472, avg=213.23, stdev=100.13, samples=22
  lat (msec)   : 20=0.08%, 50=32.50%, 100=46.72%, 250=20.70%
  cpu          : usr=0.25%, sys=2.22%, ctx=1279, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190805: Mon Jul 14 13:01:23 2025
  write: IOPS=221, BW=885KiB/s (906kB/s)(10.0MiB/11573msec); 0 zone resets
    slat (usec): min=21, max=3184, avg=114.72, stdev=197.15
    clat (msec): min=23, max=169, avg=72.12, stdev=31.27
     lat (msec): min=23, max=169, avg=72.24, stdev=31.29
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   30], 10.00th=[   33], 20.00th=[   40],
     | 30.00th=[   48], 40.00th=[   62], 50.00th=[   72], 60.00th=[   82],
     | 70.00th=[   91], 80.00th=[  100], 90.00th=[  112], 95.00th=[  125],
     | 99.00th=[  148], 99.50th=[  165], 99.90th=[  169], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  510, max= 1928, per=24.31%, avg=858.32, stdev=394.60, samples=22
   iops        : min=  127, max=  482, avg=214.41, stdev=98.74, samples=22
  lat (msec)   : 50=32.54%, 100=47.54%, 250=19.92%
  cpu          : usr=0.29%, sys=2.05%, ctx=1352, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190806: Mon Jul 14 13:01:23 2025
  write: IOPS=220, BW=883KiB/s (904kB/s)(10.0MiB/11600msec); 0 zone resets
    slat (usec): min=23, max=7571, avg=118.49, stdev=264.25
    clat (msec): min=8, max=168, avg=71.41, stdev=31.56
     lat (msec): min=8, max=168, avg=71.53, stdev=31.56
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   30], 10.00th=[   33], 20.00th=[   39],
     | 30.00th=[   47], 40.00th=[   62], 50.00th=[   70], 60.00th=[   81],
     | 70.00th=[   89], 80.00th=[  101], 90.00th=[  112], 95.00th=[  127],
     | 99.00th=[  159], 99.50th=[  161], 99.90th=[  165], 99.95th=[  169],
     | 99.99th=[  169]
   bw (  KiB/s): min=  510, max= 1944, per=24.17%, avg=853.36, stdev=406.93, samples=22
   iops        : min=  127, max=  486, avg=213.18, stdev=101.81, samples=22
  lat (msec)   : 10=0.08%, 20=0.12%, 50=34.06%, 100=45.86%, 250=19.88%
  cpu          : usr=0.35%, sys=2.14%, ctx=1267, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190807: Mon Jul 14 13:01:23 2025
  write: IOPS=220, BW=883KiB/s (904kB/s)(10.0MiB/11599msec); 0 zone resets
    slat (usec): min=22, max=5649, avg=104.19, stdev=216.06
    clat (msec): min=4, max=165, avg=71.56, stdev=31.10
     lat (msec): min=4, max=165, avg=71.67, stdev=31.11
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   31], 10.00th=[   33], 20.00th=[   38],
     | 30.00th=[   47], 40.00th=[   62], 50.00th=[   70], 60.00th=[   82],
     | 70.00th=[   92], 80.00th=[  101], 90.00th=[  111], 95.00th=[  121],
     | 99.00th=[  157], 99.50th=[  165], 99.90th=[  165], 99.95th=[  165],
     | 99.99th=[  165]
   bw (  KiB/s): min=  518, max= 1896, per=24.19%, avg=853.77, stdev=401.05, samples=22
   iops        : min=  129, max=  474, avg=213.27, stdev=100.35, samples=22
  lat (msec)   : 10=0.04%, 50=33.16%, 100=47.11%, 250=19.69%
  cpu          : usr=0.39%, sys=1.80%, ctx=1074, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=3530KiB/s (3615kB/s), 883KiB/s-885KiB/s (904kB/s-906kB/s), io=40.0MiB (41.9MB), run=11573-11602msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190818: Mon Jul 14 13:01:32 2025
  read: IOPS=218, BW=874KiB/s (895kB/s)(5048KiB/5775msec)
    slat (usec): min=925, max=8153, avg=2544.62, stdev=814.05
    clat (usec): min=4098, max=57442, avg=33972.35, stdev=5544.50
     lat (usec): min=7433, max=60298, avg=36519.03, stdev=5706.29
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25822], 10.00th=[27657], 20.00th=[29492],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[41157], 95.00th=[43779],
     | 99.00th=[49021], 99.50th=[51119], 99.90th=[55837], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  774, max=  968, per=25.04%, avg=869.82, stdev=68.98, samples=11
   iops        : min=  193, max=  242, avg=217.36, stdev=17.36, samples=11
  write: IOPS=224, BW=899KiB/s (921kB/s)(5192KiB/5775msec); 0 zone resets
    slat (usec): min=24, max=3381, avg=84.08, stdev=118.16
    clat (usec): min=13267, max=63927, avg=34994.31, stdev=5220.88
     lat (usec): min=13340, max=64053, avg=35079.88, stdev=5230.33
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28181], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[41157], 95.00th=[44827],
     | 99.00th=[52167], 99.50th=[56361], 99.90th=[62129], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  744, max= 1085, per=24.80%, avg=897.18, stdev=98.66, samples=11
   iops        : min=  186, max=  271, avg=224.18, stdev=24.67, samples=11
  lat (msec)   : 10=0.04%, 20=0.23%, 50=98.79%, 100=0.94%
  cpu          : usr=0.87%, sys=4.69%, ctx=2200, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190819: Mon Jul 14 13:01:32 2025
  read: IOPS=217, BW=869KiB/s (890kB/s)(4976KiB/5727msec)
    slat (usec): min=47, max=8029, avg=2534.25, stdev=840.08
    clat (usec): min=3229, max=74660, avg=34115.66, stdev=6227.96
     lat (usec): min=3459, max=76724, avg=36651.94, stdev=6388.32
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[26084], 10.00th=[27657], 20.00th=[29492],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33424], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[41157], 95.00th=[44827],
     | 99.00th=[53740], 99.50th=[55837], 99.90th=[71828], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  782, max=  960, per=24.97%, avg=867.45, stdev=61.79, samples=11
   iops        : min=  195, max=  240, avg=216.73, stdev=15.49, samples=11
  write: IOPS=229, BW=919KiB/s (941kB/s)(5264KiB/5727msec); 0 zone resets
    slat (usec): min=26, max=596, avg=81.81, stdev=41.29
    clat (usec): min=4900, max=62575, avg=34750.83, stdev=5592.77
     lat (usec): min=5028, max=62671, avg=34833.59, stdev=5594.93
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27657], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[38011], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[53216], 99.50th=[58983], 99.90th=[62129], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  824, max= 1064, per=25.50%, avg=922.36, stdev=78.64, samples=11
   iops        : min=  206, max=  266, avg=230.45, stdev=19.62, samples=11
  lat (msec)   : 4=0.08%, 10=0.12%, 20=0.51%, 50=97.85%, 100=1.45%
  cpu          : usr=0.51%, sys=5.15%, ctx=2222, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190820: Mon Jul 14 13:01:32 2025
  read: IOPS=215, BW=863KiB/s (884kB/s)(4984KiB/5772msec)
    slat (usec): min=1052, max=7240, avg=2568.55, stdev=829.92
    clat (usec): min=3708, max=60114, avg=34387.53, stdev=5862.81
     lat (usec): min=5926, max=62226, avg=36958.23, stdev=6035.46
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26084], 10.00th=[27657], 20.00th=[29754],
     | 30.00th=[31327], 40.00th=[32637], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[50594], 99.50th=[54264], 99.90th=[58459], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  736, max=  984, per=24.82%, avg=862.09, stdev=70.51, samples=11
   iops        : min=  184, max=  246, avg=215.36, stdev=17.55, samples=11
  write: IOPS=227, BW=911KiB/s (932kB/s)(5256KiB/5772msec); 0 zone resets
    slat (usec): min=23, max=610, avg=80.98, stdev=41.85
    clat (usec): min=5746, max=61826, avg=34753.83, stdev=5543.96
     lat (usec): min=5812, max=61935, avg=34835.82, stdev=5544.60
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27395], 10.00th=[28967], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[41681], 95.00th=[44303],
     | 99.00th=[52691], 99.50th=[55837], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  808, max= 1016, per=25.03%, avg=905.45, stdev=71.40, samples=11
   iops        : min=  202, max=  254, avg=226.27, stdev=17.87, samples=11
  lat (msec)   : 4=0.04%, 10=0.23%, 20=0.04%, 50=98.28%, 100=1.41%
  cpu          : usr=0.62%, sys=4.96%, ctx=2173, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190821: Mon Jul 14 13:01:32 2025
  read: IOPS=219, BW=878KiB/s (899kB/s)(5060KiB/5761msec)
    slat (usec): min=987, max=6252, avg=2535.77, stdev=784.50
    clat (usec): min=5965, max=64457, avg=33984.08, stdev=5457.99
     lat (usec): min=8506, max=67068, avg=36521.88, stdev=5620.01
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26084], 10.00th=[27657], 20.00th=[29754],
     | 30.00th=[31065], 40.00th=[32375], 50.00th=[33424], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[38011], 90.00th=[40633], 95.00th=[43779],
     | 99.00th=[50070], 99.50th=[53216], 99.90th=[57934], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  784, max=  976, per=25.30%, avg=879.00, stdev=60.02, samples=11
   iops        : min=  196, max=  244, avg=219.64, stdev=15.05, samples=11
  write: IOPS=224, BW=899KiB/s (921kB/s)(5180KiB/5761msec); 0 zone resets
    slat (usec): min=23, max=1045, avg=81.66, stdev=47.09
    clat (usec): min=20772, max=68499, avg=34924.98, stdev=5481.71
     lat (usec): min=20969, max=68546, avg=35007.71, stdev=5483.16
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27657], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[36963], 80.00th=[38536], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[52691], 99.50th=[61604], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  760, max= 1008, per=24.83%, avg=898.00, stdev=80.93, samples=11
   iops        : min=  190, max=  252, avg=224.36, stdev=20.24, samples=11
  lat (msec)   : 10=0.04%, 20=0.04%, 50=98.67%, 100=1.25%
  cpu          : usr=0.69%, sys=4.95%, ctx=2235, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3475KiB/s (3558kB/s), 863KiB/s-878KiB/s (884kB/s-899kB/s), io=19.6MiB (20.5MB), run=5727-5775msec
  WRITE: bw=3618KiB/s (3704kB/s), 899KiB/s-919KiB/s (921kB/s-941kB/s), io=20.4MiB (21.4MB), run=5727-5775msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190825: Mon Jul 14 13:01:36 2025
  read: IOPS=9, BW=381MiB/s (399MB/s)(40.0MiB/105msec)
    slat (nsec): min=13340k, max=13340k, avg=13340408.00, stdev= 0.00
    clat (nsec): min=86659k, max=86659k, avg=86658899.00, stdev= 0.00
     lat (nsec): min=100004k, max=100004k, avg=100004077.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[86508],  5.00th=[86508], 10.00th=[86508], 20.00th=[86508],
     | 30.00th=[86508], 40.00th=[86508], 50.00th=[86508], 60.00th=[86508],
     | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508],
     | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=12.50%, ctx=5, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=381MiB/s (399MB/s), 381MiB/s-381MiB/s (399MB/s-399MB/s), io=40.0MiB (41.9MB), run=105-105msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout