== sanity test 398c: run fio to test AIO ================= 03:37:01 (1759822621)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.151703 s, 276 MB/s
osc.lustre-OST0000-osc-ffff9120ba4a5000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=189996: Tue Oct  7 03:37:11 2025
  write: IOPS=422, BW=1691KiB/s (1731kB/s)(10.0MiB/6057msec); 0 zone resets
    slat (usec): min=39, max=20482, avg=121.29, stdev=406.64
    clat (usec): min=6390, max=60618, avg=37366.77, stdev=4605.87
     lat (usec): min=6662, max=60727, avg=37489.27, stdev=4587.97
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[30278], 10.00th=[32375], 20.00th=[34341],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[39060], 80.00th=[40633], 90.00th=[42206], 95.00th=[43779],
     | 99.00th=[52167], 99.50th=[56886], 99.90th=[60556], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min= 1584, max= 1800, per=25.08%, avg=1690.91, stdev=69.77, samples=11
   iops        : min=  396, max=  450, avg=422.73, stdev=17.44, samples=11
  lat (msec)   : 10=0.12%, 20=0.31%, 50=98.28%, 100=1.29%
  cpu          : usr=0.38%, sys=4.44%, ctx=2684, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189997: Tue Oct  7 03:37:11 2025
  write: IOPS=421, BW=1686KiB/s (1726kB/s)(10.0MiB/6074msec); 0 zone resets
    slat (usec): min=23, max=506, avg=112.97, stdev=58.90
    clat (usec): min=24672, max=60247, avg=37388.10, stdev=4259.41
     lat (usec): min=24768, max=60326, avg=37502.31, stdev=4263.57
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[30802], 10.00th=[32375], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[36439], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39060], 80.00th=[40109], 90.00th=[42206], 95.00th=[43779],
     | 99.00th=[50594], 99.50th=[56361], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min= 1592, max= 1800, per=24.97%, avg=1683.45, stdev=62.51, samples=11
   iops        : min=  398, max=  450, avg=420.82, stdev=15.60, samples=11
  lat (msec)   : 50=98.91%, 100=1.09%
  cpu          : usr=0.46%, sys=3.95%, ctx=2504, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189998: Tue Oct  7 03:37:11 2025
  write: IOPS=421, BW=1687KiB/s (1727kB/s)(10.0MiB/6071msec); 0 zone resets
    slat (usec): min=28, max=477, avg=114.93, stdev=59.47
    clat (usec): min=8941, max=60172, avg=37400.08, stdev=4287.76
     lat (usec): min=9118, max=60327, avg=37516.29, stdev=4291.01
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[31065], 10.00th=[32375], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[36439], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39060], 80.00th=[40633], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[50594], 99.50th=[56361], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min= 1576, max= 1800, per=24.98%, avg=1684.36, stdev=66.78, samples=11
   iops        : min=  394, max=  450, avg=421.09, stdev=16.69, samples=11
  lat (msec)   : 10=0.04%, 50=98.87%, 100=1.09%
  cpu          : usr=0.41%, sys=4.33%, ctx=2529, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189999: Tue Oct  7 03:37:11 2025
  write: IOPS=422, BW=1692KiB/s (1732kB/s)(10.0MiB/6053msec); 0 zone resets
    slat (usec): min=35, max=526, avg=113.39, stdev=57.14
    clat (usec): min=16870, max=85947, avg=37658.10, stdev=5238.05
     lat (usec): min=16992, max=85999, avg=37772.78, stdev=5239.69
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30802], 10.00th=[32637], 20.00th=[34341],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[39584], 80.00th=[40633], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[56886], 99.50th=[60556], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min= 1576, max= 1801, per=25.08%, avg=1691.00, stdev=72.53, samples=11
   iops        : min=  394, max=  450, avg=422.73, stdev=18.09, samples=11
  lat (msec)   : 20=0.35%, 50=97.93%, 100=1.72%
  cpu          : usr=0.66%, sys=3.80%, ctx=2630, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6743KiB/s (6905kB/s), 1686KiB/s-1692KiB/s (1726kB/s-1732kB/s), io=40.0MiB (41.9MB), run=6053-6074msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190009: Tue Oct  7 03:37:21 2025
  read: IOPS=186, BW=747KiB/s (765kB/s)(5048KiB/6759msec)
    slat (usec): min=38, max=8466, avg=3167.84, stdev=1175.53
    clat (usec): min=9909, max=68214, avg=40408.00, stdev=8300.92
     lat (usec): min=14987, max=69488, avg=43578.93, stdev=8724.17
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28443], 10.00th=[30278], 20.00th=[32900],
     | 30.00th=[34866], 40.00th=[37487], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44827], 80.00th=[47973], 90.00th=[51643], 95.00th=[54264],
     | 99.00th=[59507], 99.50th=[61080], 99.90th=[66847], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  616, max= 1008, per=25.12%, avg=743.15, stdev=128.72, samples=13
   iops        : min=  154, max=  252, avg=185.77, stdev=32.17, samples=13
  write: IOPS=192, BW=768KiB/s (787kB/s)(5192KiB/6759msec); 0 zone resets
    slat (usec): min=38, max=483, avg=96.44, stdev=50.37
    clat (usec): min=24344, max=69067, avg=40385.64, stdev=7799.16
     lat (usec): min=24450, max=69136, avg=40483.28, stdev=7796.42
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29492], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[34866], 40.00th=[37487], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46924], 90.00th=[50594], 95.00th=[54264],
     | 99.00th=[60556], 99.50th=[64226], 99.90th=[66323], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  544, max=  960, per=24.96%, avg=769.08, stdev=116.64, samples=13
   iops        : min=  136, max=  240, avg=192.15, stdev=29.19, samples=13
  lat (msec)   : 10=0.04%, 20=0.08%, 50=87.03%, 100=12.85%
  cpu          : usr=0.40%, sys=4.39%, ctx=4621, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190010: Tue Oct  7 03:37:21 2025
  read: IOPS=183, BW=734KiB/s (752kB/s)(4976KiB/6780msec)
    slat (usec): min=1045, max=12714, avg=3147.90, stdev=1129.33
    clat (usec): min=18313, max=68631, avg=40480.99, stdev=8369.82
     lat (usec): min=21017, max=74791, avg=43630.65, stdev=8772.84
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28443], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[34866], 40.00th=[37487], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44827], 80.00th=[47973], 90.00th=[51643], 95.00th=[55313],
     | 99.00th=[61604], 99.50th=[64226], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  604, max=  992, per=24.69%, avg=730.54, stdev=134.11, samples=13
   iops        : min=  151, max=  248, avg=182.62, stdev=33.54, samples=13
  write: IOPS=194, BW=776KiB/s (795kB/s)(5264KiB/6780msec); 0 zone resets
    slat (usec): min=29, max=399, avg=98.46, stdev=49.66
    clat (usec): min=10884, max=63759, avg=40242.24, stdev=7456.33
     lat (usec): min=10961, max=63841, avg=40341.93, stdev=7459.17
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29754], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34866], 40.00th=[37487], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46924], 90.00th=[50070], 95.00th=[52691],
     | 99.00th=[58459], 99.50th=[59507], 99.90th=[62129], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  560, max=  960, per=25.09%, avg=773.15, stdev=110.21, samples=13
   iops        : min=  140, max=  240, avg=193.23, stdev=27.53, samples=13
  lat (msec)   : 20=0.12%, 50=87.62%, 100=12.27%
  cpu          : usr=0.52%, sys=4.28%, ctx=4465, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190011: Tue Oct  7 03:37:21 2025
  read: IOPS=184, BW=739KiB/s (757kB/s)(4984KiB/6744msec)
    slat (usec): min=28, max=10247, avg=3191.38, stdev=1148.63
    clat (usec): min=5670, max=67821, avg=40615.35, stdev=8544.42
     lat (usec): min=5750, max=71144, avg=43808.52, stdev=8966.76
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27919], 10.00th=[30540], 20.00th=[33162],
     | 30.00th=[35390], 40.00th=[37487], 50.00th=[40109], 60.00th=[42730],
     | 70.00th=[45876], 80.00th=[48497], 90.00th=[51119], 95.00th=[54789],
     | 99.00th=[60031], 99.50th=[62653], 99.90th=[65799], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  592, max=  944, per=24.94%, avg=737.92, stdev=105.74, samples=13
   iops        : min=  148, max=  236, avg=184.38, stdev=26.42, samples=13
  write: IOPS=194, BW=779KiB/s (798kB/s)(5256KiB/6744msec); 0 zone resets
    slat (usec): min=26, max=4489, avg=102.39, stdev=153.27
    clat (usec): min=24610, max=71309, avg=40333.48, stdev=7803.46
     lat (usec): min=24688, max=71403, avg=40436.93, stdev=7801.42
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29754], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34866], 40.00th=[37487], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46924], 90.00th=[51119], 95.00th=[53740],
     | 99.00th=[59507], 99.50th=[66847], 99.90th=[70779], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  584, max=  976, per=25.16%, avg=775.23, stdev=132.47, samples=13
   iops        : min=  146, max=  244, avg=193.77, stdev=33.10, samples=13
  lat (msec)   : 10=0.12%, 20=0.20%, 50=86.64%, 100=13.05%
  cpu          : usr=0.47%, sys=4.24%, ctx=4496, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190012: Tue Oct  7 03:37:21 2025
  read: IOPS=186, BW=746KiB/s (764kB/s)(5060KiB/6780msec)
    slat (usec): min=1140, max=17234, avg=3157.15, stdev=1212.61
    clat (usec): min=6704, max=64710, avg=40397.90, stdev=8268.52
     lat (usec): min=9012, max=73336, avg=43556.85, stdev=8712.30
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27919], 10.00th=[30278], 20.00th=[32900],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[44827], 80.00th=[47973], 90.00th=[51119], 95.00th=[54264],
     | 99.00th=[60556], 99.50th=[61604], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  616, max=  968, per=25.09%, avg=742.46, stdev=121.89, samples=13
   iops        : min=  154, max=  242, avg=185.54, stdev=30.48, samples=13
  write: IOPS=191, BW=764KiB/s (782kB/s)(5180KiB/6780msec); 0 zone resets
    slat (usec): min=34, max=2663, avg=101.18, stdev=90.15
    clat (usec): min=11239, max=66065, avg=40321.99, stdev=7849.11
     lat (usec): min=11335, max=66121, avg=40424.28, stdev=7845.79
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34866], 40.00th=[36963], 50.00th=[39584], 60.00th=[42206],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[51119], 95.00th=[53216],
     | 99.00th=[60031], 99.50th=[61080], 99.90th=[64226], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  576, max=  968, per=24.62%, avg=758.69, stdev=125.85, samples=13
   iops        : min=  144, max=  242, avg=189.62, stdev=31.47, samples=13
  lat (msec)   : 10=0.04%, 20=0.08%, 50=86.60%, 100=13.28%
  cpu          : usr=0.35%, sys=4.40%, ctx=4499, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2960KiB/s (3031kB/s), 734KiB/s-747KiB/s (752kB/s-765kB/s), io=19.6MiB (20.5MB), run=6744-6780msec
  WRITE: bw=3081KiB/s (3155kB/s), 764KiB/s-779KiB/s (782kB/s-798kB/s), io=20.4MiB (21.4MB), run=6744-6780msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190017: Tue Oct  7 03:37:25 2025
  read: IOPS=10, BW=417MiB/s (437MB/s)(40.0MiB/96msec)
    slat (nsec): min=12020k, max=12020k, avg=12020422.00, stdev= 0.00
    clat (nsec): min=78500k, max=78500k, avg=78500439.00, stdev= 0.00
     lat (nsec): min=90524k, max=90524k, avg=90524298.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[78119],  5.00th=[78119], 10.00th=[78119], 20.00th=[78119],
     | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119],
     | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119],
     | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.58%, ctx=9, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=417MiB/s (437MB/s), 417MiB/s-417MiB/s (437MB/s-437MB/s), io=40.0MiB (41.9MB), run=96-96msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout