== sanity test 398c: run fio to test AIO ================= 23:08:28 (1775876908)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.127067 s, 330 MB/s
osc.lustre-OST0000-osc-ffff906ec5557000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182954: Fri Apr 10 23:08:37 2026
  write: IOPS=408, BW=1634KiB/s (1674kB/s)(10.0MiB/6265msec); 0 zone resets
    slat (usec): min=43, max=23047, avg=186.48, stdev=470.99
    clat (usec): min=4426, max=68187, avg=38850.27, stdev=6331.23
     lat (usec): min=4551, max=68475, avg=39037.89, stdev=6318.59
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[30016], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[40109],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[49546],
     | 99.00th=[55837], 99.50th=[58983], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[68682]
   bw (  KiB/s): min= 1456, max= 1848, per=25.03%, avg=1632.83, stdev=120.11, samples=12
   iops        : min=  364, max=  462, avg=408.17, stdev=29.99, samples=12
  lat (msec)   : 10=0.23%, 20=0.04%, 50=95.51%, 100=4.22%
  cpu          : usr=0.54%, sys=5.62%, ctx=2834, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182955: Fri Apr 10 23:08:37 2026
  write: IOPS=409, BW=1640KiB/s (1679kB/s)(10.0MiB/6244msec); 0 zone resets
    slat (usec): min=36, max=3383, avg=194.08, stdev=134.43
    clat (usec): min=10309, max=66644, avg=38777.13, stdev=6273.18
     lat (usec): min=10464, max=66813, avg=38972.44, stdev=6281.69
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30278], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[55837], 99.50th=[62129], 99.90th=[66323], 99.95th=[66323],
     | 99.99th=[66847]
   bw (  KiB/s): min= 1456, max= 1888, per=25.09%, avg=1636.92, stdev=130.47, samples=12
   iops        : min=  364, max=  472, avg=409.17, stdev=32.53, samples=12
  lat (msec)   : 20=0.70%, 50=95.62%, 100=3.67%
  cpu          : usr=0.62%, sys=5.91%, ctx=2893, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182956: Fri Apr 10 23:08:37 2026
  write: IOPS=407, BW=1632KiB/s (1671kB/s)(10.0MiB/6276msec); 0 zone resets
    slat (usec): min=53, max=21923, avg=199.92, stdev=447.71
    clat (usec): min=6775, max=64316, avg=38770.89, stdev=6224.36
     lat (usec): min=6939, max=64449, avg=38971.96, stdev=6212.10
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30540], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[49021],
     | 99.00th=[53216], 99.50th=[61080], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min= 1440, max= 1888, per=24.95%, avg=1627.58, stdev=127.08, samples=12
   iops        : min=  360, max=  472, avg=406.83, stdev=31.73, samples=12
  lat (msec)   : 10=0.04%, 20=0.66%, 50=95.55%, 100=3.75%
  cpu          : usr=0.45%, sys=6.22%, ctx=2803, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182957: Fri Apr 10 23:08:37 2026
  write: IOPS=407, BW=1631KiB/s (1670kB/s)(10.0MiB/6279msec); 0 zone resets
    slat (usec): min=40, max=3685, avg=190.69, stdev=139.42
    clat (usec): min=6550, max=64507, avg=38773.76, stdev=6106.39
     lat (usec): min=6809, max=64758, avg=38965.61, stdev=6113.51
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[30016], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38011], 60.00th=[40109],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[46924], 95.00th=[49021],
     | 99.00th=[53216], 99.50th=[56361], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64750]
   bw (  KiB/s): min= 1464, max= 1912, per=24.89%, avg=1623.75, stdev=126.91, samples=12
   iops        : min=  366, max=  478, avg=405.92, stdev=31.72, samples=12
  lat (msec)   : 10=0.04%, 20=0.20%, 50=96.37%, 100=3.40%
  cpu          : usr=0.61%, sys=6.02%, ctx=2988, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6523KiB/s (6680kB/s), 1631KiB/s-1640KiB/s (1670kB/s-1679kB/s), io=40.0MiB (41.9MB), run=6244-6279msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182967: Fri Apr 10 23:08:47 2026
  read: IOPS=200, BW=801KiB/s (821kB/s)(5048KiB/6300msec)
    slat (usec): min=1048, max=11065, avg=2630.55, stdev=981.79
    clat (usec): min=19851, max=70907, avg=37322.30, stdev=7067.99
     lat (usec): min=23021, max=73655, avg=39954.84, stdev=7331.82
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27657], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[62653], 99.50th=[64226], 99.90th=[68682], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  712, max=  984, per=25.10%, avg=799.58, stdev=84.89, samples=12
   iops        : min=  178, max=  246, avg=199.83, stdev=21.26, samples=12
  write: IOPS=206, BW=824KiB/s (844kB/s)(5192KiB/6300msec); 0 zone resets
    slat (usec): min=50, max=1118, avg=162.72, stdev=89.58
    clat (usec): min=19683, max=72287, avg=38015.49, stdev=6881.75
     lat (usec): min=19834, max=72431, avg=38179.29, stdev=6885.08
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[29754], 10.00th=[31065], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45876], 95.00th=[51119],
     | 99.00th=[65799], 99.50th=[68682], 99.90th=[70779], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  728, max=  936, per=24.67%, avg=818.08, stdev=72.66, samples=12
   iops        : min=  182, max=  234, avg=204.50, stdev=18.19, samples=12
  lat (msec)   : 20=0.08%, 50=93.98%, 100=5.94%
  cpu          : usr=0.49%, sys=5.94%, ctx=4419, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182968: Fri Apr 10 23:08:47 2026
  read: IOPS=197, BW=791KiB/s (810kB/s)(4976KiB/6287msec)
    slat (usec): min=1135, max=9913, avg=2595.67, stdev=973.53
    clat (usec): min=5508, max=76733, avg=37475.03, stdev=7417.02
     lat (usec): min=8058, max=78885, avg=40073.05, stdev=7662.13
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28181], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42730], 90.00th=[46924], 95.00th=[51643],
     | 99.00th=[62129], 99.50th=[66323], 99.90th=[76022], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  696, max=  928, per=24.78%, avg=789.17, stdev=74.48, samples=12
   iops        : min=  174, max=  232, avg=197.25, stdev=18.67, samples=12
  write: IOPS=209, BW=837KiB/s (857kB/s)(5264KiB/6287msec); 0 zone resets
    slat (usec): min=56, max=1198, avg=160.70, stdev=83.38
    clat (usec): min=13859, max=69539, avg=37935.80, stdev=6690.30
     lat (usec): min=14010, max=69668, avg=38097.64, stdev=6695.81
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29754], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[46400], 95.00th=[51119],
     | 99.00th=[61080], 99.50th=[63701], 99.90th=[67634], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  704, max=  960, per=25.17%, avg=834.75, stdev=84.47, samples=12
   iops        : min=  176, max=  240, avg=208.67, stdev=21.12, samples=12
  lat (msec)   : 10=0.12%, 20=0.12%, 50=93.95%, 100=5.82%
  cpu          : usr=0.43%, sys=5.85%, ctx=4470, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182969: Fri Apr 10 23:08:47 2026
  read: IOPS=199, BW=797KiB/s (816kB/s)(4984KiB/6254msec)
    slat (usec): min=62, max=9025, avg=2624.14, stdev=990.61
    clat (usec): min=16007, max=76880, avg=37440.38, stdev=7200.51
     lat (usec): min=18633, max=78662, avg=40066.72, stdev=7454.26
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[38011],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[51119],
     | 99.00th=[63701], 99.50th=[66323], 99.90th=[74974], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  680, max=  888, per=25.03%, avg=797.17, stdev=64.55, samples=12
   iops        : min=  170, max=  222, avg=199.25, stdev=16.13, samples=12
  write: IOPS=210, BW=840KiB/s (861kB/s)(5256KiB/6254msec); 0 zone resets
    slat (usec): min=58, max=879, avg=167.93, stdev=89.73
    clat (usec): min=15568, max=70452, avg=37852.29, stdev=7013.20
     lat (usec): min=15717, max=70796, avg=38021.39, stdev=7016.45
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29492], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[46924], 95.00th=[52167],
     | 99.00th=[61604], 99.50th=[64750], 99.90th=[67634], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  760, max= 1000, per=25.24%, avg=836.92, stdev=83.46, samples=12
   iops        : min=  190, max=  250, avg=209.17, stdev=20.88, samples=12
  lat (msec)   : 20=0.20%, 50=93.48%, 100=6.33%
  cpu          : usr=0.61%, sys=5.73%, ctx=4521, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182970: Fri Apr 10 23:08:47 2026
  read: IOPS=200, BW=803KiB/s (823kB/s)(5060KiB/6298msec)
    slat (usec): min=1051, max=12731, avg=2621.92, stdev=1019.65
    clat (usec): min=1702, max=79492, avg=37381.57, stdev=7251.30
     lat (usec): min=4137, max=81616, avg=40006.17, stdev=7492.30
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28181], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[42206], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[60556], 99.50th=[64750], 99.90th=[68682], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  640, max=  928, per=25.13%, avg=800.25, stdev=85.74, samples=12
   iops        : min=  160, max=  232, avg=200.00, stdev=21.47, samples=12
  write: IOPS=205, BW=822KiB/s (842kB/s)(5180KiB/6298msec); 0 zone resets
    slat (usec): min=54, max=2472, avg=166.73, stdev=118.95
    clat (usec): min=15350, max=68376, avg=38079.20, stdev=7050.90
     lat (usec): min=15633, max=68553, avg=38247.07, stdev=7053.18
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29492], 10.00th=[30802], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[47449], 95.00th=[52691],
     | 99.00th=[62129], 99.50th=[64750], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  720, max=  936, per=24.64%, avg=817.17, stdev=58.21, samples=12
   iops        : min=  180, max=  234, avg=204.25, stdev=14.59, samples=12
  lat (msec)   : 2=0.04%, 10=0.04%, 20=0.12%, 50=93.52%, 100=6.29%
  cpu          : usr=0.35%, sys=6.03%, ctx=4513, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3185KiB/s (3262kB/s), 791KiB/s-803KiB/s (810kB/s-823kB/s), io=19.6MiB (20.5MB), run=6254-6300msec
  WRITE: bw=3316KiB/s (3396kB/s), 822KiB/s-840KiB/s (842kB/s-861kB/s), io=20.4MiB (21.4MB), run=6254-6300msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182975: Fri Apr 10 23:08:50 2026
  read: IOPS=9, BW=381MiB/s (399MB/s)(40.0MiB/105msec)
    slat (nsec): min=10830k, max=10830k, avg=10829620.00, stdev= 0.00
    clat (nsec): min=89862k, max=89862k, avg=89862260.00, stdev= 0.00
     lat (nsec): min=100696k, max=100696k, avg=100695918.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[89654],  5.00th=[89654], 10.00th=[89654], 20.00th=[89654],
     | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654],
     | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654],
     | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.65%, ctx=9, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=381MiB/s (399MB/s), 381MiB/s-381MiB/s (399MB/s-399MB/s), io=40.0MiB (41.9MB), run=105-105msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout