== sanity test 398c: run fio to test AIO ================= 14:22:55 (1771442575)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.141553 s, 296 MB/s
osc.lustre-OST0000-osc-ffffa03c04a5f000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=195505: Wed Feb 18 14:23:06 2026
  write: IOPS=385, BW=1541KiB/s (1578kB/s)(10.0MiB/6645msec); 0 zone resets
    slat (usec): min=50, max=28062, avg=211.81, stdev=563.26
    clat (usec): min=9126, max=89298, avg=41026.26, stdev=7324.99
     lat (usec): min=9280, max=89458, avg=41239.29, stdev=7310.93
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[30802], 10.00th=[32900], 20.00th=[35390],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40109], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49021], 95.00th=[52691],
     | 99.00th=[67634], 99.50th=[70779], 99.90th=[85459], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min= 1384, max= 1728, per=25.00%, avg=1538.54, stdev=112.11, samples=13
   iops        : min=  346, max=  432, avg=384.62, stdev=28.04, samples=13
  lat (msec)   : 10=0.04%, 20=0.08%, 50=91.13%, 100=8.75%
  cpu          : usr=0.69%, sys=6.14%, ctx=2833, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195506: Wed Feb 18 14:23:06 2026
  write: IOPS=387, BW=1551KiB/s (1588kB/s)(10.0MiB/6602msec); 0 zone resets
    slat (usec): min=40, max=3314, avg=196.00, stdev=136.15
    clat (usec): min=7051, max=71851, avg=40990.63, stdev=7031.00
     lat (usec): min=7133, max=71981, avg=41187.74, stdev=7038.26
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[30802], 10.00th=[33162], 20.00th=[35390],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[45876], 90.00th=[49021], 95.00th=[52691],
     | 99.00th=[66323], 99.50th=[67634], 99.90th=[70779], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1376, max= 1696, per=25.14%, avg=1547.33, stdev=107.30, samples=12
   iops        : min=  344, max=  424, avg=386.83, stdev=26.83, samples=12
  lat (msec)   : 10=0.04%, 20=0.12%, 50=91.13%, 100=8.71%
  cpu          : usr=0.48%, sys=5.80%, ctx=2831, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195507: Wed Feb 18 14:23:06 2026
  write: IOPS=384, BW=1539KiB/s (1576kB/s)(10.0MiB/6655msec); 0 zone resets
    slat (usec): min=35, max=1902, avg=152.41, stdev=108.36
    clat (usec): min=22313, max=78667, avg=41064.33, stdev=6729.75
     lat (usec): min=22431, max=78766, avg=41217.90, stdev=6739.65
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[31327], 10.00th=[33424], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[45351], 90.00th=[48497], 95.00th=[52167],
     | 99.00th=[64750], 99.50th=[66847], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1352, max= 1712, per=25.03%, avg=1540.62, stdev=127.02, samples=13
   iops        : min=  338, max=  428, avg=385.15, stdev=31.75, samples=13
  lat (msec)   : 50=92.73%, 100=7.27%
  cpu          : usr=0.48%, sys=4.99%, ctx=2372, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195508: Wed Feb 18 14:23:06 2026
  write: IOPS=384, BW=1539KiB/s (1576kB/s)(10.0MiB/6655msec); 0 zone resets
    slat (usec): min=50, max=26848, avg=218.32, stdev=542.99
    clat (usec): min=16051, max=79354, avg=41033.32, stdev=6999.01
     lat (usec): min=22462, max=79619, avg=41252.89, stdev=6991.09
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[31065], 10.00th=[33424], 20.00th=[35390],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49021], 95.00th=[52691],
     | 99.00th=[66323], 99.50th=[69731], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1400, max= 1736, per=25.01%, avg=1538.92, stdev=111.73, samples=13
   iops        : min=  350, max=  434, avg=384.69, stdev=27.97, samples=13
  lat (msec)   : 20=0.04%, 50=91.29%, 100=8.67%
  cpu          : usr=0.63%, sys=6.30%, ctx=2850, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6155KiB/s (6302kB/s), 1539KiB/s-1551KiB/s (1576kB/s-1588kB/s), io=40.0MiB (41.9MB), run=6602-6655msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=195519: Wed Feb 18 14:23:15 2026
  read: IOPS=194, BW=778KiB/s (797kB/s)(5048KiB/6486msec)
    slat (usec): min=1092, max=12000, avg=2975.81, stdev=1088.68
    clat (usec): min=19074, max=69393, avg=38534.48, stdev=7086.29
     lat (usec): min=21467, max=73890, avg=41512.77, stdev=7456.52
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27657], 10.00th=[29754], 20.00th=[32375],
     | 30.00th=[34341], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[47973], 95.00th=[51119],
     | 99.00th=[56361], 99.50th=[57410], 99.90th=[65799], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  648, max=  952, per=25.08%, avg=775.83, stdev=92.14, samples=12
   iops        : min=  162, max=  238, avg=193.83, stdev=23.12, samples=12
  write: IOPS=200, BW=800KiB/s (820kB/s)(5192KiB/6486msec); 0 zone resets
    slat (usec): min=47, max=3624, avg=162.76, stdev=131.38
    clat (usec): min=18934, max=67397, avg=38891.78, stdev=6775.33
     lat (usec): min=19063, max=67556, avg=39055.92, stdev=6777.58
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29492], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[38011], 60.00th=[40109],
     | 70.00th=[42730], 80.00th=[44827], 90.00th=[47973], 95.00th=[50594],
     | 99.00th=[57934], 99.50th=[61080], 99.90th=[65799], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  622, max=  920, per=24.81%, avg=799.08, stdev=97.49, samples=12
   iops        : min=  155, max=  230, avg=199.67, stdev=24.50, samples=12
  lat (msec)   : 20=0.12%, 50=93.67%, 100=6.21%
  cpu          : usr=0.54%, sys=5.81%, ctx=4576, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195520: Wed Feb 18 14:23:15 2026
  read: IOPS=191, BW=767KiB/s (786kB/s)(4976KiB/6486msec)
    slat (usec): min=1117, max=8733, avg=2959.28, stdev=1043.73
    clat (usec): min=18787, max=62409, avg=38585.65, stdev=7069.06
     lat (usec): min=20974, max=66825, avg=41547.06, stdev=7392.73
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28181], 10.00th=[30016], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[62129], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  638, max=  920, per=24.55%, avg=759.67, stdev=92.12, samples=12
   iops        : min=  159, max=  230, avg=189.83, stdev=23.12, samples=12
  write: IOPS=202, BW=812KiB/s (831kB/s)(5264KiB/6486msec); 0 zone resets
    slat (usec): min=36, max=1791, avg=162.43, stdev=93.95
    clat (usec): min=18687, max=64210, avg=38937.08, stdev=6682.00
     lat (usec): min=18778, max=64326, avg=39100.72, stdev=6685.04
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30016], 10.00th=[31327], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[47973], 95.00th=[50594],
     | 99.00th=[56886], 99.50th=[60031], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  656, max=  968, per=25.37%, avg=817.17, stdev=110.72, samples=12
   iops        : min=  164, max=  242, avg=204.17, stdev=27.77, samples=12
  lat (msec)   : 20=0.08%, 50=93.52%, 100=6.41%
  cpu          : usr=0.48%, sys=5.83%, ctx=4475, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195521: Wed Feb 18 14:23:15 2026
  read: IOPS=192, BW=769KiB/s (788kB/s)(4984KiB/6479msec)
    slat (usec): min=1044, max=13546, avg=2954.48, stdev=1079.80
    clat (usec): min=5779, max=66176, avg=38486.52, stdev=7364.52
     lat (usec): min=7196, max=70843, avg=41443.14, stdev=7647.54
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27919], 10.00th=[29754], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[35914], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[47973], 95.00th=[51119],
     | 99.00th=[57410], 99.50th=[58459], 99.90th=[65799], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  598, max=  944, per=25.02%, avg=774.00, stdev=92.99, samples=12
   iops        : min=  149, max=  236, avg=193.42, stdev=23.35, samples=12
  write: IOPS=202, BW=811KiB/s (831kB/s)(5256KiB/6479msec); 0 zone resets
    slat (usec): min=43, max=2386, avg=164.04, stdev=114.28
    clat (usec): min=22630, max=67209, avg=38756.60, stdev=6450.84
     lat (usec): min=22817, max=67396, avg=38921.85, stdev=6446.91
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29754], 10.00th=[31327], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[47449], 95.00th=[50070],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[66847], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  648, max= 1040, per=25.05%, avg=806.92, stdev=125.57, samples=12
   iops        : min=  162, max=  260, avg=201.58, stdev=31.45, samples=12
  lat (msec)   : 10=0.04%, 20=0.08%, 50=93.95%, 100=5.94%
  cpu          : usr=0.46%, sys=5.88%, ctx=4467, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195522: Wed Feb 18 14:23:15 2026
  read: IOPS=196, BW=785KiB/s (803kB/s)(5060KiB/6449msec)
    slat (usec): min=60, max=10321, avg=2932.34, stdev=1098.31
    clat (usec): min=6271, max=70606, avg=38539.58, stdev=7410.60
     lat (usec): min=6446, max=75372, avg=41474.23, stdev=7807.45
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27919], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35914], 50.00th=[37487], 60.00th=[40109],
     | 70.00th=[42206], 80.00th=[45351], 90.00th=[48497], 95.00th=[50594],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[62653], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  608, max=  920, per=25.23%, avg=780.67, stdev=94.50, samples=12
   iops        : min=  152, max=  230, avg=195.08, stdev=23.70, samples=12
  write: IOPS=200, BW=803KiB/s (823kB/s)(5180KiB/6449msec); 0 zone resets
    slat (usec): min=53, max=1472, avg=164.25, stdev=96.39
    clat (usec): min=14881, max=65502, avg=38835.93, stdev=6833.40
     lat (usec): min=15067, max=65629, avg=39001.38, stdev=6834.84
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[29754], 10.00th=[31327], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[47973], 95.00th=[50594],
     | 99.00th=[58459], 99.50th=[62129], 99.90th=[63701], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  638, max=  928, per=25.09%, avg=808.25, stdev=88.20, samples=12
   iops        : min=  159, max=  232, avg=201.92, stdev=22.12, samples=12
  lat (msec)   : 10=0.20%, 20=0.23%, 50=93.16%, 100=6.41%
  cpu          : usr=0.53%, sys=5.83%, ctx=4554, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3094KiB/s (3168kB/s), 767KiB/s-785KiB/s (786kB/s-803kB/s), io=19.6MiB (20.5MB), run=6449-6486msec
  WRITE: bw=3221KiB/s (3298kB/s), 800KiB/s-812KiB/s (820kB/s-831kB/s), io=20.4MiB (21.4MB), run=6449-6486msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=195526: Wed Feb 18 14:23:19 2026
  read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec)
    slat (nsec): min=9640.0k, max=9640.0k, avg=9640006.00, stdev= 0.00
    clat (nsec): min=95330k, max=95330k, avg=95329767.00, stdev= 0.00
     lat (nsec): min=104975k, max=104975k, avg=104975433.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[94897],  5.00th=[94897], 10.00th=[94897], 20.00th=[94897],
     | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897],
     | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897],
     | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.34%, ctx=10, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout