== sanity test 398c: run fio to test AIO ================= 06:18:00 (1768130280)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.131467 s, 319 MB/s
osc.lustre-OST0000-osc-ffff9bae1127f000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=195452: Sun Jan 11 06:18:09 2026
  write: IOPS=469, BW=1879KiB/s (1924kB/s)(10.0MiB/5451msec); 0 zone resets
    slat (usec): min=29, max=24628, avg=108.88, stdev=487.33
    clat (usec): min=21982, max=58228, avg=33667.23, stdev=3115.95
     lat (usec): min=22049, max=58427, avg=33777.19, stdev=3134.02
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28181], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35390], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[48497], 99.90th=[54789], 99.95th=[57934],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1776, max= 1936, per=25.02%, avg=1879.20, stdev=50.52, samples=10
   iops        : min=  444, max=  484, avg=469.80, stdev=12.63, samples=10
  lat (msec)   : 50=99.80%, 100=0.20%
  cpu          : usr=0.53%, sys=4.06%, ctx=2775, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195453: Sun Jan 11 06:18:09 2026
  write: IOPS=469, BW=1878KiB/s (1923kB/s)(10.0MiB/5452msec); 0 zone resets
    slat (usec): min=25, max=536, avg=98.62, stdev=49.75
    clat (usec): min=21672, max=59494, avg=33647.86, stdev=3097.30
     lat (usec): min=21741, max=59652, avg=33747.42, stdev=3100.33
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28181], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[32637], 40.00th=[33162], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35390], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[48497], 99.90th=[55837], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min= 1776, max= 1936, per=25.01%, avg=1878.40, stdev=46.16, samples=10
   iops        : min=  444, max=  484, avg=469.60, stdev=11.54, samples=10
  lat (msec)   : 50=99.84%, 100=0.16%
  cpu          : usr=0.48%, sys=3.74%, ctx=2729, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195454: Sun Jan 11 06:18:09 2026
  write: IOPS=473, BW=1893KiB/s (1938kB/s)(10.0MiB/5410msec); 0 zone resets
    slat (usec): min=25, max=435, avg=99.28, stdev=50.16
    clat (usec): min=7321, max=67414, avg=33654.88, stdev=3473.23
     lat (usec): min=7376, max=67564, avg=33755.22, stdev=3473.65
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28181], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[34866], 80.00th=[35390], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[41681], 99.50th=[49021], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1776, max= 2050, per=25.22%, avg=1894.60, stdev=74.06, samples=10
   iops        : min=  444, max=  512, avg=473.60, stdev=18.40, samples=10
  lat (msec)   : 10=0.08%, 20=0.23%, 50=99.45%, 100=0.23%
  cpu          : usr=0.50%, sys=3.70%, ctx=2730, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195455: Sun Jan 11 06:18:09 2026
  write: IOPS=471, BW=1885KiB/s (1930kB/s)(10.0MiB/5433msec); 0 zone resets
    slat (usec): min=31, max=22987, avg=111.51, stdev=457.22
    clat (usec): min=7894, max=64855, avg=33597.21, stdev=3510.46
     lat (usec): min=7976, max=64961, avg=33709.75, stdev=3481.22
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28181], 10.00th=[29754], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33424], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35390], 90.00th=[36439], 95.00th=[37487],
     | 99.00th=[45351], 99.50th=[49021], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[64750]
   bw (  KiB/s): min= 1760, max= 1980, per=25.14%, avg=1888.40, stdev=64.29, samples=10
   iops        : min=  440, max=  495, avg=472.10, stdev=16.07, samples=10
  lat (msec)   : 10=0.12%, 20=0.16%, 50=99.34%, 100=0.39%
  cpu          : usr=0.52%, sys=4.18%, ctx=2884, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7513KiB/s (7693kB/s), 1878KiB/s-1893KiB/s (1923kB/s-1938kB/s), io=40.0MiB (41.9MB), run=5410-5452msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=195465: Sun Jan 11 06:18:20 2026
  read: IOPS=168, BW=674KiB/s (690kB/s)(5048KiB/7490msec)
    slat (usec): min=1025, max=12899, avg=3572.43, stdev=1253.15
    clat (usec): min=7731, max=68683, avg=44568.63, stdev=6791.12
     lat (usec): min=9924, max=70491, avg=48142.44, stdev=6887.98
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33817], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[42730], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[47449], 80.00th=[50070], 90.00th=[53740], 95.00th=[55837],
     | 99.00th=[60556], 99.50th=[62653], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  584, max=  728, per=25.08%, avg=671.64, stdev=44.10, samples=14
   iops        : min=  146, max=  182, avg=167.86, stdev=11.11, samples=14
  write: IOPS=173, BW=693KiB/s (710kB/s)(5192KiB/7490msec); 0 zone resets
    slat (usec): min=34, max=2829, avg=92.54, stdev=95.14
    clat (usec): min=19260, max=68217, avg=44535.93, stdev=5863.28
     lat (usec): min=19335, max=68264, avg=44629.43, stdev=5863.78
    clat percentiles (usec):
     |  1.00th=[33162],  5.00th=[35914], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[42730], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[46924], 80.00th=[49021], 90.00th=[52167], 95.00th=[54789],
     | 99.00th=[61080], 99.50th=[63701], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  552, max=  776, per=24.81%, avg=691.71, stdev=68.40, samples=14
   iops        : min=  138, max=  194, avg=172.93, stdev=17.10, samples=14
  lat (msec)   : 10=0.12%, 20=0.04%, 50=81.91%, 100=17.93%
  cpu          : usr=0.39%, sys=3.49%, ctx=4747, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195466: Sun Jan 11 06:18:20 2026
  read: IOPS=166, BW=668KiB/s (684kB/s)(4976KiB/7451msec)
    slat (usec): min=51, max=12129, avg=3601.66, stdev=1248.86
    clat (usec): min=5877, max=69877, avg=44480.61, stdev=6477.25
     lat (usec): min=11526, max=72718, avg=48083.89, stdev=6593.45
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[33817], 10.00th=[36439], 20.00th=[39060],
     | 30.00th=[41157], 40.00th=[42730], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[47449], 80.00th=[50070], 90.00th=[52691], 95.00th=[54789],
     | 99.00th=[61080], 99.50th=[62129], 99.90th=[66847], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  624, max=  744, per=24.85%, avg=665.43, stdev=38.55, samples=14
   iops        : min=  156, max=  186, avg=166.36, stdev= 9.64, samples=14
  write: IOPS=176, BW=706KiB/s (723kB/s)(5264KiB/7451msec); 0 zone resets
    slat (usec): min=33, max=520, avg=91.09, stdev=47.76
    clat (usec): min=10668, max=69045, avg=44652.42, stdev=5584.50
     lat (usec): min=10724, max=69124, avg=44744.52, stdev=5584.79
    clat percentiles (usec):
     |  1.00th=[32900],  5.00th=[35914], 10.00th=[38011], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[47449], 80.00th=[49021], 90.00th=[52167], 95.00th=[53740],
     | 99.00th=[57934], 99.50th=[61604], 99.90th=[66847], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  592, max=  816, per=25.36%, avg=706.93, stdev=79.30, samples=14
   iops        : min=  148, max=  204, avg=176.71, stdev=19.82, samples=14
  lat (msec)   : 10=0.04%, 20=0.08%, 50=81.80%, 100=18.09%
  cpu          : usr=0.39%, sys=3.49%, ctx=4715, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195467: Sun Jan 11 06:18:20 2026
  read: IOPS=167, BW=669KiB/s (685kB/s)(4984KiB/7447msec)
    slat (usec): min=34, max=10129, avg=3603.33, stdev=1299.27
    clat (usec): min=17707, max=88137, avg=44760.41, stdev=7020.03
     lat (usec): min=17742, max=90410, avg=48365.17, stdev=7189.36
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[33817], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[47973], 80.00th=[50070], 90.00th=[53216], 95.00th=[55837],
     | 99.00th=[63177], 99.50th=[65799], 99.90th=[84411], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  544, max=  768, per=25.12%, avg=672.71, stdev=58.26, samples=14
   iops        : min=  136, max=  192, avg=168.14, stdev=14.56, samples=14
  write: IOPS=176, BW=706KiB/s (723kB/s)(5256KiB/7447msec); 0 zone resets
    slat (usec): min=29, max=4098, avg=92.54, stdev=120.78
    clat (usec): min=14188, max=82731, avg=44542.45, stdev=6528.33
     lat (usec): min=14328, max=82839, avg=44635.90, stdev=6530.14
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[35914], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47449], 80.00th=[49021], 90.00th=[52167], 95.00th=[55313],
     | 99.00th=[63177], 99.50th=[66323], 99.90th=[81265], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min=  560, max=  824, per=25.17%, avg=701.86, stdev=68.66, samples=14
   iops        : min=  140, max=  206, avg=175.43, stdev=17.16, samples=14
  lat (msec)   : 20=0.62%, 50=81.45%, 100=17.93%
  cpu          : usr=0.36%, sys=3.45%, ctx=4547, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195468: Sun Jan 11 06:18:20 2026
  read: IOPS=168, BW=675KiB/s (692kB/s)(5060KiB/7491msec)
    slat (usec): min=1090, max=9147, avg=3571.70, stdev=1201.04
    clat (usec): min=17083, max=71896, avg=44554.82, stdev=6586.33
     lat (usec): min=20207, max=73924, avg=48127.97, stdev=6708.72
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[34341], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[42730], 50.00th=[44303], 60.00th=[45876],
     | 70.00th=[47449], 80.00th=[49546], 90.00th=[53216], 95.00th=[55837],
     | 99.00th=[61604], 99.50th=[63177], 99.90th=[69731], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  592, max=  776, per=25.06%, avg=671.00, stdev=52.62, samples=14
   iops        : min=  148, max=  194, avg=167.71, stdev=13.19, samples=14
  write: IOPS=172, BW=691KiB/s (708kB/s)(5180KiB/7491msec); 0 zone resets
    slat (usec): min=33, max=511, avg=90.53, stdev=47.12
    clat (usec): min=20214, max=69702, avg=44812.19, stdev=5969.19
     lat (usec): min=20322, max=69792, avg=44903.69, stdev=5969.43
    clat percentiles (usec):
     |  1.00th=[33424],  5.00th=[36439], 10.00th=[38011], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[42730], 50.00th=[44303], 60.00th=[45351],
     | 70.00th=[46924], 80.00th=[49546], 90.00th=[52691], 95.00th=[54789],
     | 99.00th=[63177], 99.50th=[66323], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  560, max=  824, per=24.76%, avg=690.36, stdev=67.29, samples=14
   iops        : min=  140, max=  206, avg=172.57, stdev=16.83, samples=14
  lat (msec)   : 20=0.04%, 50=82.30%, 100=17.66%
  cpu          : usr=0.27%, sys=3.59%, ctx=4743, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2679KiB/s (2743kB/s), 668KiB/s-675KiB/s (684kB/s-692kB/s), io=19.6MiB (20.5MB), run=7447-7491msec
  WRITE: bw=2789KiB/s (2856kB/s), 691KiB/s-706KiB/s (708kB/s-723kB/s), io=20.4MiB (21.4MB), run=7447-7491msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=195473: Sun Jan 11 06:18:23 2026
  read: IOPS=9, BW=360MiB/s (378MB/s)(40.0MiB/111msec)
    slat (nsec): min=9187.1k, max=9187.1k, avg=9187063.00, stdev= 0.00
    clat (nsec): min=96059k, max=96059k, avg=96059095.00, stdev= 0.00
     lat (nsec): min=105250k, max=105250k, avg=105250313.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[95945],  5.00th=[95945], 10.00th=[95945], 20.00th=[95945],
     | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945],
     | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945],
     | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945],
     | 99.99th=[95945]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.27%, ctx=8, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=40.0MiB (41.9MB), run=111-111msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout