== sanity test 398c: run fio to test AIO ================= 22:25:57 (1781663157)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.151855 s, 276 MB/s
osc.lustre-OST0000-osc-ffff96f77290a000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=192253: Tue Jun 16 22:26:07 2026
  write: IOPS=416, BW=1664KiB/s (1704kB/s)(10.0MiB/6152msec); 0 zone resets
    slat (usec): min=56, max=31774, avg=221.47, stdev=633.64
    clat (usec): min=12102, max=69549, avg=37952.26, stdev=8230.14
     lat (usec): min=12389, max=69727, avg=38174.91, stdev=8214.54
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[29754], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[43254], 90.00th=[48497], 95.00th=[55837],
     | 99.00th=[63177], 99.50th=[65274], 99.90th=[68682], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1152, max= 2013, per=24.95%, avg=1658.42, stdev=267.44, samples=12
   iops        : min=  288, max=  503, avg=414.58, stdev=66.83, samples=12
  lat (msec)   : 20=1.84%, 50=89.18%, 100=8.98%
  cpu          : usr=0.47%, sys=6.99%, ctx=3016, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192254: Tue Jun 16 22:26:07 2026
  write: IOPS=415, BW=1662KiB/s (1702kB/s)(10.0MiB/6162msec); 0 zone resets
    slat (usec): min=45, max=30865, avg=217.80, stdev=616.28
    clat (usec): min=4658, max=69563, avg=37957.76, stdev=8167.15
     lat (usec): min=5159, max=69844, avg=38176.61, stdev=8152.12
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[29492], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[39584], 80.00th=[43254], 90.00th=[48497], 95.00th=[54789],
     | 99.00th=[63177], 99.50th=[64750], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1160, max= 1986, per=24.92%, avg=1656.17, stdev=261.11, samples=12
   iops        : min=  290, max=  496, avg=414.00, stdev=65.22, samples=12
  lat (msec)   : 10=0.08%, 20=1.84%, 50=89.49%, 100=8.59%
  cpu          : usr=0.19%, sys=7.08%, ctx=2968, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192255: Tue Jun 16 22:26:07 2026
  write: IOPS=417, BW=1672KiB/s (1712kB/s)(10.0MiB/6126msec); 0 zone resets
    slat (usec): min=41, max=953, avg=195.08, stdev=110.09
    clat (usec): min=11429, max=69285, avg=38031.45, stdev=8251.68
     lat (usec): min=11631, max=69543, avg=38227.57, stdev=8260.32
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[29492], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[34341], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[43779], 90.00th=[48497], 95.00th=[55313],
     | 99.00th=[63177], 99.50th=[65799], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1160, max= 2091, per=24.93%, avg=1656.92, stdev=287.35, samples=12
   iops        : min=  290, max=  522, avg=414.17, stdev=71.73, samples=12
  lat (msec)   : 20=1.88%, 50=89.18%, 100=8.95%
  cpu          : usr=0.34%, sys=6.24%, ctx=2820, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192256: Tue Jun 16 22:26:07 2026
  write: IOPS=415, BW=1662KiB/s (1702kB/s)(10.0MiB/6162msec); 0 zone resets
    slat (usec): min=41, max=962, avg=181.96, stdev=105.20
    clat (usec): min=3313, max=69503, avg=37980.36, stdev=8033.32
     lat (usec): min=3444, max=69703, avg=38163.43, stdev=8047.40
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[29230], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39584], 80.00th=[43254], 90.00th=[48497], 95.00th=[54789],
     | 99.00th=[63177], 99.50th=[64750], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1144, max= 1954, per=24.88%, avg=1653.50, stdev=258.75, samples=12
   iops        : min=  286, max=  488, avg=413.33, stdev=64.64, samples=12
  lat (msec)   : 4=0.04%, 20=1.84%, 50=89.26%, 100=8.87%
  cpu          : usr=0.47%, sys=5.62%, ctx=2906, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6647KiB/s (6807kB/s), 1662KiB/s-1672KiB/s (1702kB/s-1712kB/s), io=40.0MiB (41.9MB), run=6126-6162msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=192266: Tue Jun 16 22:26:17 2026
  read: IOPS=164, BW=656KiB/s (672kB/s)(5048KiB/7690msec)
    slat (usec): min=110, max=13725, avg=3520.49, stdev=1240.17
    clat (usec): min=6269, max=73706, avg=45866.41, stdev=7808.04
     lat (usec): min=7335, max=77848, avg=49388.44, stdev=7998.39
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[33817], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[52167], 90.00th=[55837], 95.00th=[58983],
     | 99.00th=[65274], 99.50th=[68682], 99.90th=[70779], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  568, max=  736, per=25.08%, avg=653.93, stdev=50.44, samples=15
   iops        : min=  142, max=  184, avg=163.47, stdev=12.62, samples=15
  write: IOPS=168, BW=675KiB/s (691kB/s)(5192KiB/7690msec); 0 zone resets
    slat (usec): min=54, max=1814, avg=190.72, stdev=125.42
    clat (usec): min=23122, max=77234, avg=46156.81, stdev=6857.48
     lat (usec): min=24169, max=77357, avg=46348.71, stdev=6836.28
    clat percentiles (usec):
     |  1.00th=[31327],  5.00th=[35390], 10.00th=[37487], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[54789], 95.00th=[57410],
     | 99.00th=[62129], 99.50th=[69731], 99.90th=[74974], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  552, max=  840, per=24.86%, avg=674.67, stdev=77.76, samples=15
   iops        : min=  138, max=  210, avg=168.60, stdev=19.42, samples=15
  lat (msec)   : 10=0.12%, 20=0.04%, 50=72.19%, 100=27.66%
  cpu          : usr=0.39%, sys=5.68%, ctx=4594, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192267: Tue Jun 16 22:26:17 2026
  read: IOPS=161, BW=646KiB/s (662kB/s)(4976KiB/7697msec)
    slat (usec): min=1049, max=13669, avg=3593.97, stdev=1241.48
    clat (usec): min=14749, max=78080, avg=45905.76, stdev=7430.22
     lat (usec): min=17749, max=81281, avg=49503.23, stdev=7613.25
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33817], 10.00th=[36439], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[54789], 95.00th=[57934],
     | 99.00th=[64226], 99.50th=[69731], 99.90th=[74974], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  560, max=  720, per=24.62%, avg=641.80, stdev=49.44, samples=15
   iops        : min=  140, max=  180, avg=160.40, stdev=12.35, samples=15
  write: IOPS=170, BW=684KiB/s (700kB/s)(5264KiB/7697msec); 0 zone resets
    slat (usec): min=56, max=3747, avg=190.43, stdev=139.17
    clat (usec): min=12197, max=73543, avg=45970.48, stdev=6605.69
     lat (usec): min=12282, max=73707, avg=46162.17, stdev=6596.64
    clat percentiles (usec):
     |  1.00th=[31065],  5.00th=[35390], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[54264], 95.00th=[56361],
     | 99.00th=[62653], 99.50th=[66323], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  568, max=  888, per=25.17%, avg=683.00, stdev=85.69, samples=15
   iops        : min=  142, max=  222, avg=170.73, stdev=21.44, samples=15
  lat (msec)   : 20=0.16%, 50=74.49%, 100=25.35%
  cpu          : usr=0.39%, sys=5.78%, ctx=4543, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192268: Tue Jun 16 22:26:17 2026
  read: IOPS=163, BW=654KiB/s (670kB/s)(4984KiB/7618msec)
    slat (usec): min=41, max=14445, avg=3513.67, stdev=1312.28
    clat (usec): min=5995, max=87230, avg=45529.78, stdev=8085.35
     lat (usec): min=8557, max=91526, avg=49045.12, stdev=8419.47
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[33817], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[55313], 95.00th=[57934],
     | 99.00th=[66847], 99.50th=[69731], 99.90th=[83362], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min=  573, max=  856, per=25.35%, avg=660.79, stdev=79.88, samples=14
   iops        : min=  143, max=  214, avg=165.14, stdev=19.94, samples=14
  write: IOPS=172, BW=690KiB/s (707kB/s)(5256KiB/7618msec); 0 zone resets
    slat (usec): min=41, max=4050, avg=190.83, stdev=166.99
    clat (usec): min=9332, max=86751, avg=45909.71, stdev=7750.25
     lat (usec): min=9670, max=86867, avg=46101.66, stdev=7743.29
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[34866], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44303], 50.00th=[45876], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[51643], 90.00th=[54789], 95.00th=[57934],
     | 99.00th=[64750], 99.50th=[70779], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  568, max=  800, per=25.41%, avg=689.50, stdev=69.62, samples=14
   iops        : min=  142, max=  200, avg=172.36, stdev=17.40, samples=14
  lat (msec)   : 10=0.23%, 20=0.47%, 50=72.85%, 100=26.45%
  cpu          : usr=0.34%, sys=5.88%, ctx=4490, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192269: Tue Jun 16 22:26:17 2026
  read: IOPS=164, BW=659KiB/s (675kB/s)(5060KiB/7681msec)
    slat (usec): min=133, max=13899, avg=3563.67, stdev=1273.67
    clat (usec): min=19598, max=74902, avg=45619.68, stdev=7388.45
     lat (usec): min=19913, max=80485, avg=49184.89, stdev=7677.89
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[33817], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[45351], 60.00th=[46924],
     | 70.00th=[49021], 80.00th=[51119], 90.00th=[55313], 95.00th=[57934],
     | 99.00th=[64750], 99.50th=[67634], 99.90th=[71828], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  576, max=  840, per=25.21%, avg=657.13, stdev=73.47, samples=15
   iops        : min=  144, max=  210, avg=164.27, stdev=18.38, samples=15
  write: IOPS=168, BW=674KiB/s (691kB/s)(5180KiB/7681msec); 0 zone resets
    slat (usec): min=41, max=2510, avg=194.19, stdev=142.20
    clat (usec): min=16002, max=83651, avg=46218.57, stdev=6878.64
     lat (usec): min=16216, max=84138, avg=46413.85, stdev=6860.80
    clat percentiles (usec):
     |  1.00th=[31065],  5.00th=[35390], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[46400], 60.00th=[47973],
     | 70.00th=[49546], 80.00th=[51119], 90.00th=[54264], 95.00th=[56886],
     | 99.00th=[64750], 99.50th=[68682], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  592, max=  784, per=24.77%, avg=672.20, stdev=67.31, samples=15
   iops        : min=  148, max=  196, avg=168.00, stdev=16.80, samples=15
  lat (msec)   : 20=0.12%, 50=73.20%, 100=26.68%
  cpu          : usr=0.46%, sys=5.86%, ctx=4502, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2607KiB/s (2670kB/s), 646KiB/s-659KiB/s (662kB/s-675kB/s), io=19.6MiB (20.5MB), run=7618-7697msec
  WRITE: bw=2714KiB/s (2779kB/s), 674KiB/s-690KiB/s (691kB/s-707kB/s), io=20.4MiB (21.4MB), run=7618-7697msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=192274: Tue Jun 16 22:26:21 2026
  read: IOPS=8, BW=328MiB/s (344MB/s)(40.0MiB/122msec)
    slat (nsec): min=8967.8k, max=8967.8k, avg=8967830.00, stdev= 0.00
    clat (nsec): min=107239k, max=107239k, avg=107239123.00, stdev= 0.00
     lat (nsec): min=116210k, max=116210k, avg=116209873.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  108], 10.00th=[  108], 20.00th=[  108],
     | 30.00th=[  108], 40.00th=[  108], 50.00th=[  108], 60.00th=[  108],
     | 70.00th=[  108], 80.00th=[  108], 90.00th=[  108], 95.00th=[  108],
     | 99.00th=[  108], 99.50th=[  108], 99.90th=[  108], 99.95th=[  108],
     | 99.99th=[  108]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.44%, ctx=5, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=40.0MiB (41.9MB), run=122-122msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout