== sanity test 398c: run fio to test AIO ================= 23:18:57 (1784085537)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.138325 s, 303 MB/s
osc.lustre-OST0000-osc-ffff964343d78000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=194968: Tue Jul 14 23:19:06 2026
  write: IOPS=467, BW=1869KiB/s (1914kB/s)(10.0MiB/5478msec); 0 zone resets
    slat (usec): min=45, max=1104, avg=158.61, stdev=79.54
    clat (usec): min=3437, max=58971, avg=33779.17, stdev=4588.23
     lat (usec): min=3590, max=59105, avg=33938.82, stdev=4596.59
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[26608], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1704, max= 2008, per=24.94%, avg=1865.10, stdev=118.73, samples=10
   iops        : min=  426, max=  502, avg=466.20, stdev=29.60, samples=10
  lat (msec)   : 4=0.04%, 10=0.12%, 20=1.41%, 50=98.28%, 100=0.16%
  cpu          : usr=0.64%, sys=6.06%, ctx=2602, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194969: Tue Jul 14 23:19:06 2026
  write: IOPS=470, BW=1883KiB/s (1928kB/s)(10.0MiB/5439msec); 0 zone resets
    slat (usec): min=49, max=924, avg=160.61, stdev=79.83
    clat (usec): min=7931, max=63243, avg=33764.35, stdev=4580.47
     lat (usec): min=8082, max=63362, avg=33926.02, stdev=4586.07
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[26870], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[44827], 99.50th=[45351], 99.90th=[51119], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1672, max= 2153, per=25.15%, avg=1880.10, stdev=145.88, samples=10
   iops        : min=  418, max=  538, avg=470.00, stdev=36.42, samples=10
  lat (msec)   : 10=0.08%, 20=0.66%, 50=99.14%, 100=0.12%
  cpu          : usr=0.50%, sys=5.92%, ctx=2629, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194970: Tue Jul 14 23:19:06 2026
  write: IOPS=467, BW=1870KiB/s (1915kB/s)(10.0MiB/5477msec); 0 zone resets
    slat (usec): min=34, max=1494, avg=149.75, stdev=82.38
    clat (usec): min=7865, max=46909, avg=33757.90, stdev=4473.80
     lat (usec): min=8016, max=47260, avg=33908.64, stdev=4485.01
    clat percentiles (usec):
     |  1.00th=[20317],  5.00th=[26346], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min= 1680, max= 2016, per=24.88%, avg=1860.40, stdev=115.05, samples=10
   iops        : min=  420, max=  504, avg=465.10, stdev=28.76, samples=10
  lat (msec)   : 10=0.08%, 20=0.70%, 50=99.22%
  cpu          : usr=0.46%, sys=5.44%, ctx=2479, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194971: Tue Jul 14 23:19:06 2026
  write: IOPS=468, BW=1872KiB/s (1917kB/s)(10.0MiB/5470msec); 0 zone resets
    slat (usec): min=41, max=24123, avg=181.63, stdev=486.81
    clat (usec): min=6138, max=56537, avg=33748.63, stdev=4652.22
     lat (usec): min=8078, max=56857, avg=33931.31, stdev=4632.37
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[26608], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[44827], 99.50th=[46400], 99.90th=[53740], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min= 1664, max= 2064, per=25.00%, avg=1869.60, stdev=130.26, samples=10
   iops        : min=  416, max=  516, avg=467.40, stdev=32.57, samples=10
  lat (msec)   : 10=0.08%, 20=0.98%, 50=98.79%, 100=0.16%
  cpu          : usr=0.55%, sys=6.62%, ctx=2526, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7477KiB/s (7657kB/s), 1869KiB/s-1883KiB/s (1914kB/s-1928kB/s), io=40.0MiB (41.9MB), run=5439-5478msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=194981: Tue Jul 14 23:19:16 2026
  read: IOPS=185, BW=743KiB/s (761kB/s)(5048KiB/6794msec)
    slat (usec): min=1100, max=6906, avg=2822.16, stdev=846.80
    clat (usec): min=17235, max=61248, avg=40351.02, stdev=6435.61
     lat (usec): min=19848, max=64339, avg=43175.62, stdev=6528.38
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30540], 10.00th=[32375], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[38536], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[45351], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[56886], 99.50th=[57934], 99.90th=[60556], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  654, max=  864, per=25.04%, avg=739.46, stdev=53.06, samples=13
   iops        : min=  163, max=  216, avg=184.69, stdev=13.41, samples=13
  write: IOPS=191, BW=764KiB/s (783kB/s)(5192KiB/6794msec); 0 zone resets
    slat (usec): min=56, max=1824, avg=181.61, stdev=111.46
    clat (usec): min=7781, max=63223, avg=40981.87, stdev=5991.40
     lat (usec): min=8170, max=63315, avg=41164.64, stdev=5988.14
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[32900], 10.00th=[34341], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[56886], 99.50th=[57410], 99.90th=[60031], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  640, max=  904, per=24.76%, avg=761.46, stdev=71.72, samples=13
   iops        : min=  160, max=  226, avg=190.15, stdev=17.97, samples=13
  lat (msec)   : 10=0.08%, 20=0.12%, 50=92.42%, 100=7.38%
  cpu          : usr=0.57%, sys=6.02%, ctx=4528, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194982: Tue Jul 14 23:19:16 2026
  read: IOPS=183, BW=735KiB/s (752kB/s)(4976KiB/6774msec)
    slat (usec): min=1156, max=6753, avg=2824.63, stdev=844.40
    clat (usec): min=7779, max=76953, avg=40275.96, stdev=6746.22
     lat (usec): min=10566, max=79055, avg=43102.93, stdev=6838.28
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30540], 10.00th=[32637], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[58459], 99.50th=[60556], 99.90th=[74974], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  632, max=  824, per=24.89%, avg=734.92, stdev=51.66, samples=13
   iops        : min=  158, max=  206, avg=183.54, stdev=12.96, samples=13
  write: IOPS=194, BW=777KiB/s (796kB/s)(5264KiB/6774msec); 0 zone resets
    slat (usec): min=62, max=1112, avg=171.68, stdev=94.25
    clat (usec): min=11164, max=64172, avg=41068.46, stdev=5436.94
     lat (usec): min=11324, max=64393, avg=41241.23, stdev=5441.01
    clat percentiles (usec):
     |  1.00th=[30540],  5.00th=[33162], 10.00th=[34866], 20.00th=[36439],
     | 30.00th=[38011], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[48497], 95.00th=[50594],
     | 99.00th=[53740], 99.50th=[55837], 99.90th=[57410], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  664, max=  928, per=25.27%, avg=777.00, stdev=76.29, samples=13
   iops        : min=  166, max=  232, avg=194.08, stdev=19.19, samples=13
  lat (msec)   : 10=0.04%, 20=0.12%, 50=92.38%, 100=7.46%
  cpu          : usr=0.49%, sys=6.13%, ctx=4548, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194983: Tue Jul 14 23:19:16 2026
  read: IOPS=184, BW=737KiB/s (755kB/s)(4984KiB/6758msec)
    slat (usec): min=38, max=7196, avg=2814.70, stdev=875.11
    clat (usec): min=4081, max=60403, avg=40372.78, stdev=6363.20
     lat (usec): min=6845, max=63041, avg=43189.74, stdev=6441.51
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30540], 10.00th=[32375], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[38536], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[58983], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  632, max=  808, per=25.03%, avg=739.23, stdev=52.46, samples=13
   iops        : min=  158, max=  202, avg=184.62, stdev=13.09, samples=13
  write: IOPS=194, BW=778KiB/s (796kB/s)(5256KiB/6758msec); 0 zone resets
    slat (usec): min=37, max=1696, avg=176.45, stdev=103.89
    clat (usec): min=19960, max=64530, avg=41016.25, stdev=5805.63
     lat (usec): min=20190, max=64609, avg=41193.80, stdev=5808.90
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[32900], 10.00th=[34341], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[38536], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[48497], 95.00th=[51119],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[64226], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  664, max=  952, per=25.19%, avg=774.46, stdev=77.07, samples=13
   iops        : min=  166, max=  238, avg=193.46, stdev=19.41, samples=13
  lat (msec)   : 10=0.04%, 20=0.12%, 50=92.77%, 100=7.07%
  cpu          : usr=0.47%, sys=6.13%, ctx=4466, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194984: Tue Jul 14 23:19:16 2026
  read: IOPS=186, BW=745KiB/s (763kB/s)(5060KiB/6794msec)
    slat (usec): min=1215, max=7494, avg=2810.25, stdev=843.74
    clat (usec): min=12373, max=59226, avg=40036.13, stdev=6302.45
     lat (usec): min=15265, max=64800, avg=42848.79, stdev=6412.47
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30540], 10.00th=[32113], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[45351], 90.00th=[47973], 95.00th=[50594],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[57934], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  644, max=  848, per=25.18%, avg=743.69, stdev=57.30, samples=13
   iops        : min=  161, max=  212, avg=185.77, stdev=14.32, samples=13
  write: IOPS=190, BW=762KiB/s (781kB/s)(5180KiB/6794msec); 0 zone resets
    slat (usec): min=62, max=1001, avg=175.71, stdev=94.45
    clat (usec): min=9135, max=59309, avg=41247.57, stdev=5715.63
     lat (usec): min=9290, max=59420, avg=41424.46, stdev=5718.73
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[32900], 10.00th=[34866], 20.00th=[36439],
     | 30.00th=[38011], 40.00th=[39060], 50.00th=[40633], 60.00th=[42206],
     | 70.00th=[44303], 80.00th=[46400], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[58983], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  652, max=  872, per=24.67%, avg=758.46, stdev=70.27, samples=13
   iops        : min=  163, max=  218, avg=189.46, stdev=17.59, samples=13
  lat (msec)   : 10=0.04%, 20=0.16%, 50=93.24%, 100=6.56%
  cpu          : usr=0.66%, sys=6.01%, ctx=4565, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2954KiB/s (3025kB/s), 735KiB/s-745KiB/s (752kB/s-763kB/s), io=19.6MiB (20.5MB), run=6758-6794msec
  WRITE: bw=3075KiB/s (3149kB/s), 762KiB/s-778KiB/s (781kB/s-796kB/s), io=20.4MiB (21.4MB), run=6758-6794msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=194989: Tue Jul 14 23:19:20 2026
  read: IOPS=10, BW=404MiB/s (424MB/s)(40.0MiB/99msec)
    slat (nsec): min=10851k, max=10851k, avg=10851331.00, stdev= 0.00
    clat (nsec): min=82916k, max=82916k, avg=82915507.00, stdev= 0.00
     lat (nsec): min=93770k, max=93770k, avg=93770243.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[83362],  5.00th=[83362], 10.00th=[83362], 20.00th=[83362],
     | 30.00th=[83362], 40.00th=[83362], 50.00th=[83362], 60.00th=[83362],
     | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362],
     | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.20%, ctx=11, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=40.0MiB (41.9MB), run=99-99msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout