== sanity test 398c: run fio to test AIO ================= 19:39:40 (1781048380)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.108163 s, 388 MB/s
osc.lustre-OST0000-osc-ffff897860491000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=147957: Tue Jun  9 19:39:46 2026
  write: IOPS=899, BW=3599KiB/s (3686kB/s)(10.0MiB/2845msec); 0 zone resets
    slat (usec): min=19, max=25566, avg=82.98, stdev=505.82
    clat (usec): min=1888, max=37592, avg=17525.51, stdev=2377.62
     lat (usec): min=1958, max=37654, avg=17609.40, stdev=2373.08
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[14353], 10.00th=[15008], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[30016], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 3480, max= 3688, per=24.93%, avg=3587.20, stdev=100.34, samples=5
   iops        : min=  870, max=  922, avg=896.80, stdev=25.08, samples=5
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.23%, 20=90.47%, 50=9.14%
  cpu          : usr=0.88%, sys=7.24%, ctx=1361, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147958: Tue Jun  9 19:39:46 2026
  write: IOPS=903, BW=3613KiB/s (3700kB/s)(10.0MiB/2834msec); 0 zone resets
    slat (usec): min=20, max=522, avg=72.08, stdev=39.67
    clat (usec): min=1893, max=41487, avg=17606.10, stdev=2589.41
     lat (usec): min=1952, max=41550, avg=17679.18, stdev=2587.04
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[14222], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[23987], 99.50th=[32637], 99.90th=[41681], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 3416, max= 3704, per=25.01%, avg=3597.80, stdev=123.47, samples=5
   iops        : min=  854, max=  926, avg=899.40, stdev=30.90, samples=5
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.08%, 20=90.82%, 50=8.95%
  cpu          : usr=0.71%, sys=6.57%, ctx=1351, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147959: Tue Jun  9 19:39:46 2026
  write: IOPS=899, BW=3597KiB/s (3683kB/s)(10.0MiB/2847msec); 0 zone resets
    slat (usec): min=20, max=418, avg=70.53, stdev=34.79
    clat (usec): min=1857, max=38129, avg=17661.99, stdev=2620.30
     lat (usec): min=1928, max=38194, avg=17733.41, stdev=2621.82
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14353], 10.00th=[15008], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[24249], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 3456, max= 3704, per=24.91%, avg=3584.00, stdev=99.12, samples=5
   iops        : min=  864, max=  926, avg=896.00, stdev=24.78, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 20=90.20%, 50=9.73%
  cpu          : usr=0.84%, sys=6.39%, ctx=1251, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=147960: Tue Jun  9 19:39:46 2026
  write: IOPS=899, BW=3597KiB/s (3683kB/s)(10.0MiB/2847msec); 0 zone resets
    slat (usec): min=22, max=309, avg=71.97, stdev=37.10
    clat (usec): min=1901, max=37990, avg=17661.96, stdev=2535.31
     lat (usec): min=1986, max=38065, avg=17734.88, stdev=2536.68
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14484], 10.00th=[15139], 20.00th=[16057],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[23987], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011],
     | 99.99th=[38011]
   bw (  KiB/s): min= 3456, max= 3672, per=24.88%, avg=3579.20, stdev=96.76, samples=5
   iops        : min=  864, max=  918, avg=894.80, stdev=24.19, samples=5
  lat (msec)   : 2=0.04%, 4=0.04%, 20=90.66%, 50=9.26%
  cpu          : usr=0.67%, sys=6.57%, ctx=1282, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.0MiB/s (14.7MB/s), 3597KiB/s-3613KiB/s (3683kB/s-3700kB/s), io=40.0MiB (41.9MB), run=2834-2847msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=147971: Tue Jun  9 19:39:52 2026
  read: IOPS=457, BW=1830KiB/s (1874kB/s)(5048KiB/2758msec)
    slat (usec): min=635, max=5042, avg=1087.85, stdev=294.22
    clat (usec): min=9340, max=28858, avg=16577.22, stdev=2704.09
     lat (usec): min=10805, max=29903, avg=17666.37, stdev=2744.09
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12387], 10.00th=[13173], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[24249], 99.90th=[26870], 99.95th=[28967],
     | 99.99th=[28967]
   bw (  KiB/s): min= 1757, max= 1872, per=25.18%, avg=1831.40, stdev=49.13, samples=5
   iops        : min=  439, max=  468, avg=457.80, stdev=12.38, samples=5
  write: IOPS=470, BW=1883KiB/s (1928kB/s)(5192KiB/2758msec); 0 zone resets
    slat (usec): min=22, max=1150, avg=71.06, stdev=54.37
    clat (usec): min=9666, max=26840, avg=16563.11, stdev=2278.50
     lat (usec): min=9719, max=26889, avg=16635.05, stdev=2279.95
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13173], 10.00th=[13829], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[24511], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1720, max= 2016, per=24.87%, avg=1883.40, stdev=115.14, samples=5
   iops        : min=  430, max=  504, avg=470.80, stdev=28.80, samples=5
  lat (msec)   : 10=0.16%, 20=90.51%, 50=9.34%
  cpu          : usr=0.98%, sys=8.52%, ctx=2268, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147972: Tue Jun  9 19:39:52 2026
  read: IOPS=450, BW=1804KiB/s (1847kB/s)(4976KiB/2759msec)
    slat (usec): min=655, max=5407, avg=1084.94, stdev=321.15
    clat (usec): min=2773, max=27725, avg=16601.69, stdev=2835.44
     lat (usec): min=3916, max=28506, avg=17687.89, stdev=2866.30
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12387], 10.00th=[13173], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15664], 50.00th=[16319], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[19006], 90.00th=[20579], 95.00th=[21627],
     | 99.00th=[23987], 99.50th=[24511], 99.90th=[26870], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1712, max= 1856, per=24.73%, avg=1798.40, stdev=68.68, samples=5
   iops        : min=  428, max=  464, avg=449.60, stdev=17.17, samples=5
  write: IOPS=476, BW=1908KiB/s (1954kB/s)(5264KiB/2759msec); 0 zone resets
    slat (usec): min=25, max=468, avg=70.40, stdev=41.08
    clat (usec): min=3499, max=27349, avg=16598.81, stdev=2330.04
     lat (usec): min=3563, max=27445, avg=16670.12, stdev=2330.35
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13173], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15270], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[22676], 99.50th=[23200], 99.90th=[26346], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1760, max= 2064, per=25.26%, avg=1912.80, stdev=114.14, samples=5
   iops        : min=  440, max=  516, avg=478.20, stdev=28.53, samples=5
  lat (msec)   : 4=0.08%, 10=0.12%, 20=89.88%, 50=9.92%
  cpu          : usr=0.98%, sys=8.41%, ctx=2201, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147973: Tue Jun  9 19:39:52 2026
  read: IOPS=454, BW=1818KiB/s (1861kB/s)(4984KiB/2742msec)
    slat (usec): min=28, max=4908, avg=1072.50, stdev=303.44
    clat (usec): min=3538, max=27152, avg=16343.92, stdev=2615.86
     lat (usec): min=4610, max=28355, avg=17417.67, stdev=2666.38
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12387], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15533], 50.00th=[16188], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19530], 95.00th=[21103],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[25822], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1760, max= 1952, per=25.12%, avg=1826.80, stdev=76.73, samples=5
   iops        : min=  440, max=  488, avg=456.60, stdev=19.15, samples=5
  write: IOPS=479, BW=1917KiB/s (1963kB/s)(5256KiB/2742msec); 0 zone resets
    slat (usec): min=21, max=426, avg=68.40, stdev=36.93
    clat (usec): min=2344, max=32533, avg=16711.77, stdev=2546.53
     lat (usec): min=2403, max=32563, avg=16781.12, stdev=2545.51
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13304], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16450], 60.00th=[17171],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[23462], 99.50th=[25035], 99.90th=[32375], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1808, max= 2040, per=25.25%, avg=1912.00, stdev=96.33, samples=5
   iops        : min=  452, max=  510, avg=478.00, stdev=24.08, samples=5
  lat (msec)   : 4=0.16%, 10=0.12%, 20=91.60%, 50=8.12%
  cpu          : usr=0.73%, sys=8.68%, ctx=2257, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=147974: Tue Jun  9 19:39:52 2026
  read: IOPS=461, BW=1845KiB/s (1889kB/s)(5060KiB/2743msec)
    slat (usec): min=29, max=4651, avg=1074.23, stdev=308.15
    clat (usec): min=1437, max=27214, avg=16433.73, stdev=2780.91
     lat (usec): min=2115, max=28973, avg=17509.43, stdev=2830.87
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12387], 10.00th=[13173], 20.00th=[14091],
     | 30.00th=[14877], 40.00th=[15533], 50.00th=[16188], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[20055], 95.00th=[21365],
     | 99.00th=[23462], 99.50th=[25560], 99.90th=[26870], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1784, max= 1896, per=25.40%, avg=1847.00, stdev=54.23, samples=5
   iops        : min=  446, max=  474, avg=461.60, stdev=13.74, samples=5
  write: IOPS=472, BW=1888KiB/s (1934kB/s)(5180KiB/2743msec); 0 zone resets
    slat (usec): min=20, max=871, avg=72.06, stdev=45.97
    clat (usec): min=2194, max=33594, avg=16629.80, stdev=2598.32
     lat (usec): min=2246, max=33656, avg=16702.99, stdev=2596.71
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[13173], 10.00th=[13698], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17695], 80.00th=[18482], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[22938], 99.50th=[23725], 99.90th=[28181], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1808, max= 1968, per=24.95%, avg=1889.20, stdev=65.17, samples=5
   iops        : min=  452, max=  492, avg=472.20, stdev=16.19, samples=5
  lat (msec)   : 2=0.04%, 4=0.20%, 10=0.27%, 20=89.18%, 50=10.31%
  cpu          : usr=0.77%, sys=8.61%, ctx=2298, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7274KiB/s (7448kB/s), 1804KiB/s-1845KiB/s (1847kB/s-1889kB/s), io=19.6MiB (20.5MB), run=2742-2759msec
  WRITE: bw=7572KiB/s (7754kB/s), 1883KiB/s-1917KiB/s (1928kB/s-1963kB/s), io=20.4MiB (21.4MB), run=2742-2759msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=147977: Tue Jun  9 19:39:55 2026
  read: IOPS=11, BW=455MiB/s (477MB/s)(40.0MiB/88msec)
    slat (nsec): min=14626k, max=14626k, avg=14626206.00, stdev= 0.00
    clat (nsec): min=69998k, max=69998k, avg=69997867.00, stdev= 0.00
     lat (nsec): min=84651k, max=84651k, avg=84651305.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[69731],  5.00th=[69731], 10.00th=[69731], 20.00th=[69731],
     | 30.00th=[69731], 40.00th=[69731], 50.00th=[69731], 60.00th=[69731],
     | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731],
     | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=17.24%, ctx=2, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=40.0MiB (41.9MB), run=88-88msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck