== sanity test 398c: run fio to test AIO ================= 16:34:41 (1748982881)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0798126 s, 526 MB/s
osc.lustre-OST0000-osc-ffff88013693e000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30151: Tue Jun  3 16:34:45 2025
  write: IOPS=991, BW=3967KiB/s (4063kB/s)(10.0MiB/2581msec)
    slat (usec): min=11, max=770, avg=36.29, stdev=23.29
    clat (usec): min=8936, max=33421, avg=16047.41, stdev=2374.18
     lat (usec): min=8987, max=33487, avg=16084.76, stdev=2375.35
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12649], 10.00th=[13304], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15401], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17433], 90.00th=[18482], 95.00th=[20055],
     | 99.00th=[24249], 99.50th=[26084], 99.90th=[33424], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 3808, max= 4240, per=25.16%, avg=3987.20, stdev=180.81, samples=5
   iops        : min=  952, max= 1060, avg=996.80, stdev=45.20, samples=5
  lat (msec)   : 10=0.08%, 20=95.12%, 50=4.80%
  cpu          : usr=0.70%, sys=4.65%, ctx=968, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30152: Tue Jun  3 16:34:45 2025
  write: IOPS=995, BW=3981KiB/s (4077kB/s)(10.0MiB/2572msec)
    slat (usec): min=11, max=777, avg=39.18, stdev=23.65
    clat (usec): min=2519, max=33095, avg=16008.70, stdev=2407.14
     lat (usec): min=2603, max=33154, avg=16048.99, stdev=2406.99
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[12387], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[24511], 99.50th=[24773], 99.90th=[32900], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 3768, max= 4224, per=25.27%, avg=4004.80, stdev=205.49, samples=5
   iops        : min=  942, max= 1056, avg=1001.20, stdev=51.37, samples=5
  lat (msec)   : 4=0.08%, 10=0.27%, 20=95.74%, 50=3.91%
  cpu          : usr=0.93%, sys=4.98%, ctx=1236, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30153: Tue Jun  3 16:34:45 2025
  write: IOPS=990, BW=3961KiB/s (4056kB/s)(10.0MiB/2585msec)
    slat (usec): min=11, max=801, avg=38.03, stdev=24.91
    clat (usec): min=7599, max=27346, avg=16072.54, stdev=2283.70
     lat (usec): min=7638, max=27385, avg=16111.75, stdev=2283.56
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12649], 10.00th=[13173], 20.00th=[14222],
     | 30.00th=[15008], 40.00th=[15533], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[24249], 99.50th=[26084], 99.90th=[26084], 99.95th=[26346],
     | 99.99th=[27395]
   bw (  KiB/s): min= 3768, max= 4224, per=25.11%, avg=3979.20, stdev=186.98, samples=5
   iops        : min=  942, max= 1056, avg=994.80, stdev=46.75, samples=5
  lat (msec)   : 10=0.12%, 20=96.13%, 50=3.75%
  cpu          : usr=0.70%, sys=4.88%, ctx=1112, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30154: Tue Jun  3 16:34:45 2025
  write: IOPS=991, BW=3964KiB/s (4060kB/s)(10.0MiB/2583msec)
    slat (usec): min=12, max=176, avg=36.88, stdev=18.24
    clat (usec): min=1727, max=32048, avg=16060.36, stdev=2379.74
     lat (usec): min=1793, max=32107, avg=16098.26, stdev=2381.28
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12780], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15008], 40.00th=[15533], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18482], 95.00th=[19792],
     | 99.00th=[24511], 99.50th=[26608], 99.90th=[29754], 99.95th=[29754],
     | 99.99th=[32113]
   bw (  KiB/s): min= 3792, max= 4256, per=25.14%, avg=3984.00, stdev=196.53, samples=5
   iops        : min=  948, max= 1064, avg=996.00, stdev=49.13, samples=5
  lat (msec)   : 2=0.08%, 4=0.12%, 10=0.04%, 20=95.55%, 50=4.22%
  cpu          : usr=0.58%, sys=4.96%, ctx=1178, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=15.5MiB/s (16.2MB/s), 3961KiB/s-3981KiB/s (4056kB/s-4077kB/s), io=40.0MiB (41.9MB), run=2572-2585msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30166: Tue Jun  3 16:34:48 2025
   read: IOPS=598, BW=2394KiB/s (2451kB/s)(5048KiB/2109msec)
    slat (usec): min=486, max=2083, avg=776.75, stdev=149.95
    clat (usec): min=1021, max=21346, avg=12409.78, stdev=1999.79
     lat (usec): min=1966, max=22552, avg=13188.25, stdev=2030.81
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[ 9503], 10.00th=[10028], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12256], 60.00th=[12780],
     | 70.00th=[13304], 80.00th=[13829], 90.00th=[14877], 95.00th=[15795],
     | 99.00th=[18482], 99.50th=[19792], 99.90th=[21103], 99.95th=[21365],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2227, max= 2488, per=25.13%, avg=2390.75, stdev=113.45, samples=4
   iops        : min=  556, max=  622, avg=597.50, stdev=28.72, samples=4
  write: IOPS=615, BW=2462KiB/s (2521kB/s)(5192KiB/2109msec)
    slat (usec): min=13, max=331, avg=39.49, stdev=23.15
    clat (usec): min=1957, max=21109, avg=12955.84, stdev=1695.26
     lat (usec): min=2003, max=21158, avg=12996.36, stdev=1695.94
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[10421], 10.00th=[10945], 20.00th=[11600],
     | 30.00th=[12125], 40.00th=[12518], 50.00th=[12911], 60.00th=[13304],
     | 70.00th=[13698], 80.00th=[14091], 90.00th=[15008], 95.00th=[15664],
     | 99.00th=[18220], 99.50th=[19268], 99.90th=[20841], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min= 2280, max= 2664, per=24.82%, avg=2458.75, stdev=168.85, samples=4
   iops        : min=  570, max=  666, avg=614.50, stdev=42.34, samples=4
  lat (msec)   : 2=0.12%, 10=5.66%, 20=93.95%, 50=0.27%
  cpu          : usr=0.90%, sys=9.39%, ctx=2238, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30167: Tue Jun  3 16:34:48 2025
   read: IOPS=589, BW=2359KiB/s (2416kB/s)(4976KiB/2109msec)
    slat (usec): min=446, max=1655, avg=778.51, stdev=145.42
    clat (usec): min=3145, max=22014, avg=12434.73, stdev=1968.94
     lat (usec): min=3913, max=23043, avg=13214.93, stdev=1998.46
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12387], 60.00th=[12780],
     | 70.00th=[13304], 80.00th=[13960], 90.00th=[15008], 95.00th=[15795],
     | 99.00th=[17171], 99.50th=[18744], 99.90th=[21890], 99.95th=[21890],
     | 99.99th=[21890]
   bw (  KiB/s): min= 2219, max= 2456, per=24.56%, avg=2336.75, stdev=96.98, samples=4
   iops        : min=  554, max=  614, avg=584.00, stdev=24.55, samples=4
  write: IOPS=623, BW=2496KiB/s (2556kB/s)(5264KiB/2109msec)
    slat (usec): min=13, max=572, avg=39.92, stdev=26.44
    clat (usec): min=2465, max=22206, avg=12981.02, stdev=1792.43
     lat (usec): min=2503, max=22262, avg=13021.95, stdev=1792.44
    clat percentiles (usec):
     |  1.00th=[ 9634],  5.00th=[10421], 10.00th=[10945], 20.00th=[11600],
     | 30.00th=[11994], 40.00th=[12518], 50.00th=[12911], 60.00th=[13304],
     | 70.00th=[13829], 80.00th=[14353], 90.00th=[15008], 95.00th=[15795],
     | 99.00th=[18744], 99.50th=[20055], 99.90th=[21627], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2360, max= 2640, per=25.29%, avg=2504.75, stdev=143.73, samples=4
   iops        : min=  590, max=  660, avg=626.00, stdev=36.11, samples=4
  lat (msec)   : 4=0.16%, 10=5.51%, 20=93.95%, 50=0.39%
  cpu          : usr=1.57%, sys=8.68%, ctx=2182, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30168: Tue Jun  3 16:34:48 2025
   read: IOPS=592, BW=2370KiB/s (2427kB/s)(4984KiB/2103msec)
    slat (usec): min=409, max=1453, avg=780.74, stdev=140.80
    clat (usec): min=1429, max=22458, avg=12443.46, stdev=1940.12
     lat (usec): min=2147, max=23735, avg=13225.80, stdev=1966.73
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[ 9634], 10.00th=[10159], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12387], 60.00th=[12911],
     | 70.00th=[13304], 80.00th=[13960], 90.00th=[14746], 95.00th=[15795],
     | 99.00th=[17695], 99.50th=[18744], 99.90th=[22414], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 2256, max= 2568, per=25.00%, avg=2378.75, stdev=134.30, samples=4
   iops        : min=  564, max=  642, avg=594.50, stdev=33.68, samples=4
  write: IOPS=624, BW=2499KiB/s (2559kB/s)(5256KiB/2103msec)
    slat (usec): min=12, max=315, avg=39.22, stdev=20.68
    clat (usec): min=8619, max=24953, avg=12887.24, stdev=1728.87
     lat (usec): min=8675, max=24994, avg=12927.53, stdev=1729.37
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[10421], 10.00th=[10945], 20.00th=[11469],
     | 30.00th=[11994], 40.00th=[12387], 50.00th=[12649], 60.00th=[13173],
     | 70.00th=[13566], 80.00th=[14091], 90.00th=[14877], 95.00th=[15795],
     | 99.00th=[18744], 99.50th=[20055], 99.90th=[22938], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 2323, max= 2632, per=25.10%, avg=2486.75, stdev=136.96, samples=4
   iops        : min=  580, max=  658, avg=621.50, stdev=34.54, samples=4
  lat (msec)   : 2=0.04%, 4=0.04%, 10=4.80%, 20=94.69%, 50=0.43%
  cpu          : usr=1.09%, sys=9.23%, ctx=2240, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30169: Tue Jun  3 16:34:48 2025
   read: IOPS=603, BW=2414KiB/s (2472kB/s)(5060KiB/2096msec)
    slat (usec): min=18, max=1739, avg=774.43, stdev=155.09
    clat (usec): min=1397, max=22102, avg=12428.30, stdev=2058.94
     lat (usec): min=1417, max=23110, avg=13204.30, stdev=2103.47
    clat percentiles (usec):
     |  1.00th=[ 8356],  5.00th=[ 9372], 10.00th=[10028], 20.00th=[10814],
     | 30.00th=[11338], 40.00th=[11863], 50.00th=[12387], 60.00th=[12911],
     | 70.00th=[13435], 80.00th=[13960], 90.00th=[14877], 95.00th=[15664],
     | 99.00th=[17957], 99.50th=[19006], 99.90th=[20579], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 2283, max= 2536, per=25.27%, avg=2404.75, stdev=125.94, samples=4
   iops        : min=  570, max=  634, avg=601.00, stdev=31.73, samples=4
  write: IOPS=617, BW=2471KiB/s (2531kB/s)(5180KiB/2096msec)
    slat (usec): min=12, max=1020, avg=40.00, stdev=36.69
    clat (usec): min=1967, max=26873, avg=12882.42, stdev=1774.21
     lat (usec): min=2011, max=26903, avg=12923.54, stdev=1775.59
    clat percentiles (usec):
     |  1.00th=[ 9372],  5.00th=[10421], 10.00th=[10945], 20.00th=[11600],
     | 30.00th=[11994], 40.00th=[12387], 50.00th=[12780], 60.00th=[13173],
     | 70.00th=[13435], 80.00th=[13960], 90.00th=[14877], 95.00th=[15664],
     | 99.00th=[18744], 99.50th=[19792], 99.90th=[26084], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 2392, max= 2616, per=25.00%, avg=2476.75, stdev=97.76, samples=4
   iops        : min=  598, max=  654, avg=619.00, stdev=24.47, samples=4
  lat (msec)   : 2=0.16%, 4=0.08%, 10=5.55%, 20=93.83%, 50=0.39%
  cpu          : usr=1.24%, sys=9.02%, ctx=2247, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9515KiB/s (9744kB/s), 2359KiB/s-2414KiB/s (2416kB/s-2472kB/s), io=19.6MiB (20.5MB), run=2096-2109msec
  WRITE: bw=9906KiB/s (10.1MB/s), 2462KiB/s-2499KiB/s (2521kB/s-2559kB/s), io=20.4MiB (21.4MB), run=2096-2109msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30175: Tue Jun  3 16:34:49 2025
   read: IOPS=10, BW=421MiB/s (442MB/s)(40.0MiB/95msec)
    slat (nsec): min=5457.7k, max=5457.7k, avg=5457672.00, stdev= 0.00
    clat (nsec): min=86302k, max=86302k, avg=86302345.00, stdev= 0.00
     lat (nsec): min=91773k, max=91773k, avg=91772863.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[86508],  5.00th=[86508], 10.00th=[86508], 20.00th=[86508],
     | 30.00th=[86508], 40.00th=[86508], 50.00th=[86508], 60.00th=[86508],
     | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508],
     | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=5.32%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=421MiB/s (442MB/s), 421MiB/s-421MiB/s (442MB/s-442MB/s), io=40.0MiB (41.9MB), run=95-95msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck