== sanity test 398c: run fio to test AIO ================= 19:37:08 (1736987828)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0831586 s, 504 MB/s
osc.lustre-OST0000-osc-ffff88007d0c7000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=18996: Wed Jan 15 19:37:12 2025
  write: IOPS=913, BW=3653KiB/s (3741kB/s)(10.0MiB/2803msec)
    slat (usec): min=11, max=353, avg=38.96, stdev=21.95
    clat (usec): min=9895, max=28470, avg=17432.22, stdev=3416.46
     lat (usec): min=9927, max=28509, avg=17472.19, stdev=3414.90
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13566], 10.00th=[14091], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[18220], 80.00th=[19792], 90.00th=[23200], 95.00th=[25035],
     | 99.00th=[26608], 99.50th=[27657], 99.90th=[28443], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 2632, max= 4192, per=24.54%, avg=3585.60, stdev=659.63, samples=5
   iops        : min=  658, max= 1048, avg=896.40, stdev=164.91, samples=5
  lat (msec)   : 10=0.04%, 20=80.74%, 50=19.22%
  cpu          : usr=0.54%, sys=4.28%, ctx=1347, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18997: Wed Jan 15 19:37:12 2025
  write: IOPS=915, BW=3661KiB/s (3749kB/s)(10.0MiB/2797msec)
    slat (usec): min=11, max=411, avg=41.62, stdev=24.72
    clat (usec): min=2468, max=30881, avg=17409.30, stdev=3467.96
     lat (usec): min=2519, max=30909, avg=17452.02, stdev=3466.03
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13698], 10.00th=[14222], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[18220], 80.00th=[19792], 90.00th=[23200], 95.00th=[24773],
     | 99.00th=[26608], 99.50th=[26870], 99.90th=[29230], 99.95th=[29230],
     | 99.99th=[30802]
   bw (  KiB/s): min= 2688, max= 4216, per=24.66%, avg=3603.20, stdev=627.88, samples=5
   iops        : min=  672, max= 1054, avg=900.80, stdev=156.97, samples=5
  lat (msec)   : 4=0.04%, 10=0.47%, 20=79.77%, 50=19.73%
  cpu          : usr=1.07%, sys=3.97%, ctx=1570, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18998: Wed Jan 15 19:37:12 2025
  write: IOPS=913, BW=3655KiB/s (3742kB/s)(10.0MiB/2802msec)
    slat (usec): min=12, max=440, avg=41.32, stdev=24.09
    clat (usec): min=2279, max=28434, avg=17432.05, stdev=3459.22
     lat (usec): min=2335, max=28511, avg=17474.41, stdev=3457.73
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13566], 10.00th=[14222], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[18220], 80.00th=[19792], 90.00th=[23200], 95.00th=[25035],
     | 99.00th=[26346], 99.50th=[26870], 99.90th=[28443], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 2656, max= 4216, per=24.59%, avg=3593.60, stdev=648.43, samples=5
   iops        : min=  664, max= 1054, avg=898.40, stdev=162.11, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=80.62%, 50=19.18%
  cpu          : usr=0.61%, sys=4.39%, ctx=1583, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=18999: Wed Jan 15 19:37:12 2025
  write: IOPS=913, BW=3655KiB/s (3742kB/s)(10.0MiB/2802msec)
    slat (usec): min=12, max=213, avg=41.77, stdev=22.71
    clat (usec): min=3349, max=28423, avg=17417.21, stdev=3386.79
     lat (usec): min=3413, max=28472, avg=17460.02, stdev=3384.64
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13698], 10.00th=[14222], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[15795], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[18220], 80.00th=[19792], 90.00th=[23462], 95.00th=[24773],
     | 99.00th=[26084], 99.50th=[26608], 99.90th=[27657], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 2664, max= 4192, per=24.60%, avg=3595.20, stdev=641.31, samples=5
   iops        : min=  666, max= 1048, avg=898.80, stdev=160.33, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=80.90%, 50=19.02%
  cpu          : usr=0.96%, sys=4.03%, ctx=1634, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=14.3MiB/s (14.0MB/s), 3653KiB/s-3661KiB/s (3741kB/s-3749kB/s), io=40.0MiB (41.9MB), run=2797-2803msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19011: Wed Jan 15 19:37:15 2025
   read: IOPS=622, BW=2492KiB/s (2551kB/s)(5048KiB/2026msec)
    slat (usec): min=437, max=1327, avg=791.51, stdev=132.10
    clat (usec): min=3769, max=17901, avg=11929.07, stdev=1687.01
     lat (usec): min=4405, max=18814, avg=12722.45, stdev=1701.48
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[11076], 40.00th=[11469], 50.00th=[11863], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13304], 90.00th=[14091], 95.00th=[14746],
     | 99.00th=[15795], 99.50th=[16057], 99.90th=[16450], 99.95th=[17957],
     | 99.99th=[17957]
   bw (  KiB/s): min= 2352, max= 2632, per=25.04%, avg=2480.00, stdev=137.48, samples=4
   iops        : min=  588, max=  658, avg=620.00, stdev=34.37, samples=4
  write: IOPS=640, BW=2563KiB/s (2624kB/s)(5192KiB/2026msec)
    slat (usec): min=12, max=769, avg=44.67, stdev=40.99
    clat (usec): min=3729, max=16984, avg=12414.72, stdev=1380.72
     lat (usec): min=3762, max=17003, avg=12460.58, stdev=1379.62
    clat percentiles (usec):
     |  1.00th=[ 9241],  5.00th=[10290], 10.00th=[10683], 20.00th=[11338],
     | 30.00th=[11731], 40.00th=[12125], 50.00th=[12387], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13566], 90.00th=[14091], 95.00th=[14746],
     | 99.00th=[15795], 99.50th=[16450], 99.90th=[16909], 99.95th=[16909],
     | 99.99th=[16909]
   bw (  KiB/s): min= 2512, max= 2600, per=24.83%, avg=2560.00, stdev=39.73, samples=4
   iops        : min=  628, max=  650, avg=640.00, stdev= 9.93, samples=4
  lat (msec)   : 4=0.08%, 10=7.58%, 20=92.34%
  cpu          : usr=0.94%, sys=10.47%, ctx=2210, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19012: Wed Jan 15 19:37:15 2025
   read: IOPS=614, BW=2460KiB/s (2519kB/s)(4976KiB/2023msec)
    slat (usec): min=470, max=1690, avg=788.87, stdev=139.37
    clat (usec): min=4769, max=26209, avg=11980.72, stdev=1869.15
     lat (usec): min=5617, max=26854, avg=12771.45, stdev=1892.08
    clat percentiles (usec):
     |  1.00th=[ 8291],  5.00th=[ 9110], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11994], 60.00th=[12387],
     | 70.00th=[12780], 80.00th=[13435], 90.00th=[14353], 95.00th=[15008],
     | 99.00th=[16188], 99.50th=[17171], 99.90th=[23200], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 2344, max= 2624, per=24.82%, avg=2458.00, stdev=129.22, samples=4
   iops        : min=  586, max=  656, avg=614.50, stdev=32.31, samples=4
  write: IOPS=650, BW=2602KiB/s (2665kB/s)(5264KiB/2023msec)
    slat (usec): min=12, max=511, avg=44.54, stdev=29.40
    clat (usec): min=7476, max=24140, avg=12362.07, stdev=1477.22
     lat (usec): min=7518, max=24171, avg=12407.81, stdev=1476.82
    clat percentiles (usec):
     |  1.00th=[ 8979],  5.00th=[10159], 10.00th=[10552], 20.00th=[11076],
     | 30.00th=[11600], 40.00th=[11994], 50.00th=[12387], 60.00th=[12780],
     | 70.00th=[13042], 80.00th=[13435], 90.00th=[14091], 95.00th=[14746],
     | 99.00th=[15795], 99.50th=[16712], 99.90th=[21103], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 2552, max= 2696, per=25.29%, avg=2608.00, stdev=61.97, samples=4
   iops        : min=  638, max=  674, avg=652.00, stdev=15.49, samples=4
  lat (msec)   : 10=8.52%, 20=91.25%, 50=0.23%
  cpu          : usr=1.19%, sys=10.29%, ctx=2162, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19013: Wed Jan 15 19:37:15 2025
   read: IOPS=616, BW=2466KiB/s (2525kB/s)(4984KiB/2021msec)
    slat (usec): min=14, max=1656, avg=784.51, stdev=153.09
    clat (usec): min=3484, max=25315, avg=12031.81, stdev=1966.33
     lat (usec): min=3551, max=26103, avg=12818.17, stdev=1993.45
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 9110], 10.00th=[ 9765], 20.00th=[10552],
     | 30.00th=[11076], 40.00th=[11600], 50.00th=[11994], 60.00th=[12518],
     | 70.00th=[13042], 80.00th=[13566], 90.00th=[14353], 95.00th=[15139],
     | 99.00th=[16319], 99.50th=[16909], 99.90th=[24511], 99.95th=[25297],
     | 99.99th=[25297]
   bw (  KiB/s): min= 2432, max= 2496, per=24.98%, avg=2474.00, stdev=28.75, samples=4
   iops        : min=  608, max=  624, avg=618.50, stdev= 7.19, samples=4
  write: IOPS=650, BW=2601KiB/s (2663kB/s)(5256KiB/2021msec)
    slat (usec): min=13, max=665, avg=43.62, stdev=30.67
    clat (usec): min=7378, max=23089, avg=12332.56, stdev=1435.46
     lat (usec): min=7409, max=23141, avg=12377.30, stdev=1435.66
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10159], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12649],
     | 70.00th=[13042], 80.00th=[13435], 90.00th=[14091], 95.00th=[14615],
     | 99.00th=[16057], 99.50th=[17171], 99.90th=[21627], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 2424, max= 2768, per=25.20%, avg=2598.00, stdev=171.25, samples=4
   iops        : min=  606, max=  692, avg=649.50, stdev=42.81, samples=4
  lat (msec)   : 4=0.12%, 10=8.24%, 20=91.37%, 50=0.27%
  cpu          : usr=1.39%, sys=9.90%, ctx=2162, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19014: Wed Jan 15 19:37:15 2025
   read: IOPS=629, BW=2517KiB/s (2578kB/s)(5060KiB/2010msec)
    slat (usec): min=41, max=1435, avg=781.03, stdev=146.42
    clat (usec): min=1253, max=22898, avg=11914.34, stdev=1890.79
     lat (usec): min=1349, max=23671, avg=12697.10, stdev=1917.30
    clat percentiles (usec):
     |  1.00th=[ 7570],  5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10421],
     | 30.00th=[10945], 40.00th=[11469], 50.00th=[11863], 60.00th=[12256],
     | 70.00th=[12780], 80.00th=[13304], 90.00th=[14091], 95.00th=[14877],
     | 99.00th=[16450], 99.50th=[16909], 99.90th=[19268], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 2416, max= 2608, per=25.38%, avg=2514.00, stdev=92.46, samples=4
   iops        : min=  604, max=  652, avg=628.50, stdev=23.12, samples=4
  write: IOPS=644, BW=2577KiB/s (2639kB/s)(5180KiB/2010msec)
    slat (usec): min=13, max=361, avg=43.42, stdev=22.94
    clat (usec): min=3256, max=23878, avg=12307.52, stdev=1553.20
     lat (usec): min=3301, max=24007, avg=12352.08, stdev=1553.73
    clat percentiles (usec):
     |  1.00th=[ 8586],  5.00th=[10159], 10.00th=[10683], 20.00th=[11207],
     | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12518],
     | 70.00th=[13042], 80.00th=[13435], 90.00th=[14091], 95.00th=[14746],
     | 99.00th=[15533], 99.50th=[16188], 99.90th=[23725], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min= 2480, max= 2616, per=24.98%, avg=2576.00, stdev=64.33, samples=4
   iops        : min=  620, max=  654, avg=644.00, stdev=16.08, samples=4
  lat (msec)   : 2=0.16%, 4=0.08%, 10=8.24%, 20=91.37%, 50=0.16%
  cpu          : usr=1.34%, sys=10.10%, ctx=2235, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=9905KiB/s (10.1MB/s), 2460KiB/s-2517KiB/s (2519kB/s-2578kB/s), io=19.6MiB (20.5MB), run=2010-2026msec
  WRITE: bw=10.1MiB/s (10.6MB/s), 2563KiB/s-2602KiB/s (2624kB/s-2665kB/s), io=20.4MiB (21.4MB), run=2010-2026msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=19020: Wed Jan 15 19:37:16 2025
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=6782.8k, max=6782.8k, avg=6782772.00, stdev= 0.00
    clat (nsec): min=65042k, max=65042k, avg=65041838.00, stdev= 0.00
     lat (nsec): min=71837k, max=71837k, avg=71837357.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[65274],  5.00th=[65274], 10.00th=[65274], 20.00th=[65274],
     | 30.00th=[65274], 40.00th=[65274], 50.00th=[65274], 60.00th=[65274],
     | 70.00th=[65274], 80.00th=[65274], 90.00th=[65274], 95.00th=[65274],
     | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274],
     | 99.99th=[65274]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.67%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck