== sanity test 398c: run fio to test AIO ================= 09:32:30 (1740493950)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0995505 s, 421 MB/s
osc.lustre-OST0000-osc-ffff880079a8b800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=28825: Tue Feb 25 09:32:35 2025
  write: IOPS=656, BW=2627KiB/s (2690kB/s)(10.0MiB/3898msec)
    slat (usec): min=14, max=339, avg=52.16, stdev=27.17
    clat (usec): min=2591, max=33398, avg=24171.98, stdev=2699.08
     lat (usec): min=2652, max=33439, avg=24225.56, stdev=2700.36
    clat percentiles (usec):
     |  1.00th=[16450],  5.00th=[19792], 10.00th=[20841], 20.00th=[22152],
     | 30.00th=[23200], 40.00th=[23725], 50.00th=[24249], 60.00th=[25035],
     | 70.00th=[25297], 80.00th=[26084], 90.00th=[27132], 95.00th=[28443],
     | 99.00th=[30540], 99.50th=[31589], 99.90th=[33162], 99.95th=[33424],
     | 99.99th=[33424]
   bw (  KiB/s): min= 2440, max= 2816, per=24.81%, avg=2606.86, stdev=129.94, samples=7
   iops        : min=  610, max=  704, avg=651.71, stdev=32.48, samples=7
  lat (msec)   : 4=0.04%, 10=0.04%, 20=5.39%, 50=94.53%
  cpu          : usr=0.80%, sys=3.59%, ctx=1496, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28826: Tue Feb 25 09:32:35 2025
  write: IOPS=659, BW=2640KiB/s (2703kB/s)(10.0MiB/3879msec)
    slat (usec): min=12, max=509, avg=57.68, stdev=32.66
    clat (usec): min=3072, max=42382, avg=24090.48, stdev=3033.76
     lat (usec): min=3205, max=42427, avg=24149.57, stdev=3034.39
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[19530], 10.00th=[20841], 20.00th=[22152],
     | 30.00th=[22938], 40.00th=[23725], 50.00th=[24249], 60.00th=[25035],
     | 70.00th=[25297], 80.00th=[26084], 90.00th=[27132], 95.00th=[28705],
     | 99.00th=[30802], 99.50th=[31589], 99.90th=[34866], 99.95th=[40109],
     | 99.99th=[42206]
   bw (  KiB/s): min= 2506, max= 2888, per=25.03%, avg=2630.00, stdev=132.81, samples=7
   iops        : min=  626, max=  722, avg=657.43, stdev=33.28, samples=7
  lat (msec)   : 4=0.08%, 10=0.43%, 20=5.27%, 50=94.22%
  cpu          : usr=0.77%, sys=3.97%, ctx=1919, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28827: Tue Feb 25 09:32:35 2025
  write: IOPS=657, BW=2631KiB/s (2694kB/s)(10.0MiB/3892msec)
    slat (usec): min=14, max=391, avg=59.92, stdev=32.61
    clat (usec): min=3822, max=37729, avg=24145.50, stdev=2957.14
     lat (usec): min=3900, max=37796, avg=24206.95, stdev=2957.97
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[19530], 10.00th=[20841], 20.00th=[22152],
     | 30.00th=[23200], 40.00th=[23725], 50.00th=[24511], 60.00th=[25035],
     | 70.00th=[25297], 80.00th=[26084], 90.00th=[27132], 95.00th=[28443],
     | 99.00th=[31065], 99.50th=[32637], 99.90th=[37487], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 2512, max= 2816, per=24.97%, avg=2624.00, stdev=117.76, samples=7
   iops        : min=  628, max=  704, avg=656.00, stdev=29.44, samples=7
  lat (msec)   : 4=0.04%, 10=0.12%, 20=5.82%, 50=94.02%
  cpu          : usr=0.87%, sys=3.86%, ctx=2205, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=28828: Tue Feb 25 09:32:35 2025
  write: IOPS=657, BW=2628KiB/s (2691kB/s)(10.0MiB/3896msec)
    slat (usec): min=13, max=305, avg=52.12, stdev=27.91
    clat (usec): min=2391, max=40338, avg=24193.96, stdev=2901.20
     lat (usec): min=2444, max=40365, avg=24247.46, stdev=2901.00
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[19792], 10.00th=[20841], 20.00th=[22152],
     | 30.00th=[23200], 40.00th=[23725], 50.00th=[24511], 60.00th=[25035],
     | 70.00th=[25297], 80.00th=[26084], 90.00th=[27132], 95.00th=[29230],
     | 99.00th=[31065], 99.50th=[31327], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2464, max= 2824, per=24.95%, avg=2621.71, stdev=126.30, samples=7
   iops        : min=  616, max=  706, avg=655.43, stdev=31.57, samples=7
  lat (msec)   : 4=0.12%, 10=0.04%, 20=5.12%, 50=94.73%
  cpu          : usr=0.92%, sys=3.57%, ctx=1597, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.3MiB/s (10.8MB/s), 2627KiB/s-2640KiB/s (2690kB/s-2703kB/s), io=40.0MiB (41.9MB), run=3879-3898msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=28841: Tue Feb 25 09:32:40 2025
   read: IOPS=364, BW=1459KiB/s (1494kB/s)(5048KiB/3461msec)
    slat (usec): min=833, max=3221, avg=1384.19, stdev=301.02
    clat (usec): min=2788, max=33103, avg=20490.56, stdev=3167.44
     lat (usec): min=4504, max=34220, avg=21876.79, stdev=3200.16
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[15533], 10.00th=[16712], 20.00th=[17957],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[22152], 80.00th=[23200], 90.00th=[24511], 95.00th=[25560],
     | 99.00th=[28181], 99.50th=[28705], 99.90th=[32900], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1285, max= 1576, per=24.78%, avg=1436.83, stdev=109.07, samples=6
   iops        : min=  321, max=  394, avg=359.17, stdev=27.34, samples=6
  write: IOPS=375, BW=1500KiB/s (1536kB/s)(5192KiB/3461msec)
    slat (usec): min=15, max=1653, avg=53.07, stdev=57.49
    clat (usec): min=2731, max=29130, avg=21005.32, stdev=2585.86
     lat (usec): min=2778, max=29155, avg=21059.84, stdev=2584.61
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17171], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19530], 40.00th=[20317], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[22414], 80.00th=[23200], 90.00th=[24249], 95.00th=[25297],
     | 99.00th=[27132], 99.50th=[28181], 99.90th=[28967], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1301, max= 1632, per=24.89%, avg=1502.17, stdev=127.28, samples=6
   iops        : min=  325, max=  408, avg=375.50, stdev=31.90, samples=6
  lat (msec)   : 4=0.08%, 10=0.23%, 20=39.57%, 50=60.12%
  cpu          : usr=0.81%, sys=6.82%, ctx=2386, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28842: Tue Feb 25 09:32:40 2025
   read: IOPS=360, BW=1441KiB/s (1476kB/s)(4976KiB/3453msec)
    slat (usec): min=792, max=3340, avg=1372.88, stdev=282.13
    clat (usec): min=8345, max=32694, avg=20431.44, stdev=2933.32
     lat (usec): min=10453, max=33741, avg=21806.32, stdev=2964.22
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[15795], 10.00th=[16712], 20.00th=[17957],
     | 30.00th=[18744], 40.00th=[19530], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[21890], 80.00th=[22938], 90.00th=[24249], 95.00th=[25297],
     | 99.00th=[27657], 99.50th=[28443], 99.90th=[31851], 99.95th=[32637],
     | 99.99th=[32637]
   bw (  KiB/s): min= 1189, max= 1600, per=24.53%, avg=1422.17, stdev=155.64, samples=6
   iops        : min=  297, max=  400, avg=355.50, stdev=38.99, samples=6
  write: IOPS=381, BW=1524KiB/s (1561kB/s)(5264KiB/3453msec)
    slat (usec): min=15, max=347, avg=51.71, stdev=28.93
    clat (usec): min=11406, max=30686, avg=21050.70, stdev=2500.68
     lat (usec): min=11544, max=30742, avg=21104.01, stdev=2499.81
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[17171], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[22152], 80.00th=[22938], 90.00th=[24249], 95.00th=[25297],
     | 99.00th=[27919], 99.50th=[28181], 99.90th=[29754], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1464, max= 1632, per=25.39%, avg=1532.83, stdev=60.34, samples=6
   iops        : min=  366, max=  408, avg=383.17, stdev=15.13, samples=6
  lat (msec)   : 10=0.04%, 20=38.95%, 50=61.02%
  cpu          : usr=0.70%, sys=7.01%, ctx=2348, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28843: Tue Feb 25 09:32:40 2025
   read: IOPS=360, BW=1442KiB/s (1477kB/s)(4984KiB/3456msec)
    slat (usec): min=832, max=3085, avg=1386.73, stdev=287.93
    clat (usec): min=5666, max=34307, avg=20459.08, stdev=3031.63
     lat (usec): min=7416, max=35870, avg=21847.78, stdev=3046.73
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15795], 10.00th=[16909], 20.00th=[17957],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[21890], 80.00th=[22938], 90.00th=[24249], 95.00th=[25560],
     | 99.00th=[27919], 99.50th=[29230], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1288, max= 1592, per=24.74%, avg=1434.17, stdev=124.17, samples=6
   iops        : min=  322, max=  398, avg=358.50, stdev=31.07, samples=6
  write: IOPS=380, BW=1521KiB/s (1557kB/s)(5256KiB/3456msec)
    slat (usec): min=13, max=1811, avg=53.49, stdev=58.76
    clat (usec): min=7306, max=37226, avg=21074.15, stdev=2805.86
     lat (usec): min=7360, max=37268, avg=21129.16, stdev=2805.11
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[16909], 10.00th=[17957], 20.00th=[18744],
     | 30.00th=[19530], 40.00th=[20317], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[22152], 80.00th=[23200], 90.00th=[24511], 95.00th=[25560],
     | 99.00th=[28443], 99.50th=[29230], 99.90th=[36963], 99.95th=[37487],
     | 99.99th=[37487]
   bw (  KiB/s): min= 1373, max= 1688, per=25.31%, avg=1527.50, stdev=125.70, samples=6
   iops        : min=  343, max=  422, avg=381.83, stdev=31.49, samples=6
  lat (msec)   : 10=0.16%, 20=41.13%, 50=58.71%
  cpu          : usr=0.96%, sys=6.89%, ctx=2423, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=28844: Tue Feb 25 09:32:40 2025
   read: IOPS=367, BW=1469KiB/s (1504kB/s)(5060KiB/3444msec)
    slat (usec): min=25, max=3619, avg=1377.36, stdev=316.33
    clat (usec): min=2098, max=35324, avg=20437.79, stdev=3257.36
     lat (usec): min=2127, max=36518, avg=21817.12, stdev=3304.34
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15664], 10.00th=[16712], 20.00th=[17957],
     | 30.00th=[18744], 40.00th=[19530], 50.00th=[20317], 60.00th=[21103],
     | 70.00th=[21890], 80.00th=[22938], 90.00th=[24249], 95.00th=[25560],
     | 99.00th=[28443], 99.50th=[30802], 99.90th=[34866], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 1416, max= 1560, per=25.36%, avg=1470.17, stdev=57.08, samples=6
   iops        : min=  354, max=  390, avg=367.50, stdev=14.31, samples=6
  write: IOPS=376, BW=1504KiB/s (1540kB/s)(5180KiB/3444msec)
    slat (usec): min=14, max=492, avg=51.56, stdev=29.72
    clat (usec): min=3280, max=36618, avg=21094.66, stdev=2866.82
     lat (usec): min=3340, max=36650, avg=21147.62, stdev=2867.64
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[17171], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[21103], 60.00th=[21627],
     | 70.00th=[22414], 80.00th=[23200], 90.00th=[24511], 95.00th=[25297],
     | 99.00th=[29230], 99.50th=[30016], 99.90th=[33162], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min= 1405, max= 1608, per=25.02%, avg=1510.17, stdev=88.55, samples=6
   iops        : min=  351, max=  402, avg=377.50, stdev=22.20, samples=6
  lat (msec)   : 4=0.16%, 10=0.35%, 20=38.87%, 50=60.62%
  cpu          : usr=1.05%, sys=6.74%, ctx=2336, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5798KiB/s (5937kB/s), 1441KiB/s-1469KiB/s (1476kB/s-1504kB/s), io=19.6MiB (20.5MB), run=3444-3461msec
  WRITE: bw=6036KiB/s (6181kB/s), 1500KiB/s-1524KiB/s (1536kB/s-1561kB/s), io=20.4MiB (21.4MB), run=3444-3461msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=28850: Tue Feb 25 09:32:41 2025
   read: IOPS=10, BW=440MiB/s (461MB/s)(40.0MiB/91msec)
    slat (nsec): min=6189.2k, max=6189.2k, avg=6189229.00, stdev= 0.00
    clat (nsec): min=80302k, max=80302k, avg=80301954.00, stdev= 0.00
     lat (nsec): min=86510k, max=86510k, avg=86509926.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[80217],  5.00th=[80217], 10.00th=[80217], 20.00th=[80217],
     | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[80217],
     | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217],
     | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.67%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=440MiB/s (461MB/s), 440MiB/s-440MiB/s (461MB/s-461MB/s), io=40.0MiB (41.9MB), run=91-91msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout