== sanity test 398c: run fio to test AIO ================= 01:09:26 (1718860166)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.125503 s, 334 MB/s
osc.lustre-OST0000-osc-ffff88012a5ad800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=9293: Thu Jun 20 01:09:34 2024
  write: IOPS=369, BW=1477KiB/s (1513kB/s)(10.0MiB/6931msec)
    slat (usec): min=17, max=712, avg=95.40, stdev=49.10
    clat (usec): min=5689, max=80869, avg=43135.73, stdev=5384.10
     lat (usec): min=5796, max=80985, avg=43233.55, stdev=5390.91
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[31065], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[71828], 99.95th=[76022],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1360, max= 1748, per=25.04%, avg=1478.46, stdev=110.07, samples=13
   iops        : min=  340, max=  437, avg=369.62, stdev=27.52, samples=13
  lat (msec)   : 10=0.08%, 20=0.35%, 50=98.52%, 100=1.05%
  cpu          : usr=0.72%, sys=3.97%, ctx=2010, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9294: Thu Jun 20 01:09:34 2024
  write: IOPS=369, BW=1477KiB/s (1513kB/s)(10.0MiB/6931msec)
    slat (usec): min=15, max=688, avg=95.06, stdev=51.78
    clat (usec): min=10600, max=76450, avg=43135.86, stdev=5216.83
     lat (usec): min=10705, max=76700, avg=43233.31, stdev=5227.26
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[32113], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[50594], 99.50th=[51643], 99.90th=[59507], 99.95th=[71828],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1376, max= 1784, per=25.03%, avg=1478.15, stdev=109.36, samples=13
   iops        : min=  344, max=  446, avg=369.54, stdev=27.34, samples=13
  lat (msec)   : 20=0.55%, 50=98.32%, 100=1.13%
  cpu          : usr=0.81%, sys=3.71%, ctx=2085, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9295: Thu Jun 20 01:09:34 2024
  write: IOPS=369, BW=1476KiB/s (1512kB/s)(10.0MiB/6936msec)
    slat (usec): min=14, max=639, avg=94.38, stdev=51.32
    clat (usec): min=17590, max=71808, avg=43167.61, stdev=5151.87
     lat (usec): min=17704, max=71928, avg=43264.60, stdev=5158.21
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[33424], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[50594], 99.50th=[52167], 99.90th=[64226], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1368, max= 1760, per=25.00%, avg=1476.31, stdev=109.57, samples=13
   iops        : min=  342, max=  440, avg=369.08, stdev=27.39, samples=13
  lat (msec)   : 20=0.43%, 50=98.28%, 100=1.29%
  cpu          : usr=0.74%, sys=3.72%, ctx=1917, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=9296: Thu Jun 20 01:09:34 2024
  write: IOPS=369, BW=1476KiB/s (1512kB/s)(10.0MiB/6936msec)
    slat (usec): min=19, max=796, avg=97.90, stdev=52.45
    clat (usec): min=10058, max=71820, avg=43161.97, stdev=5218.15
     lat (usec): min=10132, max=71964, avg=43262.37, stdev=5224.53
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[31851], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[44303], 60.00th=[44827],
     | 70.00th=[45351], 80.00th=[46400], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[51119], 99.50th=[51643], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1384, max= 1736, per=25.02%, avg=1477.54, stdev=101.53, samples=13
   iops        : min=  346, max=  434, avg=369.38, stdev=25.38, samples=13
  lat (msec)   : 20=0.39%, 50=98.12%, 100=1.48%
  cpu          : usr=0.58%, sys=4.01%, ctx=2236, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5905KiB/s (6047kB/s), 1476KiB/s-1477KiB/s (1512kB/s-1513kB/s), io=40.0MiB (41.9MB), run=6931-6936msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=9312: Thu Jun 20 01:09:38 2024
   read: IOPS=465, BW=1863KiB/s (1907kB/s)(5048KiB/2710msec)
    slat (usec): min=643, max=3256, avg=1104.09, stdev=220.30
    clat (usec): min=2115, max=28732, avg=15941.74, stdev=2325.25
     lat (usec): min=3156, max=29978, avg=17047.71, stdev=2325.88
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12649], 10.00th=[13304], 20.00th=[14091],
     | 30.00th=[14746], 40.00th=[15270], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[21890], 99.50th=[24773], 99.90th=[28443], 99.95th=[28705],
     | 99.99th=[28705]
   bw (  KiB/s): min= 1656, max= 1896, per=24.80%, avg=1836.80, stdev=102.23, samples=5
   iops        : min=  414, max=  474, avg=459.20, stdev=25.56, samples=5
  write: IOPS=478, BW=1916KiB/s (1962kB/s)(5192KiB/2710msec)
    slat (usec): min=13, max=1294, avg=48.70, stdev=53.05
    clat (usec): min=2074, max=27149, avg=16539.55, stdev=1851.57
     lat (usec): min=2108, max=27178, avg=16589.48, stdev=1850.56
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13698], 10.00th=[14484], 20.00th=[15139],
     | 30.00th=[15664], 40.00th=[16057], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[20841], 99.50th=[22152], 99.90th=[26870], 99.95th=[27132],
     | 99.99th=[27132]
   bw (  KiB/s): min= 1784, max= 2040, per=24.72%, avg=1905.60, stdev=94.89, samples=5
   iops        : min=  446, max=  510, avg=476.40, stdev=23.72, samples=5
  lat (msec)   : 4=0.16%, 10=0.16%, 20=96.80%, 50=2.89%
  cpu          : usr=0.92%, sys=7.24%, ctx=2895, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9313: Thu Jun 20 01:09:38 2024
   read: IOPS=459, BW=1839KiB/s (1883kB/s)(4976KiB/2706msec)
    slat (usec): min=17, max=2696, avg=1103.72, stdev=228.62
    clat (usec): min=2527, max=30707, avg=16051.87, stdev=2449.69
     lat (usec): min=2685, max=31935, avg=17157.37, stdev=2449.69
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12649], 10.00th=[13304], 20.00th=[14222],
     | 30.00th=[14746], 40.00th=[15270], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18744], 95.00th=[20055],
     | 99.00th=[22676], 99.50th=[27132], 99.90th=[30278], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1664, max= 1948, per=24.51%, avg=1815.20, stdev=101.76, samples=5
   iops        : min=  416, max=  487, avg=453.80, stdev=25.44, samples=5
  write: IOPS=486, BW=1945KiB/s (1992kB/s)(5264KiB/2706msec)
    slat (usec): min=14, max=1323, avg=48.94, stdev=51.45
    clat (usec): min=7588, max=29596, avg=16531.58, stdev=1937.56
     lat (usec): min=7611, max=29663, avg=16581.65, stdev=1936.04
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13829], 10.00th=[14353], 20.00th=[15139],
     | 30.00th=[15533], 40.00th=[16057], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[22414], 99.50th=[25822], 99.90th=[28967], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1860, max= 2080, per=25.35%, avg=1954.40, stdev=95.73, samples=5
   iops        : min=  465, max=  520, avg=488.60, stdev=23.93, samples=5
  lat (msec)   : 4=0.08%, 10=0.16%, 20=95.74%, 50=4.02%
  cpu          : usr=1.03%, sys=7.13%, ctx=2980, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9314: Thu Jun 20 01:09:38 2024
   read: IOPS=460, BW=1842KiB/s (1886kB/s)(4984KiB/2706msec)
    slat (usec): min=653, max=3010, avg=1109.17, stdev=222.53
    clat (usec): min=10053, max=27597, avg=15959.91, stdev=2148.51
     lat (usec): min=11231, max=28792, avg=17070.78, stdev=2156.74
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12649], 10.00th=[13304], 20.00th=[14091],
     | 30.00th=[14746], 40.00th=[15270], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17695], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[21103], 99.50th=[21627], 99.90th=[26346], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1704, max= 2048, per=24.82%, avg=1837.60, stdev=128.67, samples=5
   iops        : min=  426, max=  512, avg=459.40, stdev=32.17, samples=5
  write: IOPS=485, BW=1942KiB/s (1989kB/s)(5256KiB/2706msec)
    slat (usec): min=15, max=325, avg=46.36, stdev=24.35
    clat (usec): min=11834, max=26112, avg=16496.06, stdev=1914.73
     lat (usec): min=11900, max=26167, avg=16543.51, stdev=1914.22
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13566], 10.00th=[14222], 20.00th=[14877],
     | 30.00th=[15533], 40.00th=[15926], 50.00th=[16450], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[18744], 95.00th=[19268],
     | 99.00th=[24249], 99.50th=[24773], 99.90th=[26084], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1752, max= 2024, per=24.98%, avg=1925.60, stdev=107.17, samples=5
   iops        : min=  438, max=  506, avg=481.40, stdev=26.79, samples=5
  lat (msec)   : 20=96.33%, 50=3.67%
  cpu          : usr=0.92%, sys=7.13%, ctx=2889, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=9315: Thu Jun 20 01:09:38 2024
   read: IOPS=467, BW=1870KiB/s (1915kB/s)(5060KiB/2706msec)
    slat (usec): min=686, max=2631, avg=1109.31, stdev=221.99
    clat (usec): min=4374, max=27422, avg=15931.50, stdev=2159.94
     lat (usec): min=5320, max=28558, avg=17042.62, stdev=2169.89
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[12780], 10.00th=[13435], 20.00th=[14091],
     | 30.00th=[14746], 40.00th=[15270], 50.00th=[15795], 60.00th=[16450],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[21365], 99.50th=[22676], 99.90th=[27132], 99.95th=[27395],
     | 99.99th=[27395]
   bw (  KiB/s): min= 1720, max= 1912, per=25.14%, avg=1861.60, stdev=79.98, samples=5
   iops        : min=  430, max=  478, avg=465.40, stdev=19.99, samples=5
  write: IOPS=478, BW=1914KiB/s (1960kB/s)(5180KiB/2706msec)
    slat (usec): min=14, max=1133, avg=47.65, stdev=39.20
    clat (usec): min=11624, max=29363, avg=16515.73, stdev=1953.64
     lat (usec): min=11660, max=29444, avg=16564.51, stdev=1953.50
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[13829], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15533], 40.00th=[16057], 50.00th=[16319], 60.00th=[16909],
     | 70.00th=[17171], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530],
     | 99.00th=[24249], 99.50th=[26084], 99.90th=[27657], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1768, max= 2072, per=24.71%, avg=1904.80, stdev=108.75, samples=5
   iops        : min=  442, max=  518, avg=476.20, stdev=27.19, samples=5
  lat (msec)   : 10=0.12%, 20=96.88%, 50=3.01%
  cpu          : usr=1.07%, sys=7.06%, ctx=2958, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7405KiB/s (7583kB/s), 1839KiB/s-1870KiB/s (1883kB/s-1915kB/s), io=19.6MiB (20.5MB), run=2706-2710msec
  WRITE: bw=7709KiB/s (7894kB/s), 1914KiB/s-1945KiB/s (1960kB/s-1992kB/s), io=20.4MiB (21.4MB), run=2706-2710msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=9320: Thu Jun 20 01:09:39 2024
   read: IOPS=7, BW=292MiB/s (306MB/s)(40.0MiB/137msec)
    slat (nsec): min=5892.7k, max=5892.7k, avg=5892655.00, stdev= 0.00
    clat (nsec): min=126315k, max=126315k, avg=126314896.00, stdev= 0.00
     lat (nsec): min=132218k, max=132218k, avg=132218393.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  127],  5.00th=[  127], 10.00th=[  127], 20.00th=[  127],
     | 30.00th=[  127], 40.00th=[  127], 50.00th=[  127], 60.00th=[  127],
     | 70.00th=[  127], 80.00th=[  127], 90.00th=[  127], 95.00th=[  127],
     | 99.00th=[  127], 99.50th=[  127], 99.90th=[  127], 99.95th=[  127],
     | 99.99th=[  127]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.41%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=40.0MiB (41.9MB), run=137-137msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout