== sanity test 398c: run fio to test AIO ================= 08:09:48 (1712146188) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.119822 s, 350 MB/s osc.lustre-OST0000-osc-ffff8801309df000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=17129: Wed Apr 3 08:09:57 2024 write: IOPS=388, BW=1552KiB/s (1590kB/s)(10.0MiB/6596msec) slat (usec): min=17, max=475, avg=82.61, stdev=43.55 clat (usec): min=3920, max=80493, avg=41039.31, stdev=6668.60 lat (usec): min=3959, max=80585, avg=41124.06, stdev=6671.74 clat percentiles (usec): | 1.00th=[15664], 5.00th=[26346], 10.00th=[33162], 20.00th=[37487], | 30.00th=[39584], 40.00th=[41157], 50.00th=[42730], 60.00th=[44303], | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47449], | 99.00th=[49546], 99.50th=[50070], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 1368, max= 1960, per=25.07%, avg=1551.08, stdev=158.85, samples=13 iops : min= 342, max= 490, avg=387.92, stdev=39.73, samples=13 lat (msec) : 4=0.04%, 10=0.39%, 20=1.05%, 50=98.01%, 100=0.51% cpu : usr=0.76%, sys=3.76%, ctx=1437, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17130: Wed Apr 3 08:09:57 2024 write: IOPS=386, BW=1547KiB/s (1584kB/s)(10.0MiB/6619msec) slat (usec): min=14, max=563, avg=82.32, stdev=41.25 clat (usec): min=13929, max=70390, avg=41167.05, stdev=6023.29 lat (usec): min=13978, max=70493, avg=41251.39, stdev=6030.02 clat percentiles (usec): | 1.00th=[21103], 5.00th=[28705], 10.00th=[33817], 20.00th=[37487], | 30.00th=[39584], 40.00th=[41157], 50.00th=[42730], 60.00th=[44303], | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47449], | 99.00th=[49021], 99.50th=[49546], 99.90th=[53740], 99.95th=[65799], | 99.99th=[70779] bw ( KiB/s): min= 1400, max= 1920, per=24.99%, avg=1546.46, stdev=135.74, samples=13 iops : min= 350, max= 480, avg=386.62, stdev=33.94, samples=13 lat (msec) : 20=0.78%, 50=98.79%, 100=0.43% cpu : usr=0.74%, sys=3.75%, ctx=1506, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17131: Wed Apr 3 08:09:57 2024 write: IOPS=386, BW=1547KiB/s (1584kB/s)(10.0MiB/6619msec) slat (usec): min=18, max=584, avg=83.87, stdev=42.92 clat (usec): min=14178, max=74882, avg=41170.13, stdev=6060.26 lat (usec): min=14270, max=74933, avg=41256.07, stdev=6066.75 clat percentiles (usec): | 1.00th=[20317], 5.00th=[28705], 10.00th=[33424], 20.00th=[38011], | 30.00th=[39584], 40.00th=[41157], 50.00th=[42730], 60.00th=[44303], | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47449], | 99.00th=[49546], 99.50th=[49546], 99.90th=[57934], 99.95th=[62653], | 99.99th=[74974] bw ( KiB/s): min= 1392, max= 1920, per=25.00%, avg=1546.77, stdev=137.07, samples=13 iops : min= 348, max= 480, avg=386.69, stdev=34.27, samples=13 lat (msec) : 20=0.94%, 50=98.75%, 100=0.31% cpu : usr=0.74%, sys=3.81%, ctx=1663, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17132: Wed Apr 3 08:09:57 2024 write: IOPS=386, BW=1548KiB/s (1585kB/s)(10.0MiB/6615msec) slat (usec): min=16, max=818, avg=86.59, stdev=50.85 clat (usec): min=8832, max=79829, avg=41137.53, stdev=6253.56 lat (usec): min=8937, max=79860, avg=41226.23, stdev=6254.12 clat percentiles (usec): | 1.00th=[19006], 5.00th=[28443], 10.00th=[33162], 20.00th=[37487], | 30.00th=[39584], 40.00th=[41157], 50.00th=[43254], 60.00th=[44303], | 70.00th=[44827], 80.00th=[45351], 90.00th=[46400], 95.00th=[47449], | 99.00th=[49546], 99.50th=[50070], 99.90th=[65799], 99.95th=[70779], | 99.99th=[80217] bw ( KiB/s): min= 1400, max= 1920, per=25.01%, avg=1547.38, stdev=137.54, samples=13 iops : min= 350, max= 480, avg=386.85, stdev=34.38, samples=13 lat (msec) : 10=0.04%, 20=1.09%, 50=98.32%, 100=0.55% cpu : usr=0.60%, sys=4.04%, ctx=1691, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6188KiB/s (6337kB/s), 1547KiB/s-1552KiB/s (1584kB/s-1590kB/s), io=40.0MiB (41.9MB), run=6596-6619msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17147: Wed Apr 3 08:10:00 2024 read: IOPS=475, BW=1901KiB/s (1946kB/s)(5048KiB/2656msec) slat (usec): min=666, max=3293, avg=1146.77, stdev=319.48 clat (usec): min=7377, max=41637, avg=15509.76, stdev=3656.35 lat (usec): min=8283, max=44936, avg=16658.30, stdev=3845.85 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11863], 10.00th=[12387], 20.00th=[13042], | 30.00th=[13698], 40.00th=[14353], 50.00th=[15008], 60.00th=[15664], | 70.00th=[16188], 80.00th=[16909], 90.00th=[18220], 95.00th=[19268], | 99.00th=[33424], 99.50th=[36963], 99.90th=[39584], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 1376, max= 2040, per=24.79%, avg=1872.60, stdev=280.75, samples=5 iops : min= 344, max= 510, avg=468.00, stdev=70.10, samples=5 write: IOPS=488, BW=1955KiB/s (2002kB/s)(5192KiB/2656msec) slat (usec): min=14, max=271, avg=42.57, stdev=22.15 clat (usec): min=7339, max=41156, avg=16137.02, stdev=3805.60 lat (usec): min=7371, max=41335, avg=16180.63, stdev=3810.90 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12911], 10.00th=[13435], 20.00th=[14222], | 30.00th=[14615], 40.00th=[15139], 50.00th=[15533], 60.00th=[15926], | 70.00th=[16450], 80.00th=[16909], 90.00th=[17957], 95.00th=[19530], | 99.00th=[36439], 99.50th=[38536], 99.90th=[40633], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 1448, max= 2131, per=24.72%, avg=1944.60, stdev=280.32, samples=5 iops : min= 362, max= 532, avg=486.00, stdev=69.96, samples=5 lat (msec) : 10=0.23%, 20=95.35%, 50=4.41% cpu : usr=0.60%, sys=8.51%, ctx=2139, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17148: Wed Apr 3 08:10:00 2024 read: IOPS=469, BW=1878KiB/s (1924kB/s)(4976KiB/2649msec) slat (usec): min=631, max=4714, avg=1151.39, stdev=326.12 clat (usec): min=9718, max=42805, avg=15508.87, stdev=3809.04 lat (usec): min=10602, max=44882, avg=16662.24, stdev=3998.51 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11863], 10.00th=[12387], 20.00th=[13173], | 30.00th=[13829], 40.00th=[14222], 50.00th=[14877], 60.00th=[15401], | 70.00th=[16057], 80.00th=[16909], 90.00th=[17957], 95.00th=[19530], | 99.00th=[34341], 99.50th=[36439], 99.90th=[40109], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 1312, max= 2035, per=24.51%, avg=1851.80, stdev=303.69, samples=5 iops : min= 328, max= 508, avg=462.80, stdev=75.81, samples=5 write: IOPS=496, BW=1987KiB/s (2035kB/s)(5264KiB/2649msec) slat (usec): min=13, max=997, avg=43.23, stdev=33.37 clat (usec): min=4681, max=41051, avg=16177.70, stdev=3774.92 lat (usec): min=4707, max=41112, avg=16221.94, stdev=3780.71 clat percentiles (usec): | 1.00th=[12125], 5.00th=[12911], 10.00th=[13435], 20.00th=[14222], | 30.00th=[14615], 40.00th=[15139], 50.00th=[15533], 60.00th=[15795], | 70.00th=[16450], 80.00th=[17171], 90.00th=[18220], 95.00th=[20579], | 99.00th=[35390], 99.50th=[36963], 99.90th=[40109], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 1600, max= 2155, per=25.38%, avg=1995.80, stdev=233.38, samples=5 iops : min= 400, max= 538, avg=498.80, stdev=58.22, samples=5 lat (msec) : 10=0.12%, 20=94.88%, 50=5.00% cpu : usr=0.94%, sys=8.19%, ctx=2165, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17149: Wed Apr 3 08:10:00 2024 read: IOPS=469, BW=1879KiB/s (1924kB/s)(4984KiB/2653msec) slat (usec): min=644, max=3978, avg=1149.82, stdev=341.11 clat (usec): min=9129, max=40107, avg=15543.53, stdev=3485.64 lat (usec): min=10008, max=42880, avg=16695.21, stdev=3686.36 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11863], 10.00th=[12518], 20.00th=[13173], | 30.00th=[13829], 40.00th=[14484], 50.00th=[15008], 60.00th=[15533], | 70.00th=[16057], 80.00th=[16909], 90.00th=[18220], 95.00th=[20317], | 99.00th=[32113], 99.50th=[33817], 99.90th=[38536], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1440, max= 2123, per=24.77%, avg=1871.00, stdev=257.19, samples=5 iops : min= 360, max= 530, avg=467.60, stdev=64.12, samples=5 write: IOPS=495, BW=1981KiB/s (2029kB/s)(5256KiB/2653msec) slat (usec): min=14, max=180, avg=43.06, stdev=20.35 clat (usec): min=7090, max=42194, avg=16189.21, stdev=3913.90 lat (usec): min=7144, max=42248, avg=16233.23, stdev=3919.00 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12649], 10.00th=[13435], 20.00th=[14222], | 30.00th=[14615], 40.00th=[15139], 50.00th=[15533], 60.00th=[15926], | 70.00th=[16319], 80.00th=[16909], 90.00th=[18220], 95.00th=[21103], | 99.00th=[36439], 99.50th=[38536], 99.90th=[41157], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 1448, max= 2128, per=24.81%, avg=1951.20, stdev=287.98, samples=5 iops : min= 362, max= 532, avg=487.80, stdev=71.99, samples=5 lat (msec) : 10=0.20%, 20=94.26%, 50=5.55% cpu : usr=0.72%, sys=8.45%, ctx=2121, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17150: Wed Apr 3 08:10:00 2024 read: IOPS=478, BW=1913KiB/s (1959kB/s)(5060KiB/2645msec) slat (usec): min=14, max=4026, avg=1132.83, stdev=343.51 clat (usec): min=2776, max=45296, avg=15460.38, stdev=3818.95 lat (usec): min=2792, max=48734, avg=16594.95, stdev=4018.04 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11731], 10.00th=[12387], 20.00th=[13173], | 30.00th=[13829], 40.00th=[14353], 50.00th=[14877], 60.00th=[15401], | 70.00th=[16057], 80.00th=[16909], 90.00th=[18220], 95.00th=[19792], | 99.00th=[33817], 99.50th=[38011], 99.90th=[43254], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 1504, max= 2056, per=25.13%, avg=1898.40, stdev=225.35, samples=5 iops : min= 376, max= 514, avg=474.60, stdev=56.34, samples=5 write: IOPS=489, BW=1958KiB/s (2005kB/s)(5180KiB/2645msec) slat (usec): min=14, max=1308, avg=45.76, stdev=52.34 clat (usec): min=10340, max=41373, avg=16328.48, stdev=3961.05 lat (usec): min=10371, max=41457, avg=16375.15, stdev=3965.60 clat percentiles (usec): | 1.00th=[11600], 5.00th=[12780], 10.00th=[13435], 20.00th=[14091], | 30.00th=[14615], 40.00th=[15139], 50.00th=[15664], 60.00th=[16057], | 70.00th=[16581], 80.00th=[17433], 90.00th=[18482], 95.00th=[21627], | 99.00th=[35390], 99.50th=[36963], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 1480, max= 2171, per=24.72%, avg=1944.60, stdev=267.59, samples=5 iops : min= 370, max= 542, avg=486.00, stdev=66.74, samples=5 lat (msec) : 4=0.12%, 10=0.27%, 20=94.49%, 50=5.12% cpu : usr=0.91%, sys=8.36%, ctx=2132, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7556KiB/s (7737kB/s), 1878KiB/s-1913KiB/s (1924kB/s-1959kB/s), io=19.6MiB (20.5MB), run=2645-2656msec WRITE: bw=7866KiB/s (8055kB/s), 1955KiB/s-1987KiB/s (2002kB/s-2035kB/s), io=20.4MiB (21.4MB), run=2645-2656msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=17156: Wed Apr 3 08:10:02 2024 read: IOPS=5, BW=214MiB/s (224MB/s)(40.0MiB/187msec) slat (nsec): min=6727.6k, max=6727.6k, avg=6727560.00, stdev= 0.00 clat (nsec): min=174705k, max=174705k, avg=174704683.00, stdev= 0.00 lat (nsec): min=181444k, max=181444k, avg=181444354.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.23%, ctx=8, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=214MiB/s (224MB/s), 214MiB/s-214MiB/s (224MB/s-224MB/s), io=40.0MiB (41.9MB), run=187-187msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout