== sanity test 398c: run fio to test AIO ================= 10:23:07 (1724768587) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0921887 s, 455 MB/s osc.lustre-OST0000-osc-ffff88012ab10800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24362: Tue Aug 27 10:23:37 2024 write: IOPS=90, BW=363KiB/s (371kB/s)(10.0MiB/28247msec) slat (usec): min=11, max=494, avg=81.25, stdev=49.80 clat (msec): min=17, max=328, avg=176.37, stdev=39.26 lat (msec): min=17, max=329, avg=176.45, stdev=39.26 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 118], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 330] bw ( KiB/s): min= 255, max= 648, per=25.11%, avg=362.04, stdev=62.58, samples=56 iops : min= 63, max= 162, avg=90.43, stdev=15.65, samples=56 lat (msec) : 20=0.04%, 50=0.43%, 100=1.29%, 250=94.02%, 500=4.22% cpu : usr=0.21%, sys=0.71%, ctx=2826, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24363: Tue Aug 27 10:23:37 2024 write: IOPS=90, BW=361KiB/s (369kB/s)(10.0MiB/28399msec) slat (usec): min=13, max=481, avg=84.59, stdev=49.75 clat (msec): min=61, max=303, avg=177.28, stdev=37.91 lat (msec): min=61, max=303, avg=177.37, stdev=37.92 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 121], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 248, max= 520, per=24.96%, avg=359.98, stdev=57.47, samples=56 iops : min= 62, max= 130, avg=89.98, stdev=14.37, samples=56 lat (msec) : 100=1.76%, 250=94.38%, 500=3.87% cpu : usr=0.17%, sys=0.77%, ctx=2866, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24364: Tue Aug 27 10:23:37 2024 write: IOPS=90, BW=361KiB/s (369kB/s)(10.0MiB/28399msec) slat (usec): min=12, max=429, avg=82.20, stdev=49.53 clat (msec): min=61, max=304, avg=177.28, stdev=38.02 lat (msec): min=61, max=304, avg=177.36, stdev=38.03 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 122], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 305] bw ( KiB/s): min= 264, max= 528, per=24.92%, avg=359.41, stdev=56.85, samples=56 iops : min= 66, max= 132, avg=89.84, stdev=14.22, samples=56 lat (msec) : 100=1.76%, 250=94.34%, 500=3.91% cpu : usr=0.19%, sys=0.73%, ctx=2773, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24365: Tue Aug 27 10:23:37 2024 write: IOPS=90, BW=361KiB/s (369kB/s)(10.0MiB/28399msec) slat (usec): min=12, max=501, avg=81.20, stdev=50.18 clat (msec): min=44, max=302, avg=177.26, stdev=38.09 lat (msec): min=44, max=302, avg=177.34, stdev=38.10 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 122], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 245], | 99.00th=[ 275], 99.50th=[ 279], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 528, per=24.95%, avg=359.82, stdev=62.80, samples=56 iops : min= 64, max= 132, avg=89.93, stdev=15.73, samples=56 lat (msec) : 50=0.04%, 100=1.45%, 250=94.14%, 500=4.38% cpu : usr=0.18%, sys=0.72%, ctx=2798, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1442KiB/s (1477kB/s), 361KiB/s-363KiB/s (369kB/s-371kB/s), io=40.0MiB (41.9MB), run=28247-28399msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24397: Tue Aug 27 10:23:55 2024 read: IOPS=71, BW=285KiB/s (292kB/s)(5048KiB/17710msec) slat (usec): min=11, max=6040, avg=1294.90, stdev=368.95 clat (usec): min=1239, max=170840, avg=73470.46, stdev=16534.80 lat (usec): min=1940, max=172275, avg=74767.37, stdev=16592.00 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 92], 95.00th=[ 97], | 99.00th=[ 120], 99.50th=[ 125], 99.90th=[ 169], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 168, max= 408, per=25.45%, avg=285.83, stdev=53.91, samples=35 iops : min= 42, max= 102, avg=71.37, stdev=13.47, samples=35 write: IOPS=73, BW=293KiB/s (300kB/s)(5192KiB/17710msec) slat (usec): min=12, max=271, avg=65.65, stdev=36.32 clat (msec): min=15, max=277, avg=145.36, stdev=39.24 lat (msec): min=15, max=277, avg=145.43, stdev=39.25 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 89], 10.00th=[ 100], 20.00th=[ 112], | 30.00th=[ 125], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 253], 99.50th=[ 264], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 224, max= 368, per=25.10%, avg=293.37, stdev=40.60, samples=35 iops : min= 56, max= 92, avg=73.26, stdev=10.12, samples=35 lat (msec) : 2=0.16%, 4=0.23%, 10=0.08%, 20=0.39%, 50=1.29% lat (msec) : 100=50.94%, 250=46.37%, 500=0.55% cpu : usr=0.24%, sys=1.51%, ctx=3672, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24398: Tue Aug 27 10:23:55 2024 read: IOPS=69, BW=279KiB/s (286kB/s)(4976KiB/17832msec) slat (usec): min=728, max=3237, avg=1303.50, stdev=321.96 clat (usec): min=1673, max=141644, avg=74546.30, stdev=14507.36 lat (msec): min=3, max=143, avg=75.85, stdev=14.53 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 93], 95.00th=[ 97], | 99.00th=[ 109], 99.50th=[ 116], 99.90th=[ 125], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 176, max= 392, per=24.81%, avg=278.57, stdev=52.64, samples=35 iops : min= 44, max= 98, avg=69.60, stdev=13.12, samples=35 write: IOPS=73, BW=295KiB/s (302kB/s)(5264KiB/17832msec) slat (usec): min=13, max=253, avg=68.07, stdev=35.67 clat (msec): min=59, max=277, avg=144.79, stdev=38.72 lat (msec): min=59, max=277, avg=144.86, stdev=38.72 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 89], 10.00th=[ 99], 20.00th=[ 112], | 30.00th=[ 123], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 201], 95.00th=[ 222], | 99.00th=[ 249], 99.50th=[ 257], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 208, max= 408, per=25.20%, avg=294.57, stdev=46.94, samples=35 iops : min= 52, max= 102, avg=73.60, stdev=11.73, samples=35 lat (msec) : 2=0.04%, 20=0.31%, 50=1.17%, 100=51.09%, 250=47.07% lat (msec) : 500=0.31% cpu : usr=0.24%, sys=1.49%, ctx=3794, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24399: Tue Aug 27 10:23:55 2024 read: IOPS=69, BW=279KiB/s (286kB/s)(4984KiB/17857msec) slat (usec): min=711, max=3652, avg=1285.77, stdev=307.77 clat (usec): min=1702, max=121832, avg=74050.11, stdev=13968.33 lat (msec): min=3, max=123, avg=75.34, stdev=14.02 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 92], 95.00th=[ 96], | 99.00th=[ 106], 99.50th=[ 108], 99.90th=[ 121], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 176, max= 400, per=24.84%, avg=279.00, stdev=62.01, samples=35 iops : min= 44, max= 100, avg=69.69, stdev=15.45, samples=35 write: IOPS=73, BW=294KiB/s (301kB/s)(5256KiB/17857msec) slat (usec): min=13, max=571, avg=68.70, stdev=41.60 clat (msec): min=37, max=277, avg=145.63, stdev=37.40 lat (msec): min=37, max=277, avg=145.70, stdev=37.41 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 90], 10.00th=[ 101], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 245], 99.50th=[ 253], 99.90th=[ 262], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 208, max= 352, per=24.96%, avg=291.80, stdev=37.07, samples=35 iops : min= 52, max= 88, avg=72.89, stdev= 9.25, samples=35 lat (msec) : 2=0.04%, 10=0.08%, 20=0.35%, 50=0.98%, 100=51.37% lat (msec) : 250=46.88%, 500=0.31% cpu : usr=0.24%, sys=1.51%, ctx=3554, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24400: Tue Aug 27 10:23:55 2024 read: IOPS=70, BW=284KiB/s (291kB/s)(5060KiB/17819msec) slat (usec): min=735, max=5716, avg=1301.68, stdev=345.65 clat (msec): min=4, max=142, avg=74.37, stdev=14.46 lat (msec): min=5, max=143, avg=75.67, stdev=14.47 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 92], 95.00th=[ 97], | 99.00th=[ 113], 99.50th=[ 122], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 152, max= 416, per=25.34%, avg=284.51, stdev=56.95, samples=35 iops : min= 38, max= 104, avg=71.09, stdev=14.24, samples=35 write: IOPS=72, BW=291KiB/s (298kB/s)(5180KiB/17819msec) slat (usec): min=14, max=1796, avg=69.11, stdev=60.47 clat (msec): min=58, max=281, avg=145.88, stdev=37.91 lat (msec): min=58, max=281, avg=145.95, stdev=37.92 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 113], | 30.00th=[ 125], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 251], 99.50th=[ 255], 99.90th=[ 275], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 184, max= 384, per=24.75%, avg=289.31, stdev=39.83, samples=35 iops : min= 46, max= 96, avg=72.29, stdev= 9.92, samples=35 lat (msec) : 10=0.08%, 20=0.16%, 50=1.37%, 100=51.60%, 250=46.29% lat (msec) : 500=0.51% cpu : usr=0.24%, sys=1.52%, ctx=3917, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1124KiB/s (1151kB/s), 279KiB/s-285KiB/s (286kB/s-292kB/s), io=19.6MiB (20.5MB), run=17710-17857msec WRITE: bw=1170KiB/s (1198kB/s), 291KiB/s-295KiB/s (298kB/s-302kB/s), io=20.4MiB (21.4MB), run=17710-17857msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24418: Tue Aug 27 10:23:57 2024 read: IOPS=11, BW=455MiB/s (477MB/s)(40.0MiB/88msec) slat (nsec): min=6063.2k, max=6063.2k, avg=6063174.00, stdev= 0.00 clat (nsec): min=76612k, max=76612k, avg=76611532.00, stdev= 0.00 lat (nsec): min=82689k, max=82689k, avg=82688963.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[77071], 5.00th=[77071], 10.00th=[77071], 20.00th=[77071], | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[77071], | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=6.90%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=40.0MiB (41.9MB), run=88-88msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout