== sanity test 398c: run fio to test AIO ================= 00:44:55 (1717562695) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.088395 s, 474 MB/s osc.lustre-OST0000-osc-ffff8801361fb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=25871: Wed Jun 5 00:45:36 2024 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39584msec) slat (usec): min=11, max=1361, avg=78.86, stdev=73.38 clat (msec): min=80, max=545, avg=247.14, stdev=83.21 lat (msec): min=80, max=545, avg=247.22, stdev=83.22 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 131], 10.00th=[ 148], 20.00th=[ 171], | 30.00th=[ 192], 40.00th=[ 213], 50.00th=[ 236], 60.00th=[ 262], | 70.00th=[ 288], 80.00th=[ 321], 90.00th=[ 368], 95.00th=[ 397], | 99.00th=[ 464], 99.50th=[ 485], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 550] bw ( KiB/s): min= 136, max= 512, per=24.91%, avg=257.62, stdev=81.08, samples=79 iops : min= 34, max= 128, avg=64.33, stdev=20.32, samples=79 lat (msec) : 100=0.39%, 250=54.26%, 500=44.96%, 750=0.39% cpu : usr=0.15%, sys=0.65%, ctx=2178, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25872: Wed Jun 5 00:45:36 2024 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39527msec) slat (usec): min=10, max=1004, avg=79.58, stdev=70.35 clat (msec): min=23, max=548, avg=246.84, stdev=85.46 lat (msec): min=23, max=549, avg=246.92, stdev=85.46 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 171], | 30.00th=[ 190], 40.00th=[ 211], 50.00th=[ 234], 60.00th=[ 264], | 70.00th=[ 292], 80.00th=[ 326], 90.00th=[ 368], 95.00th=[ 397], | 99.00th=[ 464], 99.50th=[ 498], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 128, max= 512, per=24.90%, avg=257.51, stdev=86.85, samples=79 iops : min= 32, max= 128, avg=64.22, stdev=21.76, samples=79 lat (msec) : 50=0.35%, 100=0.62%, 250=54.73%, 500=43.83%, 750=0.47% cpu : usr=0.16%, sys=0.60%, ctx=2314, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25873: Wed Jun 5 00:45:36 2024 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39584msec) slat (usec): min=11, max=955, avg=77.79, stdev=69.41 clat (msec): min=78, max=549, avg=247.15, stdev=83.75 lat (msec): min=78, max=549, avg=247.23, stdev=83.75 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 132], 10.00th=[ 148], 20.00th=[ 171], | 30.00th=[ 192], 40.00th=[ 211], 50.00th=[ 236], 60.00th=[ 264], | 70.00th=[ 288], 80.00th=[ 326], 90.00th=[ 363], 95.00th=[ 393], | 99.00th=[ 464], 99.50th=[ 498], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 550] bw ( KiB/s): min= 128, max= 488, per=24.87%, avg=257.20, stdev=81.76, samples=79 iops : min= 32, max= 122, avg=64.14, stdev=20.48, samples=79 lat (msec) : 100=0.35%, 250=54.57%, 500=44.69%, 750=0.39% cpu : usr=0.13%, sys=0.63%, ctx=2210, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25874: Wed Jun 5 00:45:36 2024 write: IOPS=64, BW=259KiB/s (265kB/s)(10.0MiB/39527msec) slat (usec): min=12, max=1782, avg=81.35, stdev=78.03 clat (msec): min=23, max=545, avg=246.85, stdev=84.48 lat (msec): min=23, max=545, avg=246.93, stdev=84.49 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 128], 10.00th=[ 146], 20.00th=[ 171], | 30.00th=[ 190], 40.00th=[ 213], 50.00th=[ 236], 60.00th=[ 262], | 70.00th=[ 292], 80.00th=[ 321], 90.00th=[ 368], 95.00th=[ 397], | 99.00th=[ 460], 99.50th=[ 481], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 550] bw ( KiB/s): min= 152, max= 496, per=24.91%, avg=257.62, stdev=83.86, samples=79 iops : min= 38, max= 124, avg=64.33, stdev=20.96, samples=79 lat (msec) : 50=0.23%, 100=0.43%, 250=54.80%, 500=44.18%, 750=0.35% cpu : usr=0.14%, sys=0.67%, ctx=2310, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1035KiB/s (1060kB/s), 259KiB/s-259KiB/s (265kB/s-265kB/s), io=40.0MiB (41.9MB), run=39527-39584msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25916: Wed Jun 5 00:45:59 2024 read: IOPS=55, BW=224KiB/s (229kB/s)(5048KiB/22555msec) slat (usec): min=14, max=4724, avg=1162.49, stdev=539.79 clat (usec): min=1484, max=285661, avg=92997.86, stdev=28986.19 lat (usec): min=1668, max=287540, avg=94162.62, stdev=29230.71 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 95], | 70.00th=[ 103], 80.00th=[ 113], 90.00th=[ 127], 95.00th=[ 144], | 99.00th=[ 190], 99.50th=[ 205], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 104, max= 384, per=25.13%, avg=223.40, stdev=58.12, samples=45 iops : min= 26, max= 96, avg=55.80, stdev=14.53, samples=45 write: IOPS=57, BW=230KiB/s (236kB/s)(5192KiB/22555msec) slat (usec): min=13, max=562, avg=72.66, stdev=52.83 clat (msec): min=31, max=384, avg=186.22, stdev=58.92 lat (msec): min=31, max=384, avg=186.30, stdev=58.93 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 138], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 271], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 120, max= 344, per=24.72%, avg=228.91, stdev=53.81, samples=45 iops : min= 30, max= 86, avg=57.18, stdev=13.47, samples=45 lat (msec) : 2=0.08%, 20=0.12%, 50=1.17%, 100=33.83%, 250=57.77% lat (msec) : 500=7.03% cpu : usr=0.32%, sys=1.35%, ctx=3252, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25917: Wed Jun 5 00:45:59 2024 read: IOPS=55, BW=221KiB/s (226kB/s)(4976KiB/22554msec) slat (usec): min=13, max=4866, avg=1159.76, stdev=523.17 clat (msec): min=2, max=285, avg=93.10, stdev=29.34 lat (msec): min=3, max=286, avg=94.27, stdev=29.55 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 113], 90.00th=[ 130], 95.00th=[ 146], | 99.00th=[ 182], 99.50th=[ 203], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 88, max= 352, per=24.71%, avg=219.67, stdev=62.92, samples=45 iops : min= 22, max= 88, avg=54.87, stdev=15.69, samples=45 write: IOPS=58, BW=233KiB/s (239kB/s)(5264KiB/22554msec) slat (usec): min=11, max=749, avg=70.64, stdev=51.51 clat (msec): min=15, max=373, avg=184.90, stdev=58.01 lat (msec): min=15, max=373, avg=184.97, stdev=58.01 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 104], 10.00th=[ 116], 20.00th=[ 138], | 30.00th=[ 153], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 226], 90.00th=[ 264], 95.00th=[ 296], | 99.00th=[ 342], 99.50th=[ 355], 99.90th=[ 359], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 112, max= 360, per=25.12%, avg=232.64, stdev=54.33, samples=45 iops : min= 28, max= 90, avg=58.11, stdev=13.62, samples=45 lat (msec) : 4=0.08%, 10=0.04%, 20=0.51%, 50=1.05%, 100=33.24% lat (msec) : 250=58.44%, 500=6.64% cpu : usr=0.30%, sys=1.37%, ctx=3113, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25918: Wed Jun 5 00:45:59 2024 read: IOPS=55, BW=222KiB/s (227kB/s)(4984KiB/22498msec) slat (usec): min=14, max=5224, avg=1165.59, stdev=559.74 clat (usec): min=1877, max=300992, avg=92423.82, stdev=28455.16 lat (msec): min=2, max=303, avg=93.59, stdev=28.69 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 113], 90.00th=[ 130], 95.00th=[ 148], | 99.00th=[ 176], 99.50th=[ 184], 99.90th=[ 257], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 136, max= 344, per=25.21%, avg=224.11, stdev=57.52, samples=44 iops : min= 34, max= 86, avg=55.98, stdev=14.38, samples=44 write: IOPS=58, BW=234KiB/s (239kB/s)(5256KiB/22498msec) slat (usec): min=12, max=525, avg=72.97, stdev=54.26 clat (msec): min=15, max=383, avg=184.92, stdev=58.11 lat (msec): min=15, max=383, avg=185.00, stdev=58.11 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 105], 10.00th=[ 118], 20.00th=[ 138], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 264], 95.00th=[ 305], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 95, max= 376, per=25.24%, avg=233.75, stdev=52.75, samples=44 iops : min= 23, max= 94, avg=58.39, stdev=13.21, samples=44 lat (msec) : 2=0.04%, 4=0.04%, 20=0.39%, 50=1.45%, 100=33.75% lat (msec) : 250=57.77%, 500=6.56% cpu : usr=0.28%, sys=1.40%, ctx=3352, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25919: Wed Jun 5 00:45:59 2024 read: IOPS=56, BW=225KiB/s (231kB/s)(5060KiB/22459msec) slat (usec): min=12, max=3942, avg=1153.05, stdev=504.66 clat (usec): min=1754, max=323159, avg=92650.74, stdev=30372.74 lat (msec): min=2, max=326, avg=93.81, stdev=30.57 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 89], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 113], 90.00th=[ 127], 95.00th=[ 144], | 99.00th=[ 186], 99.50th=[ 234], 99.90th=[ 321], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 56, max= 400, per=25.64%, avg=227.93, stdev=64.14, samples=44 iops : min= 14, max= 100, avg=56.93, stdev=16.02, samples=44 write: IOPS=57, BW=231KiB/s (236kB/s)(5180KiB/22459msec) slat (usec): min=11, max=953, avg=74.75, stdev=58.55 clat (msec): min=15, max=392, avg=185.57, stdev=59.03 lat (msec): min=15, max=392, avg=185.65, stdev=59.03 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 105], 10.00th=[ 117], 20.00th=[ 138], | 30.00th=[ 153], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 226], 90.00th=[ 268], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 127, max= 368, per=24.99%, avg=231.39, stdev=50.43, samples=44 iops : min= 31, max= 92, avg=57.80, stdev=12.63, samples=44 lat (msec) : 2=0.04%, 4=0.04%, 20=0.47%, 50=1.33%, 100=34.84% lat (msec) : 250=56.09%, 500=7.19% cpu : usr=0.21%, sys=1.45%, ctx=3437, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=890KiB/s (911kB/s), 221KiB/s-225KiB/s (226kB/s-231kB/s), io=19.6MiB (20.5MB), run=22459-22555msec WRITE: bw=926KiB/s (948kB/s), 230KiB/s-234KiB/s (236kB/s-239kB/s), io=20.4MiB (21.4MB), run=22459-22555msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=25941: Wed Jun 5 00:46:00 2024 read: IOPS=7, BW=294MiB/s (308MB/s)(40.0MiB/136msec) slat (nsec): min=7497.9k, max=7497.9k, avg=7497852.00, stdev= 0.00 clat (nsec): min=127719k, max=127719k, avg=127718807.00, stdev= 0.00 lat (nsec): min=135240k, max=135240k, avg=135239544.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.44%, ctx=13, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=40.0MiB (41.9MB), run=136-136msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout