== sanity test 398c: run fio to test AIO ================= 22:40:41 (1746153641) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.293648 s, 143 MB/s osc.lustre-OST0000-osc-ffff8800aa9bd800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=21649: Thu May 1 22:40:57 2025 write: IOPS=182, BW=729KiB/s (747kB/s)(10.0MiB/14039msec) slat (usec): min=11, max=3943, avg=103.41, stdev=251.86 clat (msec): min=8, max=168, avg=87.29, stdev=18.19 lat (msec): min=9, max=168, avg=87.40, stdev=18.20 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 77], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 91], | 70.00th=[ 96], 80.00th=[ 103], 90.00th=[ 111], 95.00th=[ 120], | 99.00th=[ 134], 99.50th=[ 155], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 169] bw ( KiB/s): min= 550, max= 896, per=24.89%, avg=726.11, stdev=90.78, samples=28 iops : min= 137, max= 224, avg=181.29, stdev=22.73, samples=28 lat (msec) : 10=0.04%, 50=0.82%, 100=75.86%, 250=23.28% cpu : usr=0.36%, sys=2.64%, ctx=964, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21650: Thu May 1 22:40:57 2025 write: IOPS=183, BW=732KiB/s (750kB/s)(10.0MiB/13986msec) slat (usec): min=11, max=3932, avg=106.73, stdev=254.16 clat (msec): min=46, max=158, avg=86.93, stdev=17.81 lat (msec): min=46, max=159, avg=87.04, stdev=17.81 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 77], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 91], | 70.00th=[ 95], 80.00th=[ 102], 90.00th=[ 112], 95.00th=[ 117], | 99.00th=[ 136], 99.50th=[ 150], 99.90th=[ 155], 99.95th=[ 157], | 99.99th=[ 159] bw ( KiB/s): min= 544, max= 912, per=25.05%, avg=730.63, stdev=93.46, samples=27 iops : min= 136, max= 228, avg=182.52, stdev=23.39, samples=27 lat (msec) : 50=0.31%, 100=77.62%, 250=22.07% cpu : usr=0.66%, sys=2.46%, ctx=1143, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21651: Thu May 1 22:40:57 2025 write: IOPS=182, BW=731KiB/s (748kB/s)(10.0MiB/14017msec) slat (usec): min=10, max=3941, avg=121.48, stdev=301.46 clat (msec): min=10, max=159, avg=86.87, stdev=17.79 lat (msec): min=11, max=159, avg=86.99, stdev=17.79 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 91], | 70.00th=[ 95], 80.00th=[ 103], 90.00th=[ 111], 95.00th=[ 118], | 99.00th=[ 130], 99.50th=[ 134], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 159] bw ( KiB/s): min= 552, max= 902, per=24.92%, avg=726.86, stdev=91.46, samples=28 iops : min= 138, max= 225, avg=181.64, stdev=22.84, samples=28 lat (msec) : 20=0.08%, 50=1.13%, 100=75.98%, 250=22.81% cpu : usr=0.80%, sys=2.45%, ctx=1165, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21653: Thu May 1 22:40:57 2025 write: IOPS=182, BW=730KiB/s (747kB/s)(10.0MiB/14034msec) slat (usec): min=10, max=6022, avg=97.69, stdev=264.63 clat (msec): min=46, max=155, avg=87.13, stdev=17.40 lat (msec): min=46, max=155, avg=87.23, stdev=17.40 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 77], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 96], 80.00th=[ 103], 90.00th=[ 110], 95.00th=[ 115], | 99.00th=[ 129], 99.50th=[ 148], 99.90th=[ 150], 99.95th=[ 155], | 99.99th=[ 157] bw ( KiB/s): min= 528, max= 904, per=24.91%, avg=726.50, stdev=95.11, samples=28 iops : min= 132, max= 226, avg=181.50, stdev=23.83, samples=28 lat (msec) : 50=0.12%, 100=76.56%, 250=23.32% cpu : usr=0.71%, sys=1.96%, ctx=907, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2918KiB/s (2988kB/s), 729KiB/s-732KiB/s (747kB/s-750kB/s), io=40.0MiB (41.9MB), run=13986-14039msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21674: Thu May 1 22:41:10 2025 read: IOPS=114, BW=456KiB/s (467kB/s)(5048KiB/11067msec) slat (usec): min=1442, max=13671, avg=4527.46, stdev=1774.33 clat (msec): min=7, max=110, avg=65.06, stdev=13.99 lat (msec): min=13, max=116, avg=69.59, stdev=14.52 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 89], | 99.00th=[ 100], 99.50th=[ 107], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 336, max= 600, per=25.08%, avg=454.73, stdev=62.43, samples=22 iops : min= 84, max= 150, avg=113.55, stdev=15.61, samples=22 write: IOPS=117, BW=469KiB/s (480kB/s)(5192KiB/11067msec) slat (usec): min=11, max=4681, avg=103.28, stdev=208.23 clat (msec): min=34, max=115, avg=67.45, stdev=12.91 lat (msec): min=34, max=115, avg=67.56, stdev=12.92 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 79], 90.00th=[ 85], 95.00th=[ 89], | 99.00th=[ 101], 99.50th=[ 105], 99.90th=[ 113], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 368, max= 656, per=24.86%, avg=469.18, stdev=77.92, samples=22 iops : min= 92, max= 164, avg=117.18, stdev=19.52, samples=22 lat (msec) : 10=0.04%, 20=0.04%, 50=10.39%, 100=88.55%, 250=0.98% cpu : usr=1.36%, sys=5.47%, ctx=2308, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21675: Thu May 1 22:41:10 2025 read: IOPS=112, BW=450KiB/s (461kB/s)(4976KiB/11059msec) slat (usec): min=1228, max=13345, avg=4498.08, stdev=1799.51 clat (msec): min=31, max=115, avg=65.37, stdev=14.04 lat (msec): min=33, max=124, avg=69.87, stdev=14.50 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 49], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 92], | 99.00th=[ 105], 99.50th=[ 112], 99.90th=[ 114], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 344, max= 576, per=24.84%, avg=450.33, stdev=63.17, samples=21 iops : min= 86, max= 144, avg=112.38, stdev=15.80, samples=21 write: IOPS=118, BW=476KiB/s (487kB/s)(5264KiB/11059msec) slat (usec): min=14, max=2847, avg=98.44, stdev=186.64 clat (msec): min=34, max=115, avg=67.57, stdev=13.37 lat (msec): min=34, max=115, avg=67.67, stdev=13.37 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 91], | 99.00th=[ 101], 99.50th=[ 107], 99.90th=[ 114], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 384, max= 672, per=25.38%, avg=478.86, stdev=66.22, samples=21 iops : min= 96, max= 168, avg=119.52, stdev=16.59, samples=21 lat (msec) : 50=10.39%, 100=88.36%, 250=1.25% cpu : usr=1.95%, sys=4.90%, ctx=2237, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21676: Thu May 1 22:41:10 2025 read: IOPS=112, BW=452KiB/s (462kB/s)(4984KiB/11037msec) slat (usec): min=1096, max=14999, avg=4616.51, stdev=1813.39 clat (msec): min=7, max=108, avg=64.97, stdev=14.13 lat (msec): min=14, max=116, avg=69.59, stdev=14.64 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 90], | 99.00th=[ 101], 99.50th=[ 105], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 343, max= 640, per=24.88%, avg=451.09, stdev=79.00, samples=22 iops : min= 85, max= 160, avg=112.50, stdev=19.82, samples=22 write: IOPS=119, BW=476KiB/s (488kB/s)(5256KiB/11037msec) slat (usec): min=15, max=2773, avg=102.33, stdev=205.34 clat (msec): min=34, max=123, avg=67.33, stdev=13.10 lat (msec): min=34, max=123, avg=67.43, stdev=13.10 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 74], 80.00th=[ 79], 90.00th=[ 84], 95.00th=[ 90], | 99.00th=[ 101], 99.50th=[ 107], 99.90th=[ 120], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 336, max= 600, per=25.08%, avg=473.18, stdev=64.72, samples=22 iops : min= 84, max= 150, avg=118.05, stdev=16.20, samples=22 lat (msec) : 10=0.04%, 50=12.19%, 100=86.48%, 250=1.29% cpu : usr=0.43%, sys=6.17%, ctx=2267, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21677: Thu May 1 22:41:10 2025 read: IOPS=115, BW=461KiB/s (473kB/s)(5060KiB/10966msec) slat (usec): min=34, max=13059, avg=4503.18, stdev=1751.47 clat (msec): min=12, max=151, avg=64.39, stdev=14.67 lat (msec): min=12, max=159, avg=68.90, stdev=15.19 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 71], 80.00th=[ 77], 90.00th=[ 83], 95.00th=[ 89], | 99.00th=[ 105], 99.50th=[ 111], 99.90th=[ 136], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 376, max= 632, per=25.64%, avg=464.90, stdev=58.12, samples=21 iops : min= 94, max= 158, avg=116.10, stdev=14.51, samples=21 write: IOPS=118, BW=472KiB/s (484kB/s)(5180KiB/10966msec) slat (usec): min=13, max=3387, avg=102.88, stdev=221.01 clat (msec): min=8, max=178, avg=67.79, stdev=14.34 lat (msec): min=8, max=178, avg=67.89, stdev=14.33 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 91], | 99.00th=[ 101], 99.50th=[ 107], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 336, max= 640, per=25.14%, avg=474.48, stdev=70.46, samples=21 iops : min= 84, max= 160, avg=118.48, stdev=17.70, samples=21 lat (msec) : 10=0.04%, 20=0.20%, 50=11.99%, 100=86.45%, 250=1.33% cpu : usr=1.19%, sys=5.38%, ctx=2316, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1813KiB/s (1857kB/s), 450KiB/s-461KiB/s (461kB/s-473kB/s), io=19.6MiB (20.5MB), run=10966-11067msec WRITE: bw=1888KiB/s (1933kB/s), 469KiB/s-476KiB/s (480kB/s-488kB/s), io=20.4MiB (21.4MB), run=10966-11067msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=21690: Thu May 1 22:41:12 2025 read: IOPS=3, BW=122MiB/s (127MB/s)(40.0MiB/329msec) slat (nsec): min=5202.3k, max=5202.3k, avg=5202332.00, stdev= 0.00 clat (nsec): min=307089k, max=307089k, avg=307088585.00, stdev= 0.00 lat (nsec): min=312314k, max=312314k, avg=312314443.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 500=100.00% cpu : usr=0.61%, sys=0.61%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=122MiB/s (127MB/s), 122MiB/s-122MiB/s (127MB/s-127MB/s), io=40.0MiB (41.9MB), run=329-329msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout