== sanity test 398c: run fio to test AIO ================= 23:05:18 (1746155118) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.21638 s, 194 MB/s osc.lustre-OST0000-osc-ffff8800b6c77000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=1091: Thu May 1 23:07:16 2025 write: IOPS=22, BW=88.9KiB/s (91.0kB/s)(10.0MiB/115222msec) slat (usec): min=13, max=11798, avg=164.65, stdev=478.74 clat (msec): min=86, max=1238, avg=718.82, stdev=172.12 lat (msec): min=86, max=1238, avg=718.99, stdev=172.13 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 443], 10.00th=[ 502], 20.00th=[ 575], | 30.00th=[ 642], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 760], | 70.00th=[ 793], 80.00th=[ 852], 90.00th=[ 944], 95.00th=[ 1011], | 99.00th=[ 1133], 99.50th=[ 1167], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1234] bw ( KiB/s): min= 24, max= 199, per=24.80%, avg=88.05, stdev=29.70, samples=230 iops : min= 6, max= 49, avg=21.67, stdev= 7.45, samples=230 lat (msec) : 100=0.04%, 250=0.62%, 500=9.18%, 750=48.12%, 1000=35.90% cpu : usr=0.12%, sys=0.44%, ctx=1468, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1092: Thu May 1 23:07:16 2025 write: IOPS=22, BW=88.9KiB/s (91.0kB/s)(10.0MiB/115222msec) slat (usec): min=12, max=10993, avg=161.40, stdev=458.67 clat (msec): min=85, max=1312, avg=718.80, stdev=169.88 lat (msec): min=85, max=1313, avg=718.96, stdev=169.88 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 443], 10.00th=[ 510], 20.00th=[ 575], | 30.00th=[ 642], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 760], | 70.00th=[ 793], 80.00th=[ 852], 90.00th=[ 936], 95.00th=[ 1003], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1267], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 24, max= 184, per=24.84%, avg=88.19, stdev=29.42, samples=230 iops : min= 6, max= 46, avg=21.80, stdev= 7.38, samples=230 lat (msec) : 100=0.12%, 250=0.59%, 500=8.44%, 750=49.22%, 1000=36.41% cpu : usr=0.09%, sys=0.48%, ctx=1502, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1093: Thu May 1 23:07:16 2025 write: IOPS=22, BW=88.9KiB/s (91.1kB/s)(10.0MiB/115123msec) slat (usec): min=12, max=5923, avg=152.64, stdev=392.62 clat (msec): min=140, max=1275, avg=718.20, stdev=170.03 lat (msec): min=140, max=1275, avg=718.35, stdev=170.04 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 447], 10.00th=[ 506], 20.00th=[ 567], | 30.00th=[ 642], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 760], | 70.00th=[ 802], 80.00th=[ 852], 90.00th=[ 936], 95.00th=[ 1011], | 99.00th=[ 1133], 99.50th=[ 1183], 99.90th=[ 1234], 99.95th=[ 1250], | 99.99th=[ 1284] bw ( KiB/s): min= 15, max= 215, per=24.91%, avg=88.44, stdev=31.81, samples=229 iops : min= 3, max= 53, avg=21.76, stdev= 7.96, samples=229 lat (msec) : 250=0.62%, 500=8.48%, 750=48.52%, 1000=36.80% cpu : usr=0.10%, sys=0.46%, ctx=1427, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1094: Thu May 1 23:07:16 2025 write: IOPS=22, BW=88.9KiB/s (91.0kB/s)(10.0MiB/115177msec) slat (usec): min=12, max=17968, avg=169.84, stdev=627.73 clat (msec): min=61, max=1266, avg=718.48, stdev=171.73 lat (msec): min=61, max=1266, avg=718.66, stdev=171.72 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 443], 10.00th=[ 506], 20.00th=[ 575], | 30.00th=[ 642], 40.00th=[ 684], 50.00th=[ 718], 60.00th=[ 760], | 70.00th=[ 802], 80.00th=[ 852], 90.00th=[ 944], 95.00th=[ 1020], | 99.00th=[ 1150], 99.50th=[ 1183], 99.90th=[ 1217], 99.95th=[ 1250], | 99.99th=[ 1267] bw ( KiB/s): min= 32, max= 208, per=24.84%, avg=88.17, stdev=27.29, samples=230 iops : min= 8, max= 52, avg=21.79, stdev= 6.87, samples=230 lat (msec) : 100=0.08%, 250=0.66%, 500=8.40%, 750=49.61%, 1000=35.27% cpu : usr=0.02%, sys=0.54%, ctx=1517, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=355KiB/s (364kB/s), 88.9KiB/s-88.9KiB/s (91.0kB/s-91.1kB/s), io=40.0MiB (41.9MB), run=115123-115222msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1198: Thu May 1 23:08:30 2025 read: IOPS=17, BW=70.1KiB/s (71.8kB/s)(5048KiB/72015msec) slat (usec): min=17, max=26223, avg=6613.20, stdev=3231.95 clat (msec): min=9, max=809, avg=287.15, stdev=74.21 lat (msec): min=14, max=820, avg=293.77, stdev=74.77 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 192], 10.00th=[ 211], 20.00th=[ 234], | 30.00th=[ 251], 40.00th=[ 268], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 321], 80.00th=[ 342], 90.00th=[ 368], 95.00th=[ 397], | 99.00th=[ 464], 99.50th=[ 510], 99.90th=[ 802], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 16, max= 152, per=25.26%, avg=69.97, stdev=28.06, samples=143 iops : min= 4, max= 38, avg=17.27, stdev= 7.04, samples=143 write: IOPS=18, BW=72.1KiB/s (73.8kB/s)(5192KiB/72015msec) slat (usec): min=16, max=7393, avg=184.69, stdev=495.90 clat (msec): min=84, max=1130, avg=601.15, stdev=161.94 lat (msec): min=84, max=1130, avg=601.33, stdev=161.97 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 359], 10.00th=[ 397], 20.00th=[ 460], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 600], 60.00th=[ 642], | 70.00th=[ 676], 80.00th=[ 718], 90.00th=[ 802], 95.00th=[ 894], | 99.00th=[ 1053], 99.50th=[ 1070], 99.90th=[ 1083], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 16, max= 135, per=24.93%, avg=71.81, stdev=22.92, samples=143 iops : min= 4, max= 33, avg=17.73, stdev= 5.76, samples=143 lat (msec) : 10=0.04%, 20=0.08%, 50=0.23%, 100=0.55%, 250=13.75% lat (msec) : 500=48.67%, 750=28.59%, 1000=7.23% cpu : usr=0.10%, sys=1.46%, ctx=2689, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1199: Thu May 1 23:08:30 2025 read: IOPS=17, BW=68.7KiB/s (70.3kB/s)(4976KiB/72431msec) slat (usec): min=1642, max=31120, avg=6726.95, stdev=3256.09 clat (msec): min=4, max=498, avg=291.63, stdev=62.47 lat (msec): min=15, max=511, avg=298.37, stdev=63.07 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 205], 10.00th=[ 222], 20.00th=[ 241], | 30.00th=[ 257], 40.00th=[ 271], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 321], 80.00th=[ 342], 90.00th=[ 372], 95.00th=[ 397], | 99.00th=[ 464], 99.50th=[ 477], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 16, max= 144, per=24.48%, avg=67.82, stdev=27.71, samples=144 iops : min= 4, max= 36, avg=16.74, stdev= 6.97, samples=144 write: IOPS=18, BW=72.7KiB/s (74.4kB/s)(5264KiB/72431msec) slat (usec): min=15, max=4497, avg=150.84, stdev=333.64 clat (msec): min=143, max=1110, avg=597.14, stdev=157.60 lat (msec): min=143, max=1111, avg=597.30, stdev=157.61 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 368], 10.00th=[ 401], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 634], | 70.00th=[ 676], 80.00th=[ 718], 90.00th=[ 793], 95.00th=[ 869], | 99.00th=[ 1045], 99.50th=[ 1062], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 23, max= 128, per=24.97%, avg=71.92, stdev=20.75, samples=144 iops : min= 5, max= 32, avg=17.76, stdev= 5.19, samples=144 lat (msec) : 10=0.08%, 100=0.16%, 250=12.89%, 500=50.23%, 750=29.06% lat (msec) : 1000=6.76% cpu : usr=0.34%, sys=1.21%, ctx=2710, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1200: Thu May 1 23:08:30 2025 read: IOPS=17, BW=68.9KiB/s (70.6kB/s)(4984KiB/72335msec) slat (usec): min=1719, max=31986, avg=6834.75, stdev=3176.73 clat (msec): min=47, max=504, avg=290.37, stdev=60.68 lat (msec): min=51, max=512, avg=297.21, stdev=61.25 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 201], 10.00th=[ 220], 20.00th=[ 243], | 30.00th=[ 257], 40.00th=[ 271], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 317], 80.00th=[ 338], 90.00th=[ 368], 95.00th=[ 388], | 99.00th=[ 481], 99.50th=[ 493], 99.90th=[ 502], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 15, max= 152, per=24.78%, avg=68.63, stdev=27.28, samples=144 iops : min= 3, max= 38, avg=16.89, stdev= 6.85, samples=144 write: IOPS=18, BW=72.7KiB/s (74.4kB/s)(5256KiB/72335msec) slat (usec): min=12, max=3230, avg=173.81, stdev=359.04 clat (msec): min=82, max=1071, avg=597.49, stdev=156.06 lat (msec): min=83, max=1071, avg=597.66, stdev=156.04 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 368], 10.00th=[ 401], 20.00th=[ 460], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 667], 80.00th=[ 718], 90.00th=[ 802], 95.00th=[ 869], | 99.00th=[ 1028], 99.50th=[ 1045], 99.90th=[ 1062], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 24, max= 120, per=24.91%, avg=71.74, stdev=21.76, samples=144 iops : min= 6, max= 30, avg=17.67, stdev= 5.42, samples=144 lat (msec) : 50=0.08%, 100=0.16%, 250=12.30%, 500=51.05%, 750=28.32% lat (msec) : 1000=7.27% cpu : usr=0.28%, sys=1.30%, ctx=2711, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1201: Thu May 1 23:08:30 2025 read: IOPS=17, BW=70.2KiB/s (71.9kB/s)(5060KiB/72090msec) slat (usec): min=1944, max=28296, avg=6870.35, stdev=3161.24 clat (msec): min=9, max=602, avg=292.23, stdev=62.39 lat (msec): min=18, max=608, avg=299.11, stdev=63.03 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 205], 10.00th=[ 224], 20.00th=[ 243], | 30.00th=[ 259], 40.00th=[ 275], 50.00th=[ 288], 60.00th=[ 305], | 70.00th=[ 321], 80.00th=[ 338], 90.00th=[ 376], 95.00th=[ 397], | 99.00th=[ 460], 99.50th=[ 493], 99.90th=[ 523], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 7, max= 136, per=25.29%, avg=70.06, stdev=25.56, samples=143 iops : min= 1, max= 34, avg=17.15, stdev= 6.47, samples=143 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(5180KiB/72090msec) slat (usec): min=15, max=14307, avg=207.14, stdev=656.86 clat (msec): min=102, max=1109, avg=597.05, stdev=157.78 lat (msec): min=102, max=1111, avg=597.26, stdev=157.81 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 368], 10.00th=[ 405], 20.00th=[ 456], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 634], | 70.00th=[ 676], 80.00th=[ 718], 90.00th=[ 793], 95.00th=[ 885], | 99.00th=[ 1028], 99.50th=[ 1053], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 23, max= 136, per=24.60%, avg=70.84, stdev=22.11, samples=144 iops : min= 5, max= 34, avg=17.34, stdev= 5.54, samples=144 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.12%, 250=12.15% lat (msec) : 500=51.52%, 750=28.52%, 1000=6.76% cpu : usr=0.25%, sys=1.37%, ctx=2718, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=277KiB/s (284kB/s), 68.7KiB/s-70.2KiB/s (70.3kB/s-71.9kB/s), io=19.6MiB (20.5MB), run=72015-72431msec WRITE: bw=288KiB/s (295kB/s), 71.9KiB/s-72.7KiB/s (73.6kB/s-74.4kB/s), io=20.4MiB (21.4MB), run=72015-72431msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=1264: Thu May 1 23:08:33 2025 read: IOPS=2, BW=87.9MiB/s (92.2MB/s)(40.0MiB/455msec) slat (nsec): min=10047k, max=10047k, avg=10047203.00, stdev= 0.00 clat (nsec): min=430681k, max=430681k, avg=430681441.00, stdev= 0.00 lat (nsec): min=440787k, max=440787k, avg=440786745.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.88%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=87.9MiB/s (92.2MB/s), 87.9MiB/s-87.9MiB/s (92.2MB/s-92.2MB/s), io=40.0MiB (41.9MB), run=455-455msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout