== sanity test 398c: run fio to test AIO ================= 02:37:11 (1746167831) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.172573 s, 243 MB/s osc.lustre-OST0000-osc-ffff88008ec18000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=3569: Fri May 2 02:39:03 2025 write: IOPS=23, BW=94.2KiB/s (96.5kB/s)(10.0MiB/108670msec) slat (usec): min=12, max=14831, avg=160.94, stdev=513.72 clat (msec): min=94, max=1167, avg=678.03, stdev=150.02 lat (msec): min=95, max=1167, avg=678.19, stdev=150.03 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 451], 10.00th=[ 498], 20.00th=[ 567], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 735], 80.00th=[ 793], 90.00th=[ 877], 95.00th=[ 969], | 99.00th=[ 1070], 99.50th=[ 1083], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1167] bw ( KiB/s): min= 8, max= 160, per=24.86%, avg=93.47, stdev=28.95, samples=217 iops : min= 2, max= 40, avg=23.11, stdev= 7.27, samples=217 lat (msec) : 100=0.04%, 250=0.62%, 500=9.53%, 750=62.30%, 1000=24.10% cpu : usr=0.11%, sys=0.48%, ctx=1328, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=3570: Fri May 2 02:39:03 2025 write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(10.0MiB/108614msec) slat (usec): min=14, max=6354, avg=155.15, stdev=421.83 clat (msec): min=217, max=1189, avg=677.64, stdev=149.26 lat (msec): min=217, max=1189, avg=677.80, stdev=149.28 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 464], 10.00th=[ 506], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 642], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 743], 80.00th=[ 793], 90.00th=[ 869], 95.00th=[ 944], | 99.00th=[ 1099], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1183] bw ( KiB/s): min= 23, max= 256, per=25.19%, avg=94.73, stdev=31.53, samples=214 iops : min= 5, max= 64, avg=23.37, stdev= 7.92, samples=214 lat (msec) : 250=0.66%, 500=8.20%, 750=62.97%, 1000=25.43% cpu : usr=0.10%, sys=0.48%, ctx=1282, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=3571: Fri May 2 02:39:03 2025 write: IOPS=23, BW=94.3KiB/s (96.6kB/s)(10.0MiB/108546msec) slat (usec): min=13, max=9602, avg=154.73, stdev=427.55 clat (msec): min=71, max=1188, avg=677.16, stdev=150.93 lat (msec): min=71, max=1188, avg=677.31, stdev=150.93 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 456], 10.00th=[ 498], 20.00th=[ 567], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 735], 80.00th=[ 793], 90.00th=[ 877], 95.00th=[ 961], | 99.00th=[ 1045], 99.50th=[ 1070], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 15, max= 184, per=24.89%, avg=93.58, stdev=27.47, samples=217 iops : min= 3, max= 46, avg=23.21, stdev= 6.90, samples=217 lat (msec) : 100=0.08%, 250=0.90%, 500=9.41%, 750=62.70%, 1000=23.71% cpu : usr=0.11%, sys=0.48%, ctx=1407, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=3572: Fri May 2 02:39:03 2025 write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(10.0MiB/108608msec) slat (usec): min=13, max=5112, avg=145.48, stdev=366.03 clat (msec): min=216, max=1166, avg=677.57, stdev=151.60 lat (msec): min=216, max=1166, avg=677.72, stdev=151.61 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 451], 10.00th=[ 510], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 735], 80.00th=[ 793], 90.00th=[ 869], 95.00th=[ 953], | 99.00th=[ 1070], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 15, max= 256, per=24.98%, avg=93.93, stdev=29.58, samples=216 iops : min= 3, max= 64, avg=23.25, stdev= 7.42, samples=216 lat (msec) : 250=1.25%, 500=7.66%, 750=63.52%, 1000=23.79% cpu : usr=0.07%, sys=0.53%, ctx=1477, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=377KiB/s (386kB/s), 94.2KiB/s-94.3KiB/s (96.5kB/s-96.6kB/s), io=40.0MiB (41.9MB), run=108546-108670msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3670: Fri May 2 02:40:13 2025 read: IOPS=18, BW=72.9KiB/s (74.6kB/s)(5048KiB/69265msec) slat (usec): min=1641, max=23696, avg=6527.58, stdev=3050.09 clat (msec): min=11, max=719, avg=281.20, stdev=70.67 lat (msec): min=17, max=726, avg=287.73, stdev=71.46 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 186], 10.00th=[ 203], 20.00th=[ 228], | 30.00th=[ 245], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 363], 95.00th=[ 401], | 99.00th=[ 472], 99.50th=[ 542], 99.90th=[ 625], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 15, max= 160, per=25.07%, avg=72.44, stdev=28.35, samples=138 iops : min= 3, max= 40, avg=17.86, stdev= 7.17, samples=138 write: IOPS=18, BW=74.0KiB/s (76.8kB/s)(5192KiB/69265msec) slat (usec): min=15, max=3336, avg=180.40, stdev=356.54 clat (msec): min=200, max=982, avg=572.17, stdev=154.22 lat (msec): min=200, max=983, avg=572.36, stdev=154.24 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 334], 10.00th=[ 368], 20.00th=[ 430], | 30.00th=[ 485], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 701], 90.00th=[ 785], 95.00th=[ 844], | 99.00th=[ 944], 99.50th=[ 961], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 24, max= 143, per=24.68%, avg=74.28, stdev=21.69, samples=138 iops : min= 6, max= 35, avg=18.31, stdev= 5.43, samples=138 lat (msec) : 20=0.04%, 50=0.16%, 100=0.20%, 250=16.17%, 500=49.45% lat (msec) : 750=27.15%, 1000=6.84% cpu : usr=0.19%, sys=1.51%, ctx=2799, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3671: Fri May 2 02:40:13 2025 read: IOPS=17, BW=71.8KiB/s (73.5kB/s)(4976KiB/69296msec) slat (usec): min=1290, max=22876, avg=6493.93, stdev=2925.76 clat (msec): min=33, max=590, avg=282.16, stdev=69.57 lat (msec): min=38, max=599, avg=288.66, stdev=70.36 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 188], 10.00th=[ 207], 20.00th=[ 228], | 30.00th=[ 245], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 368], 95.00th=[ 405], | 99.00th=[ 477], 99.50th=[ 498], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 7, max= 160, per=24.64%, avg=71.22, stdev=28.95, samples=138 iops : min= 1, max= 40, avg=17.54, stdev= 7.21, samples=138 write: IOPS=18, BW=75.0KiB/s (77.8kB/s)(5264KiB/69296msec) slat (usec): min=13, max=10073, avg=182.15, stdev=535.68 clat (msec): min=206, max=1001, avg=567.99, stdev=152.27 lat (msec): min=206, max=1001, avg=568.17, stdev=152.28 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 330], 10.00th=[ 376], 20.00th=[ 430], | 30.00th=[ 481], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 776], 95.00th=[ 835], | 99.00th=[ 936], 99.50th=[ 953], 99.90th=[ 986], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 24, max= 168, per=25.03%, avg=75.36, stdev=25.16, samples=138 iops : min= 6, max= 42, avg=18.58, stdev= 6.36, samples=138 lat (msec) : 50=0.12%, 100=0.27%, 250=16.13%, 500=49.45%, 750=27.38% lat (msec) : 1000=6.60% cpu : usr=0.39%, sys=1.27%, ctx=2675, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3672: Fri May 2 02:40:13 2025 read: IOPS=17, BW=71.9KiB/s (73.7kB/s)(4984KiB/69293msec) slat (usec): min=1723, max=23782, avg=6513.93, stdev=3043.62 clat (msec): min=31, max=629, avg=281.21, stdev=68.71 lat (msec): min=37, max=642, avg=287.73, stdev=69.61 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 190], 10.00th=[ 203], 20.00th=[ 226], | 30.00th=[ 245], 40.00th=[ 262], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 388], | 99.00th=[ 481], 99.50th=[ 542], 99.90th=[ 625], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8, max= 152, per=24.74%, avg=71.50, stdev=28.87, samples=138 iops : min= 2, max= 38, avg=17.54, stdev= 7.25, samples=138 write: IOPS=18, BW=75.9KiB/s (77.7kB/s)(5256KiB/69293msec) slat (usec): min=12, max=6763, avg=190.14, stdev=451.78 clat (msec): min=229, max=982, avg=569.44, stdev=153.16 lat (msec): min=229, max=982, avg=569.63, stdev=153.18 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 334], 10.00th=[ 368], 20.00th=[ 426], | 30.00th=[ 485], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 651], 80.00th=[ 701], 90.00th=[ 785], 95.00th=[ 844], | 99.00th=[ 919], 99.50th=[ 944], 99.90th=[ 978], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 16, max= 167, per=24.92%, avg=75.01, stdev=23.45, samples=138 iops : min= 4, max= 41, avg=18.42, stdev= 5.90, samples=138 lat (msec) : 50=0.12%, 100=0.16%, 250=15.86%, 500=50.00%, 750=26.88% lat (msec) : 1000=6.99% cpu : usr=0.52%, sys=1.15%, ctx=2698, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3673: Fri May 2 02:40:13 2025 read: IOPS=18, BW=73.7KiB/s (75.5kB/s)(5060KiB/68625msec) slat (usec): min=27, max=25512, avg=6497.78, stdev=3258.19 clat (msec): min=7, max=697, avg=281.35, stdev=73.01 lat (msec): min=7, max=701, avg=287.85, stdev=73.93 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 186], 10.00th=[ 205], 20.00th=[ 228], | 30.00th=[ 245], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 368], 95.00th=[ 405], | 99.00th=[ 514], 99.50th=[ 558], 99.90th=[ 617], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 8, max= 151, per=25.34%, avg=73.23, stdev=29.91, samples=137 iops : min= 2, max= 37, avg=18.01, stdev= 7.50, samples=137 write: IOPS=18, BW=75.5KiB/s (77.3kB/s)(5180KiB/68625msec) slat (usec): min=13, max=5405, avg=177.98, stdev=418.33 clat (msec): min=107, max=1043, avg=565.86, stdev=157.21 lat (msec): min=107, max=1043, avg=566.04, stdev=157.22 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 326], 10.00th=[ 368], 20.00th=[ 426], | 30.00th=[ 472], 40.00th=[ 523], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 693], 90.00th=[ 776], 95.00th=[ 852], | 99.00th=[ 953], 99.50th=[ 995], 99.90th=[ 1003], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 31, max= 152, per=24.78%, avg=74.58, stdev=24.74, samples=137 iops : min= 7, max= 38, avg=18.34, stdev= 6.26, samples=137 lat (msec) : 10=0.16%, 20=0.08%, 50=0.16%, 100=0.08%, 250=16.37% lat (msec) : 500=49.69%, 750=27.23%, 1000=6.09% cpu : usr=0.13%, sys=1.58%, ctx=2834, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=290KiB/s (297kB/s), 71.8KiB/s-73.7KiB/s (73.5kB/s-75.5kB/s), io=19.6MiB (20.5MB), run=68625-69296msec WRITE: bw=301KiB/s (309kB/s), 74.0KiB/s-75.0KiB/s (76.8kB/s-77.8kB/s), io=20.4MiB (21.4MB), run=68625-69296msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=3732: Fri May 2 02:40:15 2025 read: IOPS=2, BW=89.9MiB/s (94.3MB/s)(40.0MiB/445msec) slat (nsec): min=5822.6k, max=5822.6k, avg=5822618.00, stdev= 0.00 clat (nsec): min=425949k, max=425949k, avg=425949064.00, stdev= 0.00 lat (nsec): min=431801k, max=431801k, avg=431800974.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.68%, ctx=11, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=89.9MiB/s (94.3MB/s), 89.9MiB/s-89.9MiB/s (94.3MB/s-94.3MB/s), io=40.0MiB (41.9MB), run=445-445msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout