== sanity test 398c: run fio to test AIO ================= 18:47:25 (1746139645) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.146775 s, 286 MB/s osc.lustre-OST0000-osc-ffff88008706f000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=4849: Thu May 1 18:49:27 2025 write: IOPS=21, BW=86.0KiB/s (88.1kB/s)(10.0MiB/119009msec) slat (usec): min=12, max=16973, avg=156.26, stdev=486.86 clat (msec): min=204, max=1774, avg=742.64, stdev=178.01 lat (msec): min=204, max=1774, avg=742.80, stdev=178.01 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 468], 10.00th=[ 535], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 961], 95.00th=[ 1036], | 99.00th=[ 1217], 99.50th=[ 1334], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1770] bw ( KiB/s): min= 15, max= 184, per=24.94%, avg=85.54, stdev=30.25, samples=237 iops : min= 3, max= 46, avg=21.11, stdev= 7.59, samples=237 lat (msec) : 250=0.16%, 500=7.07%, 750=46.05%, 1000=39.45% cpu : usr=0.11%, sys=0.42%, ctx=1375, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4850: Thu May 1 18:49:27 2025 write: IOPS=21, BW=85.0KiB/s (88.0kB/s)(10.0MiB/119113msec) slat (usec): min=16, max=9831, avg=164.15, stdev=469.48 clat (msec): min=73, max=1688, avg=743.64, stdev=178.67 lat (msec): min=73, max=1688, avg=743.81, stdev=178.68 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 477], 10.00th=[ 542], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 953], 95.00th=[ 1036], | 99.00th=[ 1267], 99.50th=[ 1418], 99.90th=[ 1519], 99.95th=[ 1552], | 99.99th=[ 1687] bw ( KiB/s): min= 15, max= 191, per=24.93%, avg=85.52, stdev=29.88, samples=237 iops : min= 3, max= 47, avg=21.11, stdev= 7.46, samples=237 lat (msec) : 100=0.12%, 250=0.51%, 500=6.37%, 750=45.82%, 1000=41.02% cpu : usr=0.04%, sys=0.51%, ctx=1399, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4851: Thu May 1 18:49:27 2025 write: IOPS=21, BW=85.8KiB/s (87.9kB/s)(10.0MiB/119298msec) slat (usec): min=13, max=10441, avg=155.02, stdev=428.11 clat (msec): min=226, max=1535, avg=744.51, stdev=171.21 lat (msec): min=228, max=1536, avg=744.67, stdev=171.24 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 477], 10.00th=[ 542], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 701], 50.00th=[ 743], 60.00th=[ 785], | 70.00th=[ 827], 80.00th=[ 877], 90.00th=[ 953], 95.00th=[ 1036], | 99.00th=[ 1217], 99.50th=[ 1284], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1536] bw ( KiB/s): min= 23, max= 167, per=24.83%, avg=85.16, stdev=28.54, samples=238 iops : min= 5, max= 41, avg=21.02, stdev= 7.13, samples=238 lat (msec) : 250=0.04%, 500=6.33%, 750=46.02%, 1000=40.62% cpu : usr=0.06%, sys=0.52%, ctx=1428, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=4852: Thu May 1 18:49:27 2025 write: IOPS=21, BW=85.8KiB/s (87.8kB/s)(10.0MiB/119364msec) slat (usec): min=15, max=4823, avg=150.46, stdev=394.15 clat (msec): min=203, max=1496, avg=744.93, stdev=171.59 lat (msec): min=204, max=1496, avg=745.09, stdev=171.61 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 481], 10.00th=[ 542], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 785], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 961], 95.00th=[ 1036], | 99.00th=[ 1250], 99.50th=[ 1318], 99.90th=[ 1452], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 16, max= 144, per=24.84%, avg=85.20, stdev=28.57, samples=238 iops : min= 4, max= 36, avg=21.07, stdev= 7.18, samples=238 lat (msec) : 250=0.08%, 500=6.09%, 750=47.15%, 1000=39.57% cpu : usr=0.06%, sys=0.47%, ctx=1352, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=343KiB/s (351kB/s), 85.8KiB/s-86.0KiB/s (87.8kB/s-88.1kB/s), io=40.0MiB (41.9MB), run=119009-119364msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4958: Thu May 1 18:50:40 2025 read: IOPS=17, BW=70.3KiB/s (71.0kB/s)(5048KiB/71835msec) slat (usec): min=1198, max=20117, avg=6174.71, stdev=2848.81 clat (msec): min=10, max=551, avg=288.67, stdev=72.93 lat (msec): min=19, max=565, avg=294.86, stdev=73.56 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 184], 10.00th=[ 207], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 266], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 326], 80.00th=[ 342], 90.00th=[ 388], 95.00th=[ 426], | 99.00th=[ 472], 99.50th=[ 502], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 15, max= 152, per=25.01%, avg=69.78, stdev=29.03, samples=143 iops : min= 3, max= 38, avg=17.21, stdev= 7.30, samples=143 write: IOPS=18, BW=72.3KiB/s (74.0kB/s)(5192KiB/71835msec) slat (usec): min=16, max=11173, avg=173.18, stdev=475.49 clat (msec): min=203, max=1019, avg=597.21, stdev=171.12 lat (msec): min=203, max=1019, avg=597.39, stdev=171.16 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 334], 10.00th=[ 376], 20.00th=[ 447], | 30.00th=[ 489], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 684], 80.00th=[ 760], 90.00th=[ 844], 95.00th=[ 902], | 99.00th=[ 961], 99.50th=[ 978], 99.90th=[ 1003], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 16, max= 152, per=24.72%, avg=71.69, stdev=23.70, samples=143 iops : min= 4, max= 38, avg=17.69, stdev= 5.94, samples=143 lat (msec) : 20=0.04%, 50=0.16%, 100=0.23%, 250=15.23%, 500=49.22% lat (msec) : 750=24.53%, 1000=10.47% cpu : usr=0.29%, sys=1.28%, ctx=2765, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4959: Thu May 1 18:50:40 2025 read: IOPS=17, BW=69.3KiB/s (70.9kB/s)(4976KiB/71841msec) slat (usec): min=1409, max=24809, avg=6242.60, stdev=2943.10 clat (msec): min=8, max=552, avg=290.27, stdev=72.70 lat (msec): min=11, max=562, avg=296.52, stdev=73.33 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 182], 10.00th=[ 207], 20.00th=[ 234], | 30.00th=[ 253], 40.00th=[ 268], 50.00th=[ 288], 60.00th=[ 300], | 70.00th=[ 321], 80.00th=[ 347], 90.00th=[ 380], 95.00th=[ 422], | 99.00th=[ 477], 99.50th=[ 506], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 15, max= 135, per=24.63%, avg=68.73, stdev=27.88, samples=143 iops : min= 3, max= 33, avg=16.95, stdev= 6.99, samples=143 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(5264KiB/71841msec) slat (usec): min=14, max=4202, avg=155.12, stdev=332.88 clat (msec): min=163, max=1025, avg=591.54, stdev=172.64 lat (msec): min=163, max=1025, avg=591.69, stdev=172.66 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 330], 10.00th=[ 376], 20.00th=[ 439], | 30.00th=[ 485], 40.00th=[ 531], 50.00th=[ 575], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 751], 90.00th=[ 835], 95.00th=[ 919], | 99.00th=[ 986], 99.50th=[ 1003], 99.90th=[ 1020], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 8, max= 152, per=25.13%, avg=72.87, stdev=24.79, samples=143 iops : min= 2, max= 38, avg=17.99, stdev= 6.23, samples=143 lat (msec) : 10=0.04%, 50=0.23%, 100=0.08%, 250=14.22%, 500=50.62% lat (msec) : 750=24.41%, 1000=10.16% cpu : usr=0.36%, sys=1.19%, ctx=2692, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4960: Thu May 1 18:50:40 2025 read: IOPS=17, BW=69.4KiB/s (71.1kB/s)(4984KiB/71803msec) slat (usec): min=23, max=29187, avg=6218.27, stdev=3032.81 clat (msec): min=13, max=527, avg=288.34, stdev=73.11 lat (msec): min=20, max=535, avg=294.56, stdev=73.87 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 180], 10.00th=[ 203], 20.00th=[ 232], | 30.00th=[ 251], 40.00th=[ 268], 50.00th=[ 288], 60.00th=[ 300], | 70.00th=[ 321], 80.00th=[ 351], 90.00th=[ 384], 95.00th=[ 422], | 99.00th=[ 460], 99.50th=[ 485], 99.90th=[ 518], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 8, max= 191, per=24.83%, avg=69.27, stdev=30.41, samples=143 iops : min= 2, max= 47, avg=17.07, stdev= 7.62, samples=143 write: IOPS=18, BW=73.2KiB/s (74.0kB/s)(5256KiB/71803msec) slat (usec): min=16, max=4259, avg=162.50, stdev=352.87 clat (msec): min=115, max=1025, avg=594.08, stdev=167.22 lat (msec): min=115, max=1025, avg=594.25, stdev=167.24 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 334], 10.00th=[ 384], 20.00th=[ 451], | 30.00th=[ 493], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 751], 90.00th=[ 827], 95.00th=[ 894], | 99.00th=[ 969], 99.50th=[ 978], 99.90th=[ 995], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 23, max= 152, per=24.98%, avg=72.45, stdev=24.04, samples=143 iops : min= 5, max= 38, avg=17.87, stdev= 6.05, samples=143 lat (msec) : 20=0.08%, 50=0.20%, 100=0.08%, 250=14.57%, 500=49.57% lat (msec) : 750=25.39%, 1000=10.08% cpu : usr=0.08%, sys=1.46%, ctx=2660, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4961: Thu May 1 18:50:40 2025 read: IOPS=17, BW=70.6KiB/s (72.3kB/s)(5060KiB/71681msec) slat (usec): min=1513, max=26679, avg=6179.45, stdev=3035.66 clat (msec): min=4, max=597, avg=288.72, stdev=71.26 lat (msec): min=6, max=606, avg=294.91, stdev=72.05 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 184], 10.00th=[ 209], 20.00th=[ 236], | 30.00th=[ 251], 40.00th=[ 268], 50.00th=[ 284], 60.00th=[ 300], | 70.00th=[ 321], 80.00th=[ 338], 90.00th=[ 380], 95.00th=[ 422], | 99.00th=[ 468], 99.50th=[ 510], 99.90th=[ 542], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8, max= 199, per=25.23%, avg=70.40, stdev=29.98, samples=142 iops : min= 2, max= 49, avg=17.29, stdev= 7.53, samples=142 write: IOPS=18, BW=72.3KiB/s (73.0kB/s)(5180KiB/71681msec) slat (usec): min=14, max=9002, avg=208.35, stdev=587.95 clat (msec): min=178, max=1028, avg=595.49, stdev=167.85 lat (msec): min=178, max=1028, avg=595.70, stdev=167.87 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 347], 10.00th=[ 380], 20.00th=[ 447], | 30.00th=[ 489], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 751], 90.00th=[ 844], 95.00th=[ 902], | 99.00th=[ 969], 99.50th=[ 995], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 15, max= 136, per=24.70%, avg=71.64, stdev=22.49, samples=143 iops : min= 3, max= 34, avg=17.59, stdev= 5.62, samples=143 lat (msec) : 10=0.04%, 20=0.12%, 50=0.04%, 100=0.08%, 250=14.22% lat (msec) : 500=50.66%, 750=24.49%, 1000=10.23% cpu : usr=0.17%, sys=1.43%, ctx=2716, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=279KiB/s (286kB/s), 69.3KiB/s-70.6KiB/s (70.9kB/s-72.3kB/s), io=19.6MiB (20.5MB), run=71681-71841msec WRITE: bw=291KiB/s (298kB/s), 72.3KiB/s-73.3KiB/s (73.0kB/s-75.0kB/s), io=20.4MiB (21.4MB), run=71681-71841msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=5022: Thu May 1 18:50:42 2025 read: IOPS=2, BW=86.8MiB/s (90.0MB/s)(40.0MiB/461msec) slat (nsec): min=34638k, max=34638k, avg=34637945.00, stdev= 0.00 clat (nsec): min=403101k, max=403101k, avg=403100767.00, stdev= 0.00 lat (nsec): min=437764k, max=437764k, avg=437764147.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.17%, ctx=43, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=86.8MiB/s (90.0MB/s), 86.8MiB/s-86.8MiB/s (90.0MB/s-90.0MB/s), io=40.0MiB (41.9MB), run=461-461msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout