-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:47 EDT 2025 oleg160-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg160-client.virtnet: executing check_config_client /mnt/lustre oleg160-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg160-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b593e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b593e000.idle_timeout=debug disable quota as required oleg160-server: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:42:58 (1748036578) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:00 EDT 2025 waiting for dbench pid 8767 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 645 42.12 MB/sec warmup 1 sec latency 13.248 ms 2 1124 22.98 MB/sec warmup 2 sec latency 82.016 ms 2 1820 17.56 MB/sec warmup 3 sec latency 31.469 ms 2 2559 14.45 MB/sec warmup 4 sec latency 29.561 ms 2 3645 15.45 MB/sec warmup 5 sec latency 29.204 ms 2 4226 13.39 MB/sec warmup 6 sec latency 13.728 ms 2 4716 12.03 MB/sec warmup 7 sec latency 43.559 ms 2 5390 11.36 MB/sec warmup 8 sec latency 32.399 ms 2 6037 10.63 MB/sec warmup 9 sec latency 26.832 ms 2 7054 11.49 MB/sec warmup 10 sec latency 33.337 ms 2 7578 10.76 MB/sec warmup 11 sec latency 10.492 ms 2 8698 8.77 MB/sec execute 1 sec latency 49.985 ms 2 9364 5.02 MB/sec execute 2 sec latency 29.546 ms 2 10106 7.84 MB/sec execute 3 sec latency 32.801 ms 2 10930 9.00 MB/sec execute 4 sec latency 25.072 ms 2 11386 7.46 MB/sec execute 5 sec latency 62.785 ms 2 11870 6.80 MB/sec execute 6 sec latency 29.637 ms 2 12494 6.77 MB/sec execute 7 sec latency 28.602 ms 2 13131 6.52 MB/sec execute 8 sec latency 33.373 ms 2 14059 7.37 MB/sec execute 9 sec latency 31.598 ms 2 14630 7.48 MB/sec execute 10 sec latency 10.055 ms 2 15070 6.92 MB/sec execute 11 sec latency 61.898 ms 2 15670 7.08 MB/sec execute 12 sec latency 44.829 ms 2 16230 6.62 MB/sec execute 13 sec latency 30.058 ms 2 16966 6.92 MB/sec execute 14 sec latency 34.751 ms 2 17872 7.36 MB/sec execute 15 sec latency 26.516 ms 2 18405 7.08 MB/sec execute 16 sec latency 12.464 ms 2 18813 6.88 MB/sec execute 17 sec latency 42.985 ms 2 19441 6.87 MB/sec execute 18 sec latency 29.810 ms 2 20144 6.69 MB/sec execute 19 sec latency 28.824 ms 2 20960 6.98 MB/sec execute 20 sec latency 36.464 ms 2 21637 7.18 MB/sec execute 21 sec latency 20.649 ms 2 22070 6.92 MB/sec execute 22 sec latency 70.085 ms 2 22594 6.77 MB/sec execute 23 sec latency 25.811 ms 2 23214 6.76 MB/sec execute 24 sec latency 42.572 ms 2 23878 6.69 MB/sec execute 25 sec latency 31.517 ms 2 24871 7.17 MB/sec execute 26 sec latency 24.772 ms 2 25338 7.02 MB/sec execute 27 sec latency 10.996 ms 2 25774 6.83 MB/sec execute 28 sec latency 61.635 ms 2 26376 6.89 MB/sec execute 29 sec latency 37.954 ms 2 26958 6.70 MB/sec execute 30 sec latency 25.579 ms 2 27651 6.83 MB/sec execute 31 sec latency 32.108 ms 2 28556 7.04 MB/sec execute 32 sec latency 25.465 ms 2 29070 6.91 MB/sec execute 33 sec latency 12.129 ms 2 29454 6.82 MB/sec execute 34 sec latency 42.497 ms 2 30085 6.81 MB/sec execute 35 sec latency 35.639 ms 2 30767 6.72 MB/sec execute 36 sec latency 31.672 ms 2 31591 6.87 MB/sec execute 37 sec latency 34.793 ms 2 32273 6.99 MB/sec execute 38 sec latency 21.434 ms 2 32741 6.84 MB/sec execute 39 sec latency 58.958 ms 2 33226 6.76 MB/sec execute 40 sec latency 25.067 ms 2 33838 6.75 MB/sec execute 41 sec latency 33.593 ms 2 34455 6.71 MB/sec execute 42 sec latency 28.563 ms 2 35473 7.00 MB/sec execute 43 sec latency 31.049 ms 2 35961 6.92 MB/sec execute 44 sec latency 11.179 ms 2 36402 6.80 MB/sec execute 45 sec latency 51.564 ms 2 37009 6.84 MB/sec execute 46 sec latency 27.430 ms 2 37571 6.72 MB/sec execute 47 sec latency 30.256 ms 2 38371 6.81 MB/sec execute 48 sec latency 27.607 ms 2 39193 6.94 MB/sec execute 49 sec latency 34.029 ms 2 39729 6.86 MB/sec execute 50 sec latency 9.979 ms 2 40121 6.79 MB/sec execute 51 sec latency 47.347 ms 2 40748 6.79 MB/sec execute 52 sec latency 30.658 ms 2 41418 6.73 MB/sec execute 53 sec latency 30.093 ms 2 42251 6.84 MB/sec execute 54 sec latency 29.863 ms 2 42931 6.91 MB/sec execute 55 sec latency 31.487 ms 2 43389 6.82 MB/sec execute 56 sec latency 59.485 ms 2 43995 6.86 MB/sec execute 57 sec latency 30.096 ms 2 44542 6.76 MB/sec execute 58 sec latency 34.086 ms 2 45199 6.73 MB/sec execute 59 sec latency 30.055 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13098 4.583 16.751 Close 9700 1.044 4.022 Rename 562 6.323 10.486 Unlink 2580 1.810 3.280 Qpathinfo 11952 1.159 10.550 Qfileinfo 2112 0.123 1.523 Qfsinfo 2188 0.079 1.193 Sfileinfo 1074 3.036 7.846 Find 4642 0.338 6.379 WriteX 6526 0.689 8.379 ReadX 20984 0.013 2.249 LockX 44 0.899 1.109 UnlockX 44 0.921 1.159 Flush 906 18.990 70.070 Throughput 6.72781 MB/sec 2 clients 2 procs max_latency=70.085 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:12 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 18740 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 2a (84s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:22 (1748036662) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:24 EDT 2025 waiting for dbench pid 9653 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 653 42.12 MB/sec warmup 1 sec latency 12.523 ms 2 1136 22.99 MB/sec warmup 2 sec latency 73.210 ms 2 1849 17.57 MB/sec warmup 3 sec latency 27.362 ms 2 2582 14.48 MB/sec warmup 4 sec latency 27.687 ms 2 3638 15.45 MB/sec warmup 5 sec latency 33.225 ms 2 4233 13.42 MB/sec warmup 6 sec latency 9.217 ms 2 4703 12.03 MB/sec warmup 7 sec latency 44.010 ms 2 5429 11.37 MB/sec warmup 8 sec latency 28.399 ms 2 6167 10.69 MB/sec warmup 9 sec latency 25.289 ms 2 7241 11.57 MB/sec warmup 10 sec latency 28.348 ms 2 7770 10.78 MB/sec warmup 11 sec latency 14.986 ms 2 8887 6.68 MB/sec execute 1 sec latency 49.438 ms 2 9556 5.17 MB/sec execute 2 sec latency 27.316 ms PIDs: 9653 9654 9655 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 4010 Resuming dbench 2 10068 3.69 MB/sec execute 6 sec latency 3230.621 ms 2 10906 5.00 MB/sec execute 7 sec latency 3230.774 ms 2 11393 4.53 MB/sec execute 8 sec latency 41.606 ms 2 11952 4.50 MB/sec execute 9 sec latency 45.869 ms 2 12544 4.64 MB/sec execute 10 sec latency 34.939 ms 2 13223 4.68 MB/sec execute 11 sec latency 33.863 ms 2 14233 5.90 MB/sec execute 12 sec latency 27.088 ms 2 14820 5.70 MB/sec execute 13 sec latency 9.115 ms 2 15330 5.58 MB/sec execute 14 sec latency 41.565 ms 2 16022 5.65 MB/sec execute 15 sec latency 25.624 ms 2 16693 5.60 MB/sec execute 16 sec latency 29.465 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 22750 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2767 Wait for dbench to finish 2 16694 5.03 MB/sec execute 17 sec latency 1757.446 ms 2 17738 5.81 MB/sec execute 18 sec latency 1757.580 ms 2 18227 5.68 MB/sec execute 19 sec latency 11.514 ms 2 18704 5.49 MB/sec execute 20 sec latency 72.024 ms 2 19375 5.64 MB/sec execute 21 sec latency 35.006 ms 2 19978 5.45 MB/sec execute 22 sec latency 27.120 ms 2 20762 5.79 MB/sec execute 23 sec latency 33.616 ms 2 21574 6.06 MB/sec execute 24 sec latency 24.753 ms 2 22066 5.88 MB/sec execute 25 sec latency 53.263 ms 2 22712 6.01 MB/sec execute 26 sec latency 34.951 ms 2 23326 5.83 MB/sec execute 27 sec latency 29.888 ms 2 24083 6.01 MB/sec execute 28 sec latency 28.175 ms 2 24980 6.26 MB/sec execute 29 sec latency 31.770 ms 2 25469 6.15 MB/sec execute 30 sec latency 10.278 ms 2 25874 6.01 MB/sec execute 31 sec latency 55.951 ms 2 26567 6.09 MB/sec execute 32 sec latency 30.058 ms 2 27254 6.01 MB/sec execute 33 sec latency 28.711 ms 2 28148 6.27 MB/sec execute 34 sec latency 31.190 ms 2 28803 6.36 MB/sec execute 35 sec latency 17.838 ms 2 29275 6.22 MB/sec execute 36 sec latency 45.237 ms 2 29914 6.29 MB/sec execute 37 sec latency 35.628 ms 2 30573 6.16 MB/sec execute 38 sec latency 33.262 ms 2 31382 6.35 MB/sec execute 39 sec latency 29.217 ms 2 32211 6.50 MB/sec execute 40 sec latency 30.876 ms 2 32697 6.38 MB/sec execute 41 sec latency 41.376 ms 2 33185 6.31 MB/sec execute 42 sec latency 37.130 ms 2 33825 6.31 MB/sec execute 43 sec latency 26.908 ms 2 34468 6.28 MB/sec execute 44 sec latency 29.047 ms 2 35506 6.57 MB/sec execute 45 sec latency 30.261 ms 2 35994 6.50 MB/sec execute 46 sec latency 9.836 ms 2 36440 6.39 MB/sec execute 47 sec latency 46.466 ms 2 37113 6.44 MB/sec execute 48 sec latency 27.809 ms 2 37713 6.34 MB/sec execute 49 sec latency 28.592 ms 2 38504 6.48 MB/sec execute 50 sec latency 27.700 ms 2 39323 6.60 MB/sec execute 51 sec latency 29.161 ms 2 39788 6.50 MB/sec execute 52 sec latency 46.508 ms 2 40261 6.44 MB/sec execute 53 sec latency 39.540 ms 2 40903 6.44 MB/sec execute 54 sec latency 28.496 ms 2 41536 6.41 MB/sec execute 55 sec latency 34.084 ms 2 42558 6.64 MB/sec execute 56 sec latency 28.934 ms 2 43070 6.58 MB/sec execute 57 sec latency 9.591 ms 2 43524 6.50 MB/sec execute 58 sec latency 60.284 ms 2 44218 6.54 MB/sec execute 59 sec latency 26.088 ms 2 cleanup 60 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12766 4.886 3230.765 Close 9434 1.002 3.978 Rename 532 6.057 9.912 Unlink 2518 1.728 3.867 Qpathinfo 11630 1.393 1757.571 Qfileinfo 2026 0.116 1.212 Qfsinfo 2068 0.075 1.065 Sfileinfo 1032 2.828 5.650 Find 4440 0.320 6.038 WriteX 6260 0.654 10.609 ReadX 19808 0.011 0.310 LockX 40 0.845 1.157 UnlockX 40 0.902 1.263 Flush 872 18.166 72.010 Throughput 6.53789 MB/sec 2 clients 2 procs max_latency=3230.774 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:37 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 25516 Clear changelog after use: no Errors: 0 lustre_rsync took 5 seconds Changelog records consumed: 11569 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0000: changelog user 'cl2' not found PASS 2b (82s) == lustre-rsync-test test complete, duration 176 sec ===== 17:45:44 (1748036744)