-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:46 EDT 2025 oleg141-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg141-client.virtnet: executing check_config_client /mnt/lustre oleg141-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg141-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5860000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5860000.idle_timeout=debug disable quota as required oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:42:57 (1748036577) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:42:58 EDT 2025 waiting for dbench pid 8748 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 1 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 592 40.95 MB/sec warmup 1 sec latency 15.058 ms 2 1074 22.92 MB/sec warmup 2 sec latency 71.730 ms 2 1837 17.57 MB/sec warmup 3 sec latency 31.291 ms 2 2586 14.48 MB/sec warmup 4 sec latency 27.620 ms 2 3645 15.45 MB/sec warmup 5 sec latency 30.209 ms 2 4236 13.42 MB/sec warmup 6 sec latency 18.451 ms 2 4714 12.03 MB/sec warmup 7 sec latency 44.567 ms 2 5379 11.36 MB/sec warmup 8 sec latency 28.393 ms 2 6119 10.67 MB/sec warmup 9 sec latency 23.920 ms 2 7171 11.54 MB/sec warmup 10 sec latency 28.825 ms 2 7623 10.76 MB/sec warmup 11 sec latency 11.397 ms 2 8755 8.77 MB/sec execute 1 sec latency 54.762 ms 2 9449 6.08 MB/sec execute 2 sec latency 29.824 ms 2 10305 8.31 MB/sec execute 3 sec latency 34.526 ms 2 10995 9.12 MB/sec execute 4 sec latency 15.537 ms 2 11449 7.50 MB/sec execute 5 sec latency 49.064 ms 2 12066 7.77 MB/sec execute 6 sec latency 22.805 ms 2 12636 6.79 MB/sec execute 7 sec latency 29.974 ms 2 13307 6.84 MB/sec execute 8 sec latency 28.282 ms 2 14292 7.99 MB/sec execute 9 sec latency 32.087 ms 2 14774 7.48 MB/sec execute 10 sec latency 10.949 ms 2 15189 6.97 MB/sec execute 11 sec latency 60.936 ms 2 15873 7.11 MB/sec execute 12 sec latency 29.132 ms 2 16546 6.82 MB/sec execute 13 sec latency 33.751 ms 2 17416 7.25 MB/sec execute 14 sec latency 25.124 ms 2 18104 7.53 MB/sec execute 15 sec latency 22.728 ms 2 18556 7.13 MB/sec execute 16 sec latency 52.106 ms 2 19208 7.24 MB/sec execute 17 sec latency 39.966 ms 2 19828 6.91 MB/sec execute 18 sec latency 26.916 ms 2 20554 7.11 MB/sec execute 19 sec latency 32.542 ms 2 21472 7.44 MB/sec execute 20 sec latency 24.544 ms 2 22003 7.23 MB/sec execute 21 sec latency 56.781 ms 2 22474 7.06 MB/sec execute 22 sec latency 42.061 ms 2 23146 7.05 MB/sec execute 23 sec latency 30.052 ms 2 23806 6.95 MB/sec execute 24 sec latency 31.837 ms 2 24896 7.45 MB/sec execute 25 sec latency 22.939 ms 2 25416 7.29 MB/sec execute 26 sec latency 9.941 ms 2 25869 7.09 MB/sec execute 27 sec latency 53.537 ms 2 26567 7.15 MB/sec execute 28 sec latency 29.078 ms 2 27264 7.02 MB/sec execute 29 sec latency 26.092 ms 2 28167 7.28 MB/sec execute 30 sec latency 31.675 ms 2 28807 7.35 MB/sec execute 31 sec latency 21.500 ms 2 29246 7.15 MB/sec execute 32 sec latency 51.655 ms 2 29905 7.21 MB/sec execute 33 sec latency 38.880 ms 2 30556 7.04 MB/sec execute 34 sec latency 30.485 ms 2 31346 7.22 MB/sec execute 35 sec latency 29.866 ms 2 32203 7.37 MB/sec execute 36 sec latency 23.792 ms 2 32697 7.21 MB/sec execute 37 sec latency 49.336 ms 2 33190 7.11 MB/sec execute 38 sec latency 37.063 ms 2 33847 7.10 MB/sec execute 39 sec latency 30.768 ms 2 34547 7.05 MB/sec execute 40 sec latency 27.834 ms 2 35582 7.35 MB/sec execute 41 sec latency 30.020 ms 2 36086 7.25 MB/sec execute 42 sec latency 10.134 ms 2 36516 7.12 MB/sec execute 43 sec latency 53.798 ms 2 37198 7.15 MB/sec execute 44 sec latency 27.762 ms 2 37865 7.07 MB/sec execute 45 sec latency 36.345 ms 2 38711 7.19 MB/sec execute 46 sec latency 30.049 ms 2 39375 7.27 MB/sec execute 47 sec latency 16.194 ms 2 39840 7.15 MB/sec execute 48 sec latency 53.809 ms 2 40329 7.08 MB/sec execute 49 sec latency 38.184 ms 2 40951 7.07 MB/sec execute 50 sec latency 32.211 ms 2 41639 7.03 MB/sec execute 51 sec latency 27.741 ms 2 42635 7.26 MB/sec execute 52 sec latency 36.447 ms 2 43106 7.18 MB/sec execute 53 sec latency 12.413 ms 2 43542 7.08 MB/sec execute 54 sec latency 45.779 ms 2 44206 7.11 MB/sec execute 55 sec latency 30.291 ms 2 44838 7.01 MB/sec execute 56 sec latency 26.436 ms 2 45605 7.12 MB/sec execute 57 sec latency 38.493 ms 2 46444 7.21 MB/sec execute 58 sec latency 22.071 ms 2 46894 7.11 MB/sec execute 59 sec latency 58.957 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 13600 4.392 12.406 Close 10008 1.001 2.989 Rename 586 6.022 9.390 Unlink 2742 1.736 4.354 Qpathinfo 12364 1.106 10.864 Qfileinfo 2148 0.115 1.104 Qfsinfo 2270 0.075 1.144 Sfileinfo 1132 2.849 7.507 Find 4760 0.332 6.242 WriteX 6738 0.654 6.410 ReadX 21294 0.012 2.156 LockX 44 0.851 1.348 UnlockX 44 0.893 1.159 Flush 950 18.805 60.929 Throughput 7.11363 MB/sec 2 clients 2 procs max_latency=60.936 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:11 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 19342 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 2a (84s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:21 (1748036661) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:22 EDT 2025 waiting for dbench pid 9633 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 664 42.31 MB/sec warmup 1 sec latency 11.747 ms 2 1168 23.01 MB/sec warmup 2 sec latency 69.372 ms 2 1877 17.58 MB/sec warmup 3 sec latency 29.416 ms 2 2609 14.50 MB/sec warmup 4 sec latency 28.152 ms 2 3719 15.51 MB/sec warmup 5 sec latency 33.867 ms 2 4281 13.44 MB/sec warmup 6 sec latency 38.998 ms 2 4834 12.07 MB/sec warmup 7 sec latency 31.332 ms 2 5575 11.41 MB/sec warmup 8 sec latency 26.444 ms 2 6486 11.69 MB/sec warmup 9 sec latency 25.661 ms 2 7405 11.78 MB/sec warmup 10 sec latency 26.852 ms 2 7904 10.85 MB/sec warmup 11 sec latency 53.779 ms 2 9163 1.19 MB/sec execute 1 sec latency 37.629 ms 2 9995 6.79 MB/sec execute 2 sec latency 27.150 ms PIDs: 9633 9634 9635 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 4104 Resuming dbench 2 10471 3.24 MB/sec execute 6 sec latency 3293.537 ms 2 11104 4.11 MB/sec execute 7 sec latency 3293.676 ms 2 11576 3.80 MB/sec execute 8 sec latency 48.685 ms 2 12279 4.36 MB/sec execute 9 sec latency 28.671 ms 2 12999 4.27 MB/sec execute 10 sec latency 28.165 ms 2 13861 5.06 MB/sec execute 11 sec latency 31.757 ms 2 14546 5.61 MB/sec execute 12 sec latency 18.081 ms 2 15001 5.25 MB/sec execute 13 sec latency 47.435 ms 2 15683 5.54 MB/sec execute 14 sec latency 27.738 ms 2 16382 5.26 MB/sec execute 15 sec latency 22.755 ms 2 17362 5.86 MB/sec execute 16 sec latency 28.552 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 23446 Clear changelog after use: no Errors: 0 lustre_rsync took 2 seconds Changelog records consumed: 2809 Wait for dbench to finish 2 17363 5.19 MB/sec execute 18 sec latency 2023.671 ms 2 18068 5.52 MB/sec execute 19 sec latency 2023.878 ms 2 18536 5.31 MB/sec execute 20 sec latency 56.707 ms 2 19149 5.50 MB/sec execute 21 sec latency 33.077 ms 2 19737 5.29 MB/sec execute 22 sec latency 30.732 ms 2 20385 5.29 MB/sec execute 23 sec latency 33.480 ms 2 21395 5.87 MB/sec execute 24 sec latency 31.163 ms 2 21866 5.75 MB/sec execute 25 sec latency 10.595 ms 2 22304 5.60 MB/sec execute 26 sec latency 46.996 ms 2 23010 5.72 MB/sec execute 27 sec latency 32.826 ms 2 23727 5.64 MB/sec execute 28 sec latency 24.873 ms 2 24691 5.96 MB/sec execute 29 sec latency 29.798 ms 2 25313 6.07 MB/sec execute 30 sec latency 10.138 ms 2 25777 5.92 MB/sec execute 31 sec latency 44.637 ms 2 26430 6.01 MB/sec execute 32 sec latency 31.728 ms 2 27044 5.87 MB/sec execute 33 sec latency 28.578 ms 2 27853 6.09 MB/sec execute 34 sec latency 29.775 ms 2 28680 6.28 MB/sec execute 35 sec latency 24.849 ms 2 29154 6.14 MB/sec execute 36 sec latency 44.773 ms 2 29665 6.07 MB/sec execute 37 sec latency 33.287 ms 2 30307 6.08 MB/sec execute 38 sec latency 31.367 ms 2 31011 6.06 MB/sec execute 39 sec latency 25.902 ms 2 32043 6.39 MB/sec execute 40 sec latency 31.979 ms 2 32592 6.31 MB/sec execute 41 sec latency 9.262 ms 2 33023 6.25 MB/sec execute 42 sec latency 48.685 ms 2 33697 6.26 MB/sec execute 43 sec latency 30.052 ms 2 34373 6.20 MB/sec execute 44 sec latency 30.411 ms 2 35313 6.39 MB/sec execute 45 sec latency 35.219 ms 2 35930 6.45 MB/sec execute 46 sec latency 9.752 ms 2 36391 6.35 MB/sec execute 47 sec latency 43.336 ms 2 37019 6.40 MB/sec execute 48 sec latency 31.258 ms 2 37649 6.29 MB/sec execute 49 sec latency 33.644 ms 2 38429 6.44 MB/sec execute 50 sec latency 30.321 ms 2 39276 6.56 MB/sec execute 51 sec latency 37.850 ms 2 39757 6.45 MB/sec execute 52 sec latency 51.104 ms 2 40226 6.40 MB/sec execute 53 sec latency 36.661 ms 2 40857 6.40 MB/sec execute 54 sec latency 32.261 ms 2 41488 6.35 MB/sec execute 55 sec latency 38.942 ms 2 42425 6.50 MB/sec execute 56 sec latency 25.176 ms 2 43033 6.55 MB/sec execute 57 sec latency 10.323 ms 2 43489 6.46 MB/sec execute 58 sec latency 50.681 ms 2 44167 6.50 MB/sec execute 59 sec latency 32.844 ms 2 44789 6.42 MB/sec execute 60 sec latency 26.590 ms 2 cleanup 61 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12818 4.634 2023.873 Close 9446 0.985 3.291 Rename 540 5.962 9.638 Unlink 2532 1.723 3.103 Qpathinfo 11664 1.638 3293.659 Qfileinfo 2066 0.113 1.248 Qfsinfo 2084 0.075 1.370 Sfileinfo 1032 2.798 6.385 Find 4466 0.313 6.221 WriteX 6356 0.639 5.080 ReadX 20080 0.011 1.376 LockX 44 0.817 1.099 UnlockX 44 0.839 1.066 Flush 888 19.004 56.701 Throughput 6.41542 MB/sec 2 clients 2 procs max_latency=3293.676 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:36 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 26254 Clear changelog after use: no Errors: 0 lustre_rsync took 5 seconds Changelog records consumed: 11649 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0000: changelog user 'cl2' not found PASS 2b (82s) == lustre-rsync-test test complete, duration 177 sec ===== 17:45:43 (1748036743)