-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:55 EDT 2025 oleg128-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg128-client.virtnet: executing check_config_client /mnt/lustre oleg128-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg128-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6dba000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6dba000.idle_timeout=debug disable quota as required oleg128-server: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:05 (1748036585) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:07 EDT 2025 waiting for dbench pid 9627 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 535 37.48 MB/sec warmup 1 sec latency 14.098 ms 2 980 21.90 MB/sec warmup 2 sec latency 87.977 ms 2 1647 17.46 MB/sec warmup 3 sec latency 40.280 ms 2 2356 13.95 MB/sec warmup 4 sec latency 29.016 ms 2 3197 13.71 MB/sec warmup 5 sec latency 33.127 ms 2 3926 13.35 MB/sec warmup 6 sec latency 22.262 ms 2 4369 11.60 MB/sec warmup 7 sec latency 67.511 ms 2 5002 11.28 MB/sec warmup 8 sec latency 38.249 ms 2 5585 10.15 MB/sec warmup 9 sec latency 30.158 ms 2 6327 10.22 MB/sec warmup 10 sec latency 27.481 ms 2 7243 10.52 MB/sec warmup 11 sec latency 31.649 ms 2 8108 1.77 MB/sec execute 1 sec latency 60.928 ms 2 8761 5.15 MB/sec execute 2 sec latency 29.769 ms 2 9451 4.55 MB/sec execute 3 sec latency 32.571 ms 2 10124 6.24 MB/sec execute 4 sec latency 35.596 ms 2 10934 7.48 MB/sec execute 5 sec latency 23.955 ms 2 11382 6.45 MB/sec execute 6 sec latency 47.603 ms 2 11836 6.02 MB/sec execute 7 sec latency 42.195 ms 2 12458 6.10 MB/sec execute 8 sec latency 32.403 ms 2 13088 5.81 MB/sec execute 9 sec latency 33.541 ms 2 14001 6.69 MB/sec execute 10 sec latency 33.704 ms 2 14571 6.93 MB/sec execute 11 sec latency 12.970 ms 2 14999 6.44 MB/sec execute 12 sec latency 62.565 ms 2 15585 6.64 MB/sec execute 13 sec latency 35.022 ms 2 16132 6.23 MB/sec execute 14 sec latency 27.886 ms 2 16772 6.15 MB/sec execute 15 sec latency 31.990 ms 2 17766 6.95 MB/sec execute 16 sec latency 32.570 ms 2 18250 6.74 MB/sec execute 17 sec latency 10.666 ms 2 18681 6.45 MB/sec execute 18 sec latency 62.488 ms 2 19325 6.57 MB/sec execute 19 sec latency 31.967 ms 2 19947 6.31 MB/sec execute 20 sec latency 31.395 ms 2 20675 6.59 MB/sec execute 21 sec latency 33.209 ms 2 21455 6.83 MB/sec execute 22 sec latency 32.175 ms 2 21915 6.65 MB/sec execute 23 sec latency 12.437 ms 2 22315 6.45 MB/sec execute 24 sec latency 48.099 ms 2 22973 6.53 MB/sec execute 25 sec latency 37.817 ms 2 23663 6.41 MB/sec execute 26 sec latency 22.986 ms 2 24333 6.59 MB/sec execute 27 sec latency 32.005 ms 2 25144 6.80 MB/sec execute 28 sec latency 24.594 ms 2 25603 6.61 MB/sec execute 29 sec latency 56.966 ms 2 26071 6.51 MB/sec execute 30 sec latency 25.146 ms 2 26686 6.51 MB/sec execute 31 sec latency 40.395 ms 2 27310 6.42 MB/sec execute 32 sec latency 26.961 ms 2 28229 6.67 MB/sec execute 33 sec latency 31.130 ms 2 28811 6.74 MB/sec execute 34 sec latency 10.527 ms 2 29231 6.58 MB/sec execute 35 sec latency 57.285 ms 2 29829 6.65 MB/sec execute 36 sec latency 39.395 ms 2 30373 6.49 MB/sec execute 37 sec latency 37.142 ms 2 30993 6.46 MB/sec execute 38 sec latency 35.121 ms 2 31960 6.78 MB/sec execute 39 sec latency 35.073 ms 2 32416 6.69 MB/sec execute 40 sec latency 11.330 ms 2 32859 6.56 MB/sec execute 41 sec latency 64.038 ms 2 33470 6.61 MB/sec execute 42 sec latency 27.482 ms 2 34047 6.49 MB/sec execute 43 sec latency 30.797 ms 2 34739 6.58 MB/sec execute 44 sec latency 26.236 ms 2 35622 6.74 MB/sec execute 45 sec latency 30.046 ms 2 36072 6.65 MB/sec execute 46 sec latency 10.766 ms 2 36483 6.54 MB/sec execute 47 sec latency 53.185 ms 2 37117 6.59 MB/sec execute 48 sec latency 31.072 ms 2 37708 6.48 MB/sec execute 49 sec latency 32.711 ms 2 38435 6.62 MB/sec execute 50 sec latency 29.156 ms 2 39262 6.73 MB/sec execute 51 sec latency 29.869 ms 2 39736 6.62 MB/sec execute 52 sec latency 51.501 ms 2 40185 6.57 MB/sec execute 53 sec latency 51.773 ms 2 40831 6.57 MB/sec execute 54 sec latency 30.190 ms 2 41487 6.51 MB/sec execute 55 sec latency 33.646 ms 2 42418 6.66 MB/sec execute 56 sec latency 35.326 ms 2 43041 6.71 MB/sec execute 57 sec latency 10.118 ms 2 43485 6.62 MB/sec execute 58 sec latency 47.477 ms 2 44136 6.65 MB/sec execute 59 sec latency 27.297 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12894 4.639 13.572 Close 9522 1.064 12.115 Rename 542 6.488 19.991 Unlink 2536 1.852 12.945 Qpathinfo 11650 1.165 10.099 Qfileinfo 2036 0.125 1.322 Qfsinfo 2098 0.081 1.150 Sfileinfo 1066 3.059 5.815 Find 4476 0.353 7.168 WriteX 6354 0.691 6.891 ReadX 19918 0.013 1.297 LockX 40 0.884 1.041 UnlockX 40 0.937 1.264 Flush 920 19.348 64.028 Throughput 6.65329 MB/sec 2 clients 2 procs max_latency=64.038 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:19 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 7 seconds Changelog records consumed: 18310 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 2a (85s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:30 (1748036670) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:33 EDT 2025 waiting for dbench pid 10595 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 525 36.10 MB/sec warmup 1 sec latency 14.114 ms 2 971 21.81 MB/sec warmup 2 sec latency 33.594 ms 2 1624 17.44 MB/sec warmup 3 sec latency 32.739 ms 2 2291 13.88 MB/sec warmup 4 sec latency 29.088 ms 2 3000 13.41 MB/sec warmup 5 sec latency 36.230 ms 2 3846 13.25 MB/sec warmup 6 sec latency 25.320 ms 2 4303 11.55 MB/sec warmup 7 sec latency 66.439 ms 2 4799 10.55 MB/sec warmup 8 sec latency 41.390 ms 2 5426 10.11 MB/sec warmup 9 sec latency 26.328 ms 2 6049 9.57 MB/sec warmup 10 sec latency 35.827 ms 2 7084 10.45 MB/sec warmup 11 sec latency 30.852 ms 2 7985 1.43 MB/sec execute 1 sec latency 59.040 ms 2 8570 5.09 MB/sec execute 2 sec latency 29.998 ms PIDs: 10595 10596 10597 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 3574 Resuming dbench 2 8799 1.95 MB/sec execute 5 sec latency 2924.440 ms 2 9480 2.19 MB/sec execute 6 sec latency 2924.458 ms 2 10309 3.60 MB/sec execute 7 sec latency 35.780 ms 2 10981 4.51 MB/sec execute 8 sec latency 15.254 ms 2 11393 4.14 MB/sec execute 9 sec latency 60.800 ms 2 11857 4.08 MB/sec execute 10 sec latency 35.572 ms 2 12453 4.29 MB/sec execute 11 sec latency 35.970 ms 2 13071 4.23 MB/sec execute 12 sec latency 28.932 ms 2 13946 5.00 MB/sec execute 13 sec latency 36.746 ms 2 14559 5.31 MB/sec execute 14 sec latency 14.864 ms 2 14976 5.02 MB/sec execute 15 sec latency 65.038 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 21884 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 2483 Wait for dbench to finish 2 14977 4.57 MB/sec execute 17 sec latency 1526.771 ms 2 15498 4.55 MB/sec execute 18 sec latency 35.452 ms 2 16085 4.61 MB/sec execute 19 sec latency 35.798 ms 2 16673 4.62 MB/sec execute 20 sec latency 34.496 ms 2 17565 5.04 MB/sec execute 21 sec latency 34.789 ms 2 18150 5.23 MB/sec execute 22 sec latency 11.282 ms 2 18553 5.05 MB/sec execute 23 sec latency 78.926 ms 2 19169 5.22 MB/sec execute 24 sec latency 25.092 ms 2 19698 5.04 MB/sec execute 25 sec latency 33.445 ms 2 20275 5.03 MB/sec execute 26 sec latency 32.094 ms 2 21290 5.56 MB/sec execute 27 sec latency 31.403 ms 2 21760 5.48 MB/sec execute 28 sec latency 11.034 ms 2 22199 5.33 MB/sec execute 29 sec latency 57.911 ms 2 22812 5.45 MB/sec execute 30 sec latency 24.886 ms 2 23346 5.31 MB/sec execute 31 sec latency 30.970 ms 2 24092 5.48 MB/sec execute 32 sec latency 27.680 ms 2 24955 5.72 MB/sec execute 33 sec latency 31.780 ms 2 25409 5.64 MB/sec execute 34 sec latency 10.438 ms 2 25828 5.52 MB/sec execute 35 sec latency 59.611 ms 2 26465 5.61 MB/sec execute 36 sec latency 33.852 ms 2 27045 5.49 MB/sec execute 37 sec latency 36.249 ms 2 27800 5.70 MB/sec execute 38 sec latency 28.714 ms 2 28580 5.83 MB/sec execute 39 sec latency 32.191 ms 2 29079 5.75 MB/sec execute 40 sec latency 11.069 ms 2 29411 5.65 MB/sec execute 41 sec latency 58.142 ms 2 30055 5.72 MB/sec execute 42 sec latency 30.813 ms 2 30679 5.62 MB/sec execute 43 sec latency 28.629 ms 2 31393 5.80 MB/sec execute 44 sec latency 34.972 ms 2 32187 5.95 MB/sec execute 45 sec latency 27.135 ms 2 32641 5.84 MB/sec execute 46 sec latency 52.309 ms 2 33036 5.79 MB/sec execute 47 sec latency 80.059 ms 2 33656 5.81 MB/sec execute 48 sec latency 27.466 ms 2 34324 5.76 MB/sec execute 49 sec latency 28.392 ms 2 35133 5.89 MB/sec execute 50 sec latency 40.567 ms 2 35822 6.00 MB/sec execute 51 sec latency 23.606 ms 2 36288 5.91 MB/sec execute 52 sec latency 57.414 ms 2 36893 5.97 MB/sec execute 53 sec latency 37.158 ms 2 37454 5.88 MB/sec execute 54 sec latency 34.872 ms 2 38104 5.86 MB/sec execute 55 sec latency 30.462 ms 2 39137 6.10 MB/sec execute 56 sec latency 28.990 ms 2 39708 6.05 MB/sec execute 57 sec latency 9.596 ms 2 40146 6.01 MB/sec execute 58 sec latency 48.882 ms 2 40833 6.02 MB/sec execute 59 sec latency 33.095 ms 2 41513 5.98 MB/sec execute 60 sec latency 24.670 ms 2 cleanup 61 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 12004 5.403 2924.450 Close 8886 1.073 2.648 Rename 512 6.519 11.265 Unlink 2406 1.876 3.565 Qpathinfo 10964 1.157 7.563 Qfileinfo 1932 0.126 1.641 Qfsinfo 2010 0.082 1.145 Sfileinfo 1000 3.038 6.277 Find 4278 0.361 7.296 WriteX 5998 0.692 6.922 ReadX 19168 0.012 1.663 LockX 40 0.901 1.106 UnlockX 40 0.931 1.365 Flush 860 19.797 80.039 Throughput 5.98287 MB/sec 2 clients 2 procs max_latency=2924.458 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:46 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 24366 Clear changelog after use: no Errors: 0 lustre_rsync took 4 seconds Changelog records consumed: 11219 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0001: changelog user 'cl2' not found lustre-MDT0000: changelog user 'cl2' not found PASS 2b (84s) == lustre-rsync-test test complete, duration 179 sec ===== 17:45:54 (1748036754)