-----============= acceptance-small: lustre-rsync-test ============----- Fri May 23 17:42:51 EDT 2025 oleg446-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/lustre-rsync-test.*ex: No such file or directory excepting tests: 2b oleg446-client.virtnet: executing check_config_client /mnt/lustre oleg446-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg446-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5cc2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5cc2000.idle_timeout=debug disable quota as required oleg446-server: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 nobody:x:99: == lustre-rsync-test test 2a: Replicate files created by dbench. ========================================================== 17:43:03 (1748036583) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' lustre-MDT0000: Registered changelog user cl1 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2a.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:43:06 EDT 2025 waiting for dbench pid 8772 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs failed to create barrier semaphore 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 453 31.81 MB/sec warmup 1 sec latency 18.812 ms 2 990 21.91 MB/sec warmup 2 sec latency 57.628 ms 2 1680 17.48 MB/sec warmup 3 sec latency 31.347 ms 2 2239 13.39 MB/sec warmup 4 sec latency 20.525 ms 2 2996 13.41 MB/sec warmup 5 sec latency 48.539 ms 2 3922 13.35 MB/sec warmup 6 sec latency 22.426 ms 2 4477 11.67 MB/sec warmup 7 sec latency 38.434 ms 2 5252 11.33 MB/sec warmup 8 sec latency 24.623 ms 2 6037 10.63 MB/sec warmup 9 sec latency 22.552 ms 2 7232 11.57 MB/sec warmup 10 sec latency 23.253 ms 2 7825 10.81 MB/sec warmup 11 sec latency 49.281 ms 2 9296 1.48 MB/sec execute 1 sec latency 28.743 ms 2 10317 8.29 MB/sec execute 2 sec latency 26.554 ms 2 11106 9.36 MB/sec execute 3 sec latency 11.142 ms 2 11646 7.50 MB/sec execute 4 sec latency 40.013 ms 2 12443 7.76 MB/sec execute 5 sec latency 21.279 ms 2 13228 7.35 MB/sec execute 6 sec latency 29.434 ms 2 14337 9.10 MB/sec execute 7 sec latency 26.318 ms 2 14930 8.38 MB/sec execute 8 sec latency 37.082 ms 2 15612 8.50 MB/sec execute 9 sec latency 43.207 ms 2 16336 7.78 MB/sec execute 10 sec latency 23.921 ms 2 17361 8.43 MB/sec execute 11 sec latency 29.427 ms 2 18146 8.71 MB/sec execute 12 sec latency 19.901 ms 2 18683 8.17 MB/sec execute 13 sec latency 55.198 ms 2 19447 8.22 MB/sec execute 14 sec latency 31.793 ms 2 20203 7.93 MB/sec execute 15 sec latency 26.645 ms 2 21313 8.72 MB/sec execute 16 sec latency 25.261 ms 2 21899 8.41 MB/sec execute 17 sec latency 9.291 ms 2 22464 8.17 MB/sec execute 18 sec latency 39.231 ms 2 23185 8.09 MB/sec execute 19 sec latency 28.331 ms 2 24074 8.25 MB/sec execute 20 sec latency 29.139 ms 2 25107 8.60 MB/sec execute 21 sec latency 19.827 ms 2 25621 8.27 MB/sec execute 22 sec latency 57.765 ms 2 26339 8.32 MB/sec execute 23 sec latency 21.949 ms 2 27042 8.03 MB/sec execute 24 sec latency 41.991 ms 2 27890 8.25 MB/sec execute 25 sec latency 29.995 ms 2 28786 8.44 MB/sec execute 26 sec latency 20.341 ms 2 29303 8.18 MB/sec execute 27 sec latency 45.677 ms 2 30038 8.21 MB/sec execute 28 sec latency 27.446 ms 2 30815 8.05 MB/sec execute 29 sec latency 24.659 ms 2 31942 8.47 MB/sec execute 30 sec latency 26.051 ms 2 32569 8.31 MB/sec execute 31 sec latency 8.311 ms 2 33127 8.18 MB/sec execute 32 sec latency 58.924 ms 2 33859 8.14 MB/sec execute 33 sec latency 26.067 ms 2 34842 8.24 MB/sec execute 34 sec latency 24.154 ms 2 35775 8.44 MB/sec execute 35 sec latency 25.673 ms 2 36309 8.25 MB/sec execute 36 sec latency 37.846 ms 2 37017 8.27 MB/sec execute 37 sec latency 37.366 ms 2 37792 8.12 MB/sec execute 38 sec latency 30.513 ms 2 38779 8.32 MB/sec execute 39 sec latency 23.607 ms 2 39513 8.35 MB/sec execute 40 sec latency 10.748 ms 2 40050 8.20 MB/sec execute 41 sec latency 43.597 ms 2 40849 8.21 MB/sec execute 42 sec latency 24.630 ms 2 41624 8.14 MB/sec execute 43 sec latency 27.873 ms 2 42736 8.40 MB/sec execute 44 sec latency 28.345 ms 2 43312 8.29 MB/sec execute 45 sec latency 53.526 ms 2 43860 8.19 MB/sec execute 46 sec latency 29.627 ms 2 44605 8.16 MB/sec execute 47 sec latency 26.844 ms 2 45517 8.28 MB/sec execute 48 sec latency 29.104 ms 2 46432 8.37 MB/sec execute 49 sec latency 28.424 ms 2 46963 8.23 MB/sec execute 50 sec latency 38.528 ms 2 47716 8.25 MB/sec execute 51 sec latency 25.322 ms 2 48507 8.16 MB/sec execute 52 sec latency 28.712 ms 2 49514 8.29 MB/sec execute 53 sec latency 26.361 ms 2 50218 8.31 MB/sec execute 54 sec latency 8.226 ms 2 50797 8.23 MB/sec execute 55 sec latency 49.611 ms 2 51546 8.21 MB/sec execute 56 sec latency 32.335 ms 2 52445 8.26 MB/sec execute 57 sec latency 25.461 ms 2 53479 8.39 MB/sec execute 58 sec latency 21.410 ms 2 54034 8.27 MB/sec execute 59 sec latency 46.388 ms 2 cleanup 60 sec 0 cleanup 60 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 16014 3.741 11.137 Close 11753 0.839 2.305 Rename 676 5.278 8.363 Unlink 3228 1.492 2.507 Qpathinfo 14490 0.960 7.132 Qfileinfo 2529 0.104 1.082 Qfsinfo 2652 0.062 0.945 Sfileinfo 1300 2.429 6.582 Find 5598 0.291 5.378 WriteX 7895 0.565 6.648 ReadX 25056 0.009 0.134 LockX 52 0.729 1.623 UnlockX 52 0.755 1.502 Flush 1120 15.625 58.913 Throughput 8.27206 MB/sec 2 clients 2 procs max_latency=58.924 ms stopping dbench in /mnt/lustre/d2a.lustre-rsync-test at Fri May 23 17:44:18 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test /mnt/lustre/d2a.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2a.lustre-rsync-test dbench successfully finished Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 8 seconds Changelog records consumed: 22295 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0000: changelog user 'cl1' not found PASS 2a (85s) == lustre-rsync-test test 2b: Replicate files changed by dbench. ========================================================== 17:44:28 (1748036668) mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl2' lustre-MDT0000: Registered changelog user cl2 waiting child_pid=$(pgrep dbench), 360 secs left ... looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' copying required dbench files to chroot directory /mnt/lustre/d2b.lustre-rsync-test '/usr/bin/dbench' -> 'dbench' lib64/libpopt.so.0 lib64/libc.so.6 lib64/ld-linux-x86-64.so.2 running 'dbench 2 -t 60' in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:44:29 EDT 2025 waiting for dbench pid 9655 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 60 seconds with load 'client.txt' and minimum warmup 12 secs 0 of 2 processes prepared for launch 0 sec 2 of 2 processes prepared for launch 0 sec releasing clients 2 675 42.35 MB/sec warmup 1 sec latency 60.766 ms 2 1385 26.00 MB/sec warmup 2 sec latency 21.903 ms 2 2139 17.78 MB/sec warmup 3 sec latency 26.715 ms 2 3189 17.13 MB/sec warmup 4 sec latency 26.641 ms 2 3982 16.03 MB/sec warmup 5 sec latency 19.691 ms 2 4549 13.67 MB/sec warmup 6 sec latency 44.117 ms 2 5344 12.97 MB/sec warmup 7 sec latency 27.835 ms 2 6154 12.03 MB/sec warmup 8 sec latency 24.190 ms 2 7323 13.08 MB/sec warmup 9 sec latency 24.716 ms 2 7882 11.92 MB/sec warmup 10 sec latency 43.076 ms 2 8606 11.69 MB/sec warmup 11 sec latency 30.929 ms 2 10463 15.10 MB/sec execute 1 sec latency 26.363 ms 2 11181 12.20 MB/sec execute 2 sec latency 7.619 ms PIDs: 9655 9656 9657 Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 0 Clear changelog after use: no Errors: 0 lustre_rsync took 3 seconds Changelog records consumed: 4642 Resuming dbench 2 11472 4.80 MB/sec execute 5 sec latency 2749.832 ms 2 12245 5.52 MB/sec execute 6 sec latency 2745.262 ms 2 13050 5.27 MB/sec execute 7 sec latency 25.801 ms 2 14187 7.14 MB/sec execute 8 sec latency 22.381 ms 2 14850 6.76 MB/sec execute 9 sec latency 8.037 ms 2 15409 6.49 MB/sec execute 10 sec latency 43.881 ms 2 16179 6.52 MB/sec execute 11 sec latency 33.995 ms 2 17126 7.13 MB/sec execute 12 sec latency 28.846 ms 2 18093 7.58 MB/sec execute 13 sec latency 20.619 ms 2 18657 7.16 MB/sec execute 14 sec latency 39.838 ms 2 19403 7.28 MB/sec execute 15 sec latency 30.128 ms Stopping dbench Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 26937 Clear changelog after use: no Errors: 0 lustre_rsync took 1 seconds Changelog records consumed: 3189 Wait for dbench to finish 2 19405 6.60 MB/sec execute 17 sec latency 1575.473 ms 2 20197 6.44 MB/sec execute 18 sec latency 1575.629 ms 2 21333 7.20 MB/sec execute 19 sec latency 30.399 ms 2 21986 7.02 MB/sec execute 20 sec latency 10.555 ms 2 22521 6.87 MB/sec execute 21 sec latency 50.310 ms 2 23295 6.87 MB/sec execute 22 sec latency 31.700 ms 2 24245 7.18 MB/sec execute 23 sec latency 28.367 ms 2 25183 7.41 MB/sec execute 24 sec latency 21.266 ms 2 25758 7.19 MB/sec execute 25 sec latency 45.760 ms 2 26512 7.26 MB/sec execute 26 sec latency 37.190 ms 2 27278 7.13 MB/sec execute 27 sec latency 22.925 ms 2 28361 7.61 MB/sec execute 28 sec latency 28.174 ms 2 28969 7.47 MB/sec execute 29 sec latency 11.512 ms 2 29531 7.35 MB/sec execute 30 sec latency 42.485 ms 2 30273 7.34 MB/sec execute 31 sec latency 29.716 ms 2 31183 7.46 MB/sec execute 32 sec latency 23.874 ms 2 32211 7.71 MB/sec execute 33 sec latency 25.163 ms 2 32761 7.52 MB/sec execute 34 sec latency 45.746 ms 2 33480 7.57 MB/sec execute 35 sec latency 26.392 ms 2 34293 7.46 MB/sec execute 36 sec latency 28.545 ms 2 35295 7.67 MB/sec execute 37 sec latency 25.106 ms 2 35988 7.71 MB/sec execute 38 sec latency 11.743 ms 2 36547 7.62 MB/sec execute 39 sec latency 44.140 ms 2 37307 7.60 MB/sec execute 40 sec latency 30.815 ms 2 38083 7.54 MB/sec execute 41 sec latency 29.968 ms 2 39183 7.83 MB/sec execute 42 sec latency 24.920 ms 2 39756 7.72 MB/sec execute 43 sec latency 46.958 ms 2 40358 7.64 MB/sec execute 44 sec latency 27.791 ms 2 41148 7.62 MB/sec execute 45 sec latency 24.775 ms 2 42054 7.75 MB/sec execute 46 sec latency 26.497 ms 2 42974 7.86 MB/sec execute 47 sec latency 21.635 ms 2 43500 7.73 MB/sec execute 48 sec latency 42.631 ms 2 44236 7.76 MB/sec execute 49 sec latency 32.597 ms 2 45015 7.68 MB/sec execute 50 sec latency 23.578 ms 2 46144 7.93 MB/sec execute 51 sec latency 28.705 ms 2 46798 7.85 MB/sec execute 52 sec latency 7.957 ms 2 47345 7.77 MB/sec execute 53 sec latency 48.195 ms 2 48115 7.76 MB/sec execute 54 sec latency 33.727 ms 2 49024 7.82 MB/sec execute 55 sec latency 27.827 ms 2 49961 7.96 MB/sec execute 56 sec latency 21.587 ms 2 50501 7.84 MB/sec execute 57 sec latency 45.591 ms 2 51250 7.87 MB/sec execute 58 sec latency 34.244 ms 2 52043 7.80 MB/sec execute 59 sec latency 24.697 ms 2 53052 7.92 MB/sec execute 60 sec latency 28.477 ms 2 cleanup 61 sec 0 cleanup 61 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 15314 4.083 2749.822 Close 11168 0.828 2.344 Rename 654 5.182 8.772 Unlink 3106 1.476 2.700 Qpathinfo 13778 1.151 2745.251 Qfileinfo 2414 0.103 1.144 Qfsinfo 2576 0.061 0.895 Sfileinfo 1228 2.408 4.800 Find 5366 0.281 5.246 WriteX 7544 0.559 4.733 ReadX 24280 0.009 0.267 LockX 52 0.744 0.951 UnlockX 52 0.781 1.063 Flush 1058 15.417 50.302 Throughput 7.91817 MB/sec 2 clients 2 procs max_latency=2749.832 ms stopping dbench in /mnt/lustre/d2b.lustre-rsync-test at Fri May 23 17:45:43 EDT 2025 with return code 0 clean dbench files in /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test /mnt/lustre/d2b.lustre-rsync-test removed directory: 'clients/client0/~dmtmp/PARADOX' removed directory: 'clients/client0/~dmtmp/SEED' removed directory: 'clients/client0/~dmtmp/COREL' removed directory: 'clients/client0/~dmtmp/PWRPNT' removed directory: 'clients/client0/~dmtmp/WORD' removed directory: 'clients/client0/~dmtmp/WORDPRO' removed directory: 'clients/client0/~dmtmp/PM' removed directory: 'clients/client0/~dmtmp/EXCEL' removed directory: 'clients/client0/~dmtmp/ACCESS' removed directory: 'clients/client0/~dmtmp' removed directory: 'clients/client0' removed directory: 'clients/client1/~dmtmp/PARADOX' removed directory: 'clients/client1/~dmtmp/PWRPNT' removed directory: 'clients/client1/~dmtmp/PM' removed directory: 'clients/client1/~dmtmp/COREL' removed directory: 'clients/client1/~dmtmp/WORDPRO' removed directory: 'clients/client1/~dmtmp/WORD' removed directory: 'clients/client1/~dmtmp/SEED' removed directory: 'clients/client1/~dmtmp/ACCESS' removed directory: 'clients/client1/~dmtmp/EXCEL' removed directory: 'clients/client1/~dmtmp' removed directory: 'clients/client1' removed directory: 'clients' removed 'client.txt' removed 'dbench' removed 'lib64/libpopt.so.0' removed 'lib64/libc.so.6' removed 'lib64/ld-linux-x86-64.so.2' /mnt/lustre/d2b.lustre-rsync-test dbench successfully finished Starting replication Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Statuslog: /tmp/lustre_rsync.log Changelog registration: cl2 Starting changelog record: 30125 Clear changelog after use: no Errors: 0 lustre_rsync took 5 seconds Changelog records consumed: 14073 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 lustre-MDT0000: changelog user 'cl2' not found PASS 2b (82s) == lustre-rsync-test test complete, duration 179 sec ===== 17:45:50 (1748036750)