== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:27:36 (1788157656) Stopping clients: oleg129-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg129-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg129-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg129-server oleg129-server: oleg129-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: MODOPTS_LIBCFS= oleg129-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg129-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg129-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 889 (time 1788157873.94 total 10.01 last 88.84) - open/close 1740 (time 1788157883.94 total 20.01 last 85.09) - open/close 2640 (time 1788157893.96 total 30.04 last 89.75) - open/close 3661 (time 1788157903.97 total 40.05 last 101.99) total: 4500 open/close in 49.43 seconds: 91.04 ops/second - /unlink 1256 (time 1788157928.02 total 10.01 last 125.50) - /unlink 2561 (time 1788157938.02 total 20.01 last 130.48) - /unlink 3963 (time 1788157948.03 total 30.02 last 140.11) total: 4500 /unlink in 33.97 seconds: 132.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 814 (time 1788157966.87 total 10.01 last 81.34) - open/close 1621 (time 1788157976.88 total 20.01 last 80.67) - open/close 2441 (time 1788157986.90 total 30.03 last 81.81) - open/close 3240 (time 1788157996.90 total 40.04 last 79.87) - open/close 4070 (time 1788158006.92 total 50.05 last 82.89) total: 4500 open/close in 55.23 seconds: 81.48 ops/second - /unlink 1789 (time 1788158026.61 total 10.00 last 178.86) - /unlink 3355 (time 1788158036.61 total 20.01 last 156.56) total: 4500 /unlink in 28.34 seconds: 158.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 897 (time 1788158060.13 total 10.00 last 89.68) - open/close 1743 (time 1788158070.16 total 20.04 last 84.32) - open/close 2722 (time 1788158080.17 total 30.04 last 97.88) - open/close 3498 (time 1788158090.17 total 40.04 last 77.57) - open/close 4312 (time 1788158100.17 total 50.04 last 81.40) total: 4500 open/close in 52.47 seconds: 85.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1343 (time 1788158117.17 total 10.01 last 134.18) - /unlink 2797 (time 1788158127.18 total 20.01 last 145.36) - /unlink 4386 (time 1788158137.18 total 30.01 last 158.87) total: 4500 /unlink in 31.00 seconds: 145.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 840 (time 1788158152.68 total 10.01 last 83.90) - open/close 1536 (time 1788158162.69 total 20.02 last 69.51) - open/close 2343 (time 1788158172.70 total 30.03 last 80.65) - open/close 3136 (time 1788158182.71 total 40.05 last 79.17) - open/close 3817 (time 1788158192.72 total 50.05 last 68.05) total: 4500 open/close in 58.83 seconds: 76.49 ops/second - /unlink 1567 (time 1788158216.05 total 10.00 last 156.63) - /unlink 3213 (time 1788158226.06 total 20.02 last 164.40) total: 4500 /unlink in 29.41 seconds: 153.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 880 (time 1788158250.38 total 10.01 last 87.93) - open/close 1699 (time 1788158260.38 total 20.01 last 81.86) - open/close 2519 (time 1788158270.39 total 30.02 last 81.93) - open/close 3313 (time 1788158280.39 total 40.02 last 79.38) - open/close 4208 (time 1788158290.40 total 50.03 last 89.45) total: 4500 open/close in 53.61 seconds: 83.94 ops/second - /unlink 1736 (time 1788158309.04 total 10.00 last 173.53) - /unlink 3232 (time 1788158319.04 total 20.01 last 149.57) total: 4500 /unlink in 28.18 seconds: 159.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 788 (time 1788158342.42 total 10.01 last 78.74) - open/close 1663 (time 1788158352.43 total 20.01 last 87.46) - open/close 2704 (time 1788158362.43 total 30.02 last 104.06) - open/close 3615 (time 1788158372.44 total 40.03 last 90.99) - open/close 4382 (time 1788158382.45 total 50.03 last 76.67) total: 4500 open/close in 51.16 seconds: 87.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1515 (time 1788158397.46 total 10.00 last 151.45) - /unlink 2836 (time 1788158407.46 total 20.01 last 132.05) - /unlink 3971 (time 1788158417.48 total 30.02 last 113.30) total: 4500 /unlink in 34.12 seconds: 131.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 611 (time 1788158437.72 total 10.00 last 61.10) - open/close 1233 (time 1788158447.72 total 20.00 last 62.18) - open/close 1935 (time 1788158457.75 total 30.03 last 70.00) - open/close 2598 (time 1788158467.75 total 40.04 last 66.26) - open/close 3332 (time 1788158477.76 total 50.04 last 73.37) - open/close 4178 (time 1788158487.77 total 60.05 last 84.51) total: 4500 open/close in 64.03 seconds: 70.28 ops/second - /unlink 1278 (time 1788158505.99 total 10.00 last 127.76) - /unlink 2846 (time 1788158516.00 total 20.01 last 156.72) - /unlink 4380 (time 1788158526.00 total 30.01 last 153.33) total: 4500 /unlink in 30.96 seconds: 145.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 746 (time 1788158541.11 total 10.01 last 74.55) - open/close 1550 (time 1788158551.13 total 20.02 last 80.26) - open/close 2383 (time 1788158561.13 total 30.03 last 83.28) - open/close 3155 (time 1788158571.14 total 40.04 last 77.10) - open/close 3926 (time 1788158581.15 total 50.04 last 77.06) - open/close 4490 (time 1788158591.16 total 60.05 last 56.34) total: 4500 open/close in 60.33 seconds: 74.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1290 (time 1788158607.28 total 10.00 last 128.97) - /unlink 2602 (time 1788158617.28 total 20.01 last 131.12) - /unlink 3943 (time 1788158627.29 total 30.01 last 134.03) total: 4500 /unlink in 34.04 seconds: 132.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 760 (time 1788158646.15 total 10.00 last 75.96) - open/close 1565 (time 1788158656.16 total 20.01 last 80.46) - open/close 2391 (time 1788158666.16 total 30.01 last 82.59) - open/close 3127 (time 1788158676.17 total 40.02 last 73.52) - open/close 3936 (time 1788158686.18 total 50.03 last 80.84) total: 4500 open/close in 56.83 seconds: 79.18 ops/second - /unlink 1568 (time 1788158707.03 total 10.00 last 156.74) - /unlink 3112 (time 1788158717.04 total 20.01 last 154.36) total: 4500 /unlink in 29.80 seconds: 150.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 749 (time 1788158742.48 total 10.00 last 74.87) - open/close 1535 (time 1788158752.48 total 20.01 last 78.58) - open/close 2403 (time 1788158762.48 total 30.01 last 86.75) - open/close 3275 (time 1788158772.49 total 40.02 last 87.16) - open/close 4052 (time 1788158782.49 total 50.02 last 77.68) total: 4500 open/close in 56.27 seconds: 79.98 ops/second - /unlink 1257 (time 1788158803.44 total 10.01 last 125.62) - /unlink 2445 (time 1788158813.45 total 20.02 last 118.60) - /unlink 3927 (time 1788158823.47 total 30.04 last 147.98) total: 4500 /unlink in 34.15 seconds: 131.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 733 (time 1788158842.72 total 10.00 last 73.29) - open/close 1551 (time 1788158852.72 total 20.01 last 81.75) - open/close 2361 (time 1788158862.72 total 30.01 last 80.99) - open/close 3144 (time 1788158872.73 total 40.01 last 78.27) - open/close 4112 (time 1788158882.73 total 50.02 last 96.78) total: 4500 open/close in 56.19 seconds: 80.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1285 (time 1788158904.00 total 10.00 last 128.45) - /unlink 2754 (time 1788158914.03 total 20.03 last 146.56) - /unlink 4285 (time 1788158924.03 total 30.03 last 153.09) total: 4500 /unlink in 31.72 seconds: 141.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 912 (time 1788158939.86 total 10.00 last 91.19) - open/close 1755 (time 1788158949.89 total 20.03 last 84.03) - open/close 2611 (time 1788158959.89 total 30.04 last 85.56) - open/close 3537 (time 1788158969.89 total 40.04 last 92.59) - open/close 4360 (time 1788158979.90 total 50.05 last 82.21) total: 4500 open/close in 51.55 seconds: 87.29 ops/second - /unlink 1540 (time 1788158995.77 total 10.00 last 153.99) - /unlink 3190 (time 1788159005.77 total 20.00 last 165.00) total: 4500 /unlink in 28.55 seconds: 157.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 783 (time 1788159028.96 total 10.00 last 78.29) - open/close 1612 (time 1788159038.97 total 20.00 last 82.87) - open/close 2456 (time 1788159048.97 total 30.01 last 84.40) - open/close 3316 (time 1788159058.98 total 40.02 last 85.90) - open/close 4137 (time 1788159068.98 total 50.02 last 82.06) total: 4500 open/close in 54.76 seconds: 82.17 ops/second - /unlink 1422 (time 1788159088.57 total 10.00 last 142.19) - /unlink 2772 (time 1788159098.58 total 20.01 last 134.93) - /unlink 4366 (time 1788159108.58 total 30.01 last 159.39) total: 4500 /unlink in 30.86 seconds: 145.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 802 (time 1788159124.13 total 10.00 last 80.18) - open/close 1652 (time 1788159134.15 total 20.02 last 84.86) - open/close 2409 (time 1788159144.18 total 30.05 last 75.46) - open/close 3182 (time 1788159154.18 total 40.05 last 77.30) - open/close 3906 (time 1788159164.18 total 50.05 last 72.38) total: 4500 open/close in 57.07 seconds: 78.86 ops/second - /unlink 1647 (time 1788159185.68 total 10.00 last 164.68) - /unlink 3089 (time 1788159195.69 total 20.01 last 144.06) total: 4500 /unlink in 29.49 seconds: 152.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 665 (time 1788159220.35 total 10.01 last 66.45) - open/close 1380 (time 1788159230.35 total 20.01 last 71.48) - open/close 2196 (time 1788159240.37 total 30.03 last 81.42) - open/close 3042 (time 1788159250.37 total 40.04 last 84.58) - open/close 3871 (time 1788159260.38 total 50.04 last 82.85) total: 4500 open/close in 57.96 seconds: 77.64 ops/second - /unlink 1572 (time 1788159282.23 total 10.00 last 157.14) - /unlink 3077 (time 1788159292.23 total 20.01 last 150.45) - /unlink 4290 (time 1788159302.24 total 30.01 last 121.26) total: 4500 /unlink in 31.99 seconds: 140.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 904 (time 1788159321.52 total 10.01 last 90.33) - open/close 1667 (time 1788159331.52 total 20.01 last 76.28) - open/close 2371 (time 1788159341.53 total 30.02 last 70.36) - open/close 3091 (time 1788159351.53 total 40.02 last 71.95) - open/close 3791 (time 1788159361.54 total 50.03 last 69.96) - open/close 4447 (time 1788159371.55 total 60.04 last 65.54) total: 4500 open/close in 60.96 seconds: 73.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1229 (time 1788159387.13 total 10.01 last 122.80) - /unlink 2607 (time 1788159397.14 total 20.02 last 137.58) - /unlink 3953 (time 1788159407.15 total 30.03 last 134.55) total: 4500 /unlink in 33.71 seconds: 133.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 704 (time 1788159426.26 total 10.00 last 70.40) - open/close 1377 (time 1788159436.28 total 20.02 last 67.20) - open/close 2065 (time 1788159446.28 total 30.02 last 68.75) - open/close 2840 (time 1788159456.29 total 40.03 last 77.47) - open/close 3646 (time 1788159466.29 total 50.03 last 80.55) - open/close 4469 (time 1788159476.30 total 60.04 last 82.24) total: 4500 open/close in 60.31 seconds: 74.62 ops/second - /unlink 1406 (time 1788159490.95 total 10.00 last 140.55) - /unlink 2840 (time 1788159500.95 total 20.01 last 143.37) - /unlink 4256 (time 1788159510.96 total 30.01 last 141.59) total: 4500 /unlink in 31.69 seconds: 142.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 846 (time 1788159527.69 total 10.01 last 84.56) - open/close 1615 (time 1788159537.69 total 20.01 last 76.88) - open/close 2321 (time 1788159547.71 total 30.03 last 70.46) - open/close 3085 (time 1788159557.72 total 40.04 last 76.33) - open/close 3814 (time 1788159567.73 total 50.04 last 72.86) total: 4500 open/close in 59.39 seconds: 75.77 ops/second - /unlink 1566 (time 1788159592.49 total 10.00 last 156.59) - /unlink 3145 (time 1788159602.50 total 20.01 last 157.82) total: 4500 /unlink in 29.35 seconds: 153.32 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95246 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg129-server unloading modules via unload_modules_local on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local oleg129-server: modules unloaded.