== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:52:12 (1783446732) Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg227-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg227-server oleg227-server: oleg227-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg227-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg227-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 930 (time 1783446877.47 total 10.01 last 92.92) - open/close 1868 (time 1783446887.48 total 20.01 last 93.75) - open/close 2904 (time 1783446897.49 total 30.02 last 103.52) - open/close 3970 (time 1783446907.49 total 40.02 last 106.57) total: 4500 open/close in 45.57 seconds: 98.75 ops/second - /unlink 2045 (time 1783446925.47 total 10.00 last 204.48) - /unlink 4073 (time 1783446935.48 total 20.01 last 202.62) total: 4500 /unlink in 22.20 seconds: 202.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 995 (time 1783446950.25 total 10.01 last 99.43) - open/close 1970 (time 1783446960.26 total 20.02 last 97.41) - open/close 3112 (time 1783446970.26 total 30.02 last 114.13) - open/close 4332 (time 1783446980.27 total 40.03 last 121.89) total: 4500 open/close in 41.55 seconds: 108.29 ops/second - /unlink 1858 (time 1783446994.62 total 10.00 last 185.76) - /unlink 3745 (time 1783447004.62 total 20.00 last 188.65) total: 4500 /unlink in 24.64 seconds: 182.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 861 (time 1783447022.43 total 10.01 last 86.05) - open/close 1799 (time 1783447032.44 total 20.01 last 93.74) - open/close 2716 (time 1783447042.44 total 30.01 last 91.70) - open/close 3714 (time 1783447052.44 total 40.02 last 99.76) total: 4500 open/close in 46.96 seconds: 95.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 2035 (time 1783447071.87 total 10.00 last 203.47) - /unlink 4020 (time 1783447081.88 total 20.01 last 198.41) total: 4500 /unlink in 22.31 seconds: 201.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 1022 (time 1783447097.75 total 10.00 last 102.16) - open/close 2133 (time 1783447107.75 total 20.01 last 111.04) - open/close 3196 (time 1783447117.76 total 30.01 last 106.25) - open/close 4279 (time 1783447127.77 total 40.02 last 108.22) total: 4500 open/close in 42.37 seconds: 106.21 ops/second - /unlink 1804 (time 1783447143.05 total 10.00 last 180.39) - /unlink 3778 (time 1783447153.05 total 20.00 last 197.37) total: 4500 /unlink in 23.54 seconds: 191.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 1120 (time 1783447169.79 total 10.01 last 111.92) - open/close 2169 (time 1783447179.79 total 20.01 last 104.85) - open/close 3148 (time 1783447189.80 total 30.02 last 97.81) - open/close 4338 (time 1783447199.81 total 40.03 last 118.91) total: 4500 open/close in 41.32 seconds: 108.92 ops/second - /unlink 1898 (time 1783447213.89 total 10.01 last 189.68) - /unlink 3878 (time 1783447223.89 total 20.01 last 197.98) total: 4500 /unlink in 23.01 seconds: 195.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 1051 (time 1783447239.65 total 10.00 last 105.08) - open/close 2060 (time 1783447249.66 total 20.01 last 100.86) - open/close 3049 (time 1783447259.66 total 30.01 last 98.83) - open/close 4050 (time 1783447269.67 total 40.02 last 100.06) total: 4500 open/close in 45.15 seconds: 99.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1798 (time 1783447288.33 total 10.01 last 179.65) - /unlink 3909 (time 1783447298.33 total 20.01 last 211.03) total: 4500 /unlink in 22.96 seconds: 196.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 1219 (time 1783447314.28 total 10.00 last 121.88) - open/close 2174 (time 1783447324.28 total 20.01 last 95.43) - open/close 3128 (time 1783447334.29 total 30.01 last 95.35) - open/close 4086 (time 1783447344.29 total 40.02 last 95.75) total: 4500 open/close in 44.73 seconds: 100.61 ops/second - /unlink 1822 (time 1783447362.12 total 10.00 last 182.18) - /unlink 3649 (time 1783447372.12 total 20.00 last 182.64) total: 4500 /unlink in 24.43 seconds: 184.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 890 (time 1783447389.59 total 10.00 last 88.96) - open/close 1903 (time 1783447399.60 total 20.01 last 101.21) - open/close 2944 (time 1783447409.61 total 30.02 last 104.06) - open/close 4035 (time 1783447419.61 total 40.02 last 109.07) total: 4500 open/close in 43.52 seconds: 103.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 2090 (time 1783447435.70 total 10.00 last 208.99) - /unlink 4138 (time 1783447445.70 total 20.00 last 204.78) total: 4500 /unlink in 21.73 seconds: 207.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1099 (time 1783447460.01 total 10.00 last 109.87) - open/close 2241 (time 1783447470.01 total 20.01 last 114.12) - open/close 3566 (time 1783447480.02 total 30.01 last 132.45) total: 4500 open/close in 36.79 seconds: 122.30 ops/second - /unlink 2270 (time 1783447499.07 total 10.01 last 226.73) total: 4500 /unlink in 19.90 seconds: 226.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1248 (time 1783447521.66 total 10.01 last 124.68) - open/close 2514 (time 1783447531.66 total 20.01 last 126.59) - open/close 3766 (time 1783447541.67 total 30.02 last 125.14) total: 4500 open/close in 36.88 seconds: 122.02 ops/second - /unlink 2052 (time 1783447561.26 total 10.01 last 205.09) - /unlink 3949 (time 1783447571.27 total 20.01 last 189.52) total: 4500 /unlink in 22.97 seconds: 195.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1226 (time 1783447587.30 total 10.00 last 122.56) - open/close 2439 (time 1783447597.31 total 20.01 last 121.22) - open/close 3706 (time 1783447607.32 total 30.02 last 126.63) total: 4500 open/close in 36.13 seconds: 124.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2352 (time 1783447625.87 total 10.00 last 235.18) total: 4500 /unlink in 19.82 seconds: 227.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1140 (time 1783447648.29 total 10.01 last 113.94) - open/close 2141 (time 1783447658.30 total 20.01 last 100.05) - open/close 3124 (time 1783447668.30 total 30.01 last 98.28) - open/close 4248 (time 1783447678.30 total 40.02 last 112.34) total: 4500 open/close in 42.40 seconds: 106.14 ops/second - /unlink 1892 (time 1783447693.42 total 10.00 last 189.19) - /unlink 3994 (time 1783447703.42 total 20.00 last 210.19) total: 4500 /unlink in 22.68 seconds: 198.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1166 (time 1783447719.07 total 10.01 last 116.48) - open/close 2463 (time 1783447729.09 total 20.03 last 129.50) - open/close 3662 (time 1783447739.10 total 30.04 last 119.72) total: 4500 open/close in 39.60 seconds: 113.64 ops/second - /unlink 787 (time 1783447764.97 total 10.01 last 78.63) - /unlink 1385 (time 1783447774.98 total 20.01 last 59.77) - /unlink 2342 (time 1783447784.98 total 30.01 last 95.69) - /unlink 3402 (time 1783447794.99 total 40.03 last 105.89) - /unlink 4335 (time 1783447804.99 total 50.03 last 93.26) total: 4500 /unlink in 51.71 seconds: 87.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 783 (time 1783447822.17 total 10.01 last 78.25) - open/close 1464 (time 1783447832.18 total 20.02 last 68.01) - open/close 2021 (time 1783447842.21 total 30.05 last 55.55) - open/close 2834 (time 1783447852.21 total 40.06 last 81.25) - open/close 3613 (time 1783447862.22 total 50.06 last 77.84) - open/close 4430 (time 1783447872.23 total 60.07 last 81.65) total: 4500 open/close in 60.87 seconds: 73.93 ops/second - /unlink 1528 (time 1783447886.75 total 10.00 last 152.77) - /unlink 2940 (time 1783447896.76 total 20.01 last 141.12) - /unlink 4291 (time 1783447906.76 total 30.01 last 135.05) total: 4500 /unlink in 31.72 seconds: 141.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 695 (time 1783447922.68 total 10.00 last 69.48) - open/close 1426 (time 1783447932.68 total 20.00 last 73.08) - open/close 2119 (time 1783447942.68 total 30.01 last 69.29) - open/close 2841 (time 1783447952.69 total 40.01 last 72.17) - open/close 3595 (time 1783447962.69 total 50.01 last 75.37) - open/close 4275 (time 1783447972.69 total 60.02 last 67.99) total: 4500 open/close in 62.89 seconds: 71.55 ops/second - /unlink 1459 (time 1783447989.80 total 10.00 last 145.88) - /unlink 2813 (time 1783447999.80 total 20.00 last 135.36) - /unlink 4125 (time 1783448009.81 total 30.01 last 131.17) total: 4500 /unlink in 33.07 seconds: 136.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 709 (time 1783448027.41 total 10.00 last 70.90) - open/close 1335 (time 1783448037.43 total 20.01 last 62.53) - open/close 2026 (time 1783448047.45 total 30.04 last 68.94) - open/close 2658 (time 1783448057.45 total 40.04 last 63.17) - open/close 3405 (time 1783448067.46 total 50.05 last 74.66) - open/close 4144 (time 1783448077.47 total 60.06 last 73.83) total: 4500 open/close in 65.02 seconds: 69.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1470 (time 1783448096.19 total 10.01 last 146.86) - /unlink 2792 (time 1783448106.19 total 20.01 last 132.16) - /unlink 4175 (time 1783448116.20 total 30.01 last 138.27) total: 4500 /unlink in 33.07 seconds: 136.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 593 (time 1783448135.45 total 10.01 last 59.23) - open/close 1204 (time 1783448145.47 total 20.03 last 60.97) - open/close 2020 (time 1783448155.49 total 30.05 last 81.45) - open/close 2751 (time 1783448165.49 total 40.05 last 73.08) - open/close 3430 (time 1783448175.50 total 50.06 last 67.90) - open/close 4183 (time 1783448185.50 total 60.06 last 75.27) total: 4500 open/close in 64.00 seconds: 70.31 ops/second - /unlink 1570 (time 1783448203.63 total 10.01 last 156.92) - /unlink 2772 (time 1783448213.63 total 20.01 last 120.18) - /unlink 4032 (time 1783448223.63 total 30.01 last 125.98) total: 4500 /unlink in 33.37 seconds: 134.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 683 (time 1783448241.45 total 10.01 last 68.20) - open/close 1437 (time 1783448251.46 total 20.02 last 75.34) - open/close 2159 (time 1783448261.48 total 30.04 last 72.04) - open/close 2924 (time 1783448271.50 total 40.06 last 76.39) - open/close 3632 (time 1783448281.51 total 50.07 last 70.74) - open/close 4407 (time 1783448291.52 total 60.08 last 77.43) total: 4500 open/close in 61.19 seconds: 73.55 ops/second - /unlink 1417 (time 1783448306.99 total 10.00 last 141.67) - /unlink 2858 (time 1783448317.01 total 20.02 last 143.86) - /unlink 4158 (time 1783448327.01 total 30.02 last 129.99) total: 4500 /unlink in 33.34 seconds: 134.95 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94749 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local oleg227-server: modules unloaded.