== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:50:58 (1779994258) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg339-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg339-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg339-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg339-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 829 (time 1779994450.52 total 10.00 last 82.86) - open/close 1698 (time 1779994460.53 total 20.01 last 86.84) - open/close 2543 (time 1779994470.55 total 30.03 last 84.31) - open/close 3351 (time 1779994480.56 total 40.04 last 80.77) - open/close 4083 (time 1779994490.56 total 50.04 last 73.15) total: 4500 open/close in 55.10 seconds: 81.67 ops/second - /unlink 1186 (time 1779994509.89 total 10.01 last 118.43) - /unlink 2480 (time 1779994519.90 total 20.02 last 129.28) - /unlink 3731 (time 1779994529.90 total 30.03 last 125.05) total: 4500 /unlink in 36.78 seconds: 122.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 590 (time 1779994551.58 total 10.01 last 58.96) - open/close 1367 (time 1779994561.58 total 20.01 last 77.70) - open/close 2143 (time 1779994571.59 total 30.02 last 77.49) - open/close 2896 (time 1779994581.61 total 40.04 last 75.17) - open/close 3626 (time 1779994591.62 total 50.05 last 72.95) - open/close 4444 (time 1779994601.62 total 60.05 last 81.75) total: 4500 open/close in 60.75 seconds: 74.08 ops/second - /unlink 1494 (time 1779994616.47 total 10.00 last 149.34) - /unlink 2978 (time 1779994626.47 total 20.01 last 148.35) - /unlink 4389 (time 1779994636.47 total 30.01 last 141.07) total: 4500 /unlink in 30.96 seconds: 145.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 858 (time 1779994652.01 total 10.01 last 85.70) - open/close 1651 (time 1779994662.01 total 20.02 last 79.27) - open/close 2487 (time 1779994672.02 total 30.03 last 83.52) - open/close 3362 (time 1779994682.03 total 40.03 last 87.43) - open/close 4220 (time 1779994692.04 total 50.04 last 85.70) total: 4500 open/close in 53.50 seconds: 84.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1278 (time 1779994710.39 total 10.01 last 127.66) - /unlink 2482 (time 1779994720.40 total 20.01 last 120.40) - /unlink 3798 (time 1779994730.40 total 30.01 last 131.58) total: 4500 /unlink in 34.71 seconds: 129.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 802 (time 1779994749.37 total 10.02 last 80.08) - open/close 1621 (time 1779994759.38 total 20.03 last 81.78) - open/close 2384 (time 1779994769.40 total 30.05 last 76.18) - open/close 3187 (time 1779994779.40 total 40.05 last 80.25) - open/close 3886 (time 1779994789.41 total 50.06 last 69.88) total: 4500 open/close in 57.72 seconds: 77.96 ops/second - /unlink 1447 (time 1779994810.95 total 10.00 last 144.66) - /unlink 2804 (time 1779994820.96 total 20.02 last 135.51) - /unlink 4172 (time 1779994830.96 total 30.02 last 136.79) total: 4500 /unlink in 32.71 seconds: 137.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 849 (time 1779994847.57 total 10.01 last 84.85) - open/close 1642 (time 1779994857.57 total 20.01 last 79.30) - open/close 2421 (time 1779994867.57 total 30.01 last 77.83) - open/close 3260 (time 1779994877.58 total 40.02 last 83.87) - open/close 3793 (time 1779994887.59 total 50.03 last 53.23) - open/close 4414 (time 1779994897.59 total 60.04 last 62.07) total: 4500 open/close in 60.99 seconds: 73.78 ops/second - /unlink 1352 (time 1779994911.64 total 10.00 last 135.19) - /unlink 2731 (time 1779994921.64 total 20.01 last 137.84) - /unlink 4157 (time 1779994931.64 total 30.01 last 142.57) total: 4500 /unlink in 32.69 seconds: 137.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 826 (time 1779994948.84 total 10.01 last 82.53) - open/close 1645 (time 1779994958.85 total 20.01 last 81.85) - open/close 2416 (time 1779994968.86 total 30.02 last 77.04) - open/close 3254 (time 1779994978.86 total 40.03 last 83.74) - open/close 4092 (time 1779994988.86 total 50.03 last 83.80) total: 4500 open/close in 54.71 seconds: 82.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1489 (time 1779995007.31 total 10.00 last 148.90) - /unlink 2702 (time 1779995017.31 total 20.00 last 121.29) - /unlink 4038 (time 1779995027.31 total 30.00 last 133.58) total: 4500 /unlink in 33.58 seconds: 134.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 862 (time 1779995044.92 total 10.00 last 86.18) - open/close 1561 (time 1779995054.92 total 20.00 last 69.89) - open/close 2354 (time 1779995064.93 total 30.01 last 79.26) - open/close 3068 (time 1779995074.94 total 40.02 last 71.28) - open/close 3631 (time 1779995084.96 total 50.04 last 56.23) - open/close 4364 (time 1779995094.96 total 60.04 last 73.29) total: 4500 open/close in 61.75 seconds: 72.88 ops/second - /unlink 1411 (time 1779995110.81 total 10.01 last 140.96) - /unlink 2666 (time 1779995120.81 total 20.01 last 125.45) - /unlink 3958 (time 1779995130.81 total 30.02 last 129.16) total: 4500 /unlink in 34.03 seconds: 132.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 764 (time 1779995151.27 total 10.01 last 76.35) - open/close 1504 (time 1779995161.29 total 20.03 last 73.84) - open/close 2349 (time 1779995171.30 total 30.03 last 84.46) - open/close 3187 (time 1779995181.30 total 40.04 last 83.77) - open/close 3897 (time 1779995191.31 total 50.04 last 70.97) total: 4500 open/close in 58.29 seconds: 77.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1312 (time 1779995214.72 total 10.01 last 131.06) - /unlink 2419 (time 1779995224.73 total 20.02 last 110.57) - /unlink 3510 (time 1779995234.73 total 30.02 last 109.09) total: 4500 /unlink in 38.51 seconds: 116.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 699 (time 1779995257.44 total 10.01 last 69.85) - open/close 1406 (time 1779995267.44 total 20.01 last 70.69) - open/close 2246 (time 1779995277.45 total 30.01 last 83.96) - open/close 3061 (time 1779995287.46 total 40.02 last 81.45) - open/close 3828 (time 1779995297.46 total 50.03 last 76.66) total: 4500 open/close in 58.61 seconds: 76.78 ops/second - /unlink 1543 (time 1779995320.29 total 10.00 last 154.29) - /unlink 2835 (time 1779995330.30 total 20.00 last 129.16) - /unlink 4186 (time 1779995340.30 total 30.01 last 135.00) total: 4500 /unlink in 32.11 seconds: 140.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 806 (time 1779995356.68 total 10.00 last 80.58) - open/close 1578 (time 1779995366.69 total 20.02 last 77.10) - open/close 2276 (time 1779995376.69 total 30.02 last 69.77) - open/close 3076 (time 1779995386.69 total 40.02 last 80.00) - open/close 3879 (time 1779995396.70 total 50.02 last 80.27) total: 4500 open/close in 57.22 seconds: 78.64 ops/second - /unlink 1469 (time 1779995418.31 total 10.01 last 146.74) - /unlink 2853 (time 1779995428.32 total 20.01 last 138.35) - /unlink 4262 (time 1779995438.32 total 30.01 last 140.90) total: 4500 /unlink in 31.95 seconds: 140.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 842 (time 1779995454.92 total 10.00 last 84.19) - open/close 1730 (time 1779995464.92 total 20.00 last 88.80) - open/close 2551 (time 1779995474.93 total 30.01 last 82.03) - open/close 3349 (time 1779995484.94 total 40.01 last 79.78) - open/close 4117 (time 1779995494.94 total 50.01 last 76.78) total: 4500 open/close in 54.76 seconds: 82.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1419 (time 1779995513.93 total 10.01 last 141.74) - /unlink 2817 (time 1779995523.94 total 20.02 last 139.74) - /unlink 4259 (time 1779995533.94 total 30.02 last 144.19) total: 4500 /unlink in 31.68 seconds: 142.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 791 (time 1779995549.62 total 10.00 last 79.07) - open/close 1572 (time 1779995559.63 total 20.01 last 78.05) - open/close 2321 (time 1779995569.64 total 30.02 last 74.84) - open/close 2961 (time 1779995579.64 total 40.02 last 63.99) - open/close 3566 (time 1779995589.64 total 50.02 last 60.49) - open/close 4096 (time 1779995599.64 total 60.02 last 52.99) total: 4500 open/close in 66.11 seconds: 68.07 ops/second - /unlink 1348 (time 1779995620.09 total 10.00 last 134.80) - /unlink 2610 (time 1779995630.10 total 20.01 last 126.13) - /unlink 3890 (time 1779995640.11 total 30.02 last 127.86) total: 4500 /unlink in 34.65 seconds: 129.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 651 (time 1779995659.13 total 10.00 last 65.10) - open/close 1386 (time 1779995669.15 total 20.02 last 73.35) - open/close 2079 (time 1779995679.16 total 30.02 last 69.29) - open/close 2672 (time 1779995689.17 total 40.03 last 59.24) - open/close 3351 (time 1779995699.19 total 50.05 last 67.76) - open/close 3955 (time 1779995709.20 total 60.06 last 60.35) total: 4500 open/close in 69.67 seconds: 64.59 ops/second - /unlink 1186 (time 1779995734.09 total 10.03 last 118.28) - /unlink 2266 (time 1779995744.09 total 20.03 last 108.00) - /unlink 3419 (time 1779995754.09 total 30.03 last 115.27) total: 4500 /unlink in 39.22 seconds: 114.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 671 (time 1779995778.98 total 10.02 last 66.97) - open/close 1407 (time 1779995789.00 total 20.04 last 73.46) - open/close 2254 (time 1779995799.00 total 30.04 last 84.69) - open/close 3115 (time 1779995809.01 total 40.04 last 86.09) - open/close 3901 (time 1779995819.02 total 50.05 last 78.50) total: 4500 open/close in 57.85 seconds: 77.79 ops/second - /unlink 1424 (time 1779995841.44 total 10.01 last 142.21) - /unlink 2792 (time 1779995851.46 total 20.03 last 136.60) - /unlink 4060 (time 1779995861.46 total 30.04 last 126.71) total: 4500 /unlink in 32.94 seconds: 136.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 808 (time 1779995879.04 total 10.01 last 80.74) - open/close 1578 (time 1779995889.05 total 20.02 last 76.93) - open/close 2285 (time 1779995899.06 total 30.03 last 70.62) - open/close 2981 (time 1779995909.07 total 40.03 last 69.57) - open/close 3556 (time 1779995919.08 total 50.05 last 57.42) - open/close 4240 (time 1779995929.09 total 60.05 last 68.36) total: 4500 open/close in 63.35 seconds: 71.04 ops/second - /unlink 1357 (time 1779995946.98 total 10.00 last 135.69) - /unlink 2557 (time 1779995956.98 total 20.00 last 119.97) - /unlink 3819 (time 1779995966.99 total 30.01 last 126.13) total: 4500 /unlink in 34.81 seconds: 129.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 758 (time 1779995986.69 total 10.00 last 75.77) - open/close 1483 (time 1779995996.71 total 20.02 last 72.37) - open/close 2230 (time 1779996006.71 total 30.02 last 74.69) - open/close 2840 (time 1779996016.76 total 40.07 last 60.72) - open/close 3424 (time 1779996026.76 total 50.08 last 58.37) - open/close 4026 (time 1779996036.77 total 60.08 last 60.16) total: 4500 open/close in 67.15 seconds: 67.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1395 (time 1779996057.95 total 10.00 last 139.47) - /unlink 2713 (time 1779996067.95 total 20.01 last 131.76) - /unlink 4008 (time 1779996077.96 total 30.01 last 129.40) total: 4500 /unlink in 36.11 seconds: 124.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 580 (time 1779996099.71 total 10.01 last 57.97) - open/close 1183 (time 1779996109.71 total 20.01 last 60.29) - open/close 1944 (time 1779996119.73 total 30.02 last 75.98) - open/close 2628 (time 1779996129.73 total 40.02 last 68.39) - open/close 3354 (time 1779996139.73 total 50.03 last 72.57) - open/close 4072 (time 1779996149.74 total 60.04 last 71.75) total: 4500 open/close in 65.99 seconds: 68.20 ops/second - /unlink 1018 (time 1779996170.19 total 10.00 last 101.77) - /unlink 2206 (time 1779996180.19 total 20.01 last 118.77) - /unlink 3469 (time 1779996190.19 total 30.01 last 126.30) total: 4500 /unlink in 37.75 seconds: 119.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 736 (time 1779996212.51 total 10.01 last 73.54) - open/close 1495 (time 1779996222.52 total 20.02 last 75.85) - open/close 2252 (time 1779996232.52 total 30.02 last 75.67) - open/close 3037 (time 1779996242.53 total 40.03 last 78.42) - open/close 3828 (time 1779996252.53 total 50.03 last 79.08) total: 4500 open/close in 58.20 seconds: 77.32 ops/second - /unlink 1404 (time 1779996274.91 total 10.00 last 140.40) - /unlink 2789 (time 1779996284.92 total 20.00 last 138.45) - /unlink 4094 (time 1779996294.92 total 30.00 last 130.49) total: 4500 /unlink in 33.34 seconds: 134.96 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11166: 94055 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local oleg339-server: modules unloaded.