== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:20:54 (1788906054) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg322-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg322-server oleg322-server: oleg322-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg322-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg322-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 933 (time 1788906200.51 total 10.01 last 93.23) - open/close 1881 (time 1788906210.53 total 20.02 last 94.66) - open/close 2930 (time 1788906220.53 total 30.02 last 104.90) - open/close 3907 (time 1788906230.54 total 40.03 last 97.62) total: 4500 open/close in 45.89 seconds: 98.05 ops/second - /unlink 1943 (time 1788906248.80 total 10.00 last 194.25) - /unlink 3703 (time 1788906258.81 total 20.00 last 175.97) total: 4500 /unlink in 24.59 seconds: 183.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 952 (time 1788906276.17 total 10.00 last 95.18) - open/close 1904 (time 1788906286.19 total 20.01 last 95.09) - open/close 2915 (time 1788906296.19 total 30.02 last 101.02) - open/close 3937 (time 1788906306.20 total 40.03 last 102.15) total: 4500 open/close in 45.42 seconds: 99.07 ops/second - /unlink 2209 (time 1788906324.03 total 10.00 last 220.86) - /unlink 4127 (time 1788906334.03 total 20.01 last 191.73) total: 4500 /unlink in 21.96 seconds: 204.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 1022 (time 1788906348.71 total 10.00 last 102.19) - open/close 1961 (time 1788906358.71 total 20.01 last 93.86) - open/close 2937 (time 1788906368.72 total 30.01 last 97.55) - open/close 3832 (time 1788906378.72 total 40.01 last 89.47) total: 4500 open/close in 46.98 seconds: 95.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1853 (time 1788906398.31 total 10.00 last 185.30) - /unlink 3771 (time 1788906408.31 total 20.00 last 191.74) total: 4500 /unlink in 23.45 seconds: 191.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 832 (time 1788906424.55 total 10.01 last 83.08) - open/close 1335 (time 1788906434.56 total 20.02 last 50.27) - open/close 1888 (time 1788906444.56 total 30.02 last 55.28) - open/close 2412 (time 1788906454.59 total 40.05 last 52.25) - open/close 2828 (time 1788906464.61 total 50.07 last 41.53) - open/close 3166 (time 1788906474.64 total 60.10 last 33.70) - open/close 3542 (time 1788906484.64 total 70.10 last 37.60) - open/close 4164 (time 1788906494.64 total 80.10 last 62.16) total: 4500 open/close in 84.86 seconds: 53.03 ops/second - /unlink 1263 (time 1788906513.90 total 10.00 last 126.24) - /unlink 2383 (time 1788906523.92 total 20.02 last 111.77) - /unlink 3113 (time 1788906533.93 total 30.03 last 72.94) - /unlink 3893 (time 1788906543.94 total 40.04 last 77.98) total: 4500 /unlink in 46.22 seconds: 97.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 701 (time 1788906564.63 total 10.01 last 70.04) - open/close 1411 (time 1788906574.64 total 20.02 last 70.94) - open/close 2140 (time 1788906584.65 total 30.03 last 72.83) - open/close 2862 (time 1788906594.65 total 40.03 last 72.19) - open/close 3551 (time 1788906604.66 total 50.04 last 68.85) - open/close 4218 (time 1788906614.66 total 60.04 last 66.70) total: 4500 open/close in 63.94 seconds: 70.38 ops/second - /unlink 1304 (time 1788906632.63 total 10.01 last 130.24) - /unlink 2578 (time 1788906642.63 total 20.01 last 127.38) - /unlink 3884 (time 1788906652.63 total 30.02 last 130.56) total: 4500 /unlink in 34.67 seconds: 129.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 657 (time 1788906671.37 total 10.01 last 65.64) - open/close 1297 (time 1788906681.38 total 20.02 last 63.95) - open/close 1888 (time 1788906691.39 total 30.03 last 59.06) - open/close 2555 (time 1788906701.40 total 40.03 last 66.65) - open/close 3243 (time 1788906711.40 total 50.04 last 68.75) - open/close 3927 (time 1788906721.41 total 60.04 last 68.39) total: 4500 open/close in 68.67 seconds: 65.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1225 (time 1788906744.34 total 10.01 last 122.42) - /unlink 2492 (time 1788906754.35 total 20.02 last 126.58) - /unlink 3896 (time 1788906764.35 total 30.02 last 140.34) total: 4500 /unlink in 34.73 seconds: 129.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 688 (time 1788906783.14 total 10.01 last 68.71) - open/close 1434 (time 1788906793.14 total 20.02 last 74.55) - open/close 2237 (time 1788906803.15 total 30.03 last 80.25) - open/close 3003 (time 1788906813.16 total 40.04 last 76.53) - open/close 3780 (time 1788906823.19 total 50.06 last 77.50) total: 4500 open/close in 59.65 seconds: 75.44 ops/second - /unlink 1539 (time 1788906846.33 total 10.01 last 153.80) - /unlink 3401 (time 1788906856.33 total 20.01 last 186.16) total: 4500 /unlink in 26.09 seconds: 172.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 1035 (time 1788906874.79 total 10.00 last 103.50) - open/close 2092 (time 1788906884.79 total 20.00 last 105.69) - open/close 3259 (time 1788906894.79 total 30.00 last 116.70) - open/close 4470 (time 1788906904.79 total 40.01 last 121.06) total: 4500 open/close in 40.17 seconds: 112.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 2632 (time 1788906916.94 total 10.00 last 263.17) total: 4500 /unlink in 17.84 seconds: 252.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1226 (time 1788906936.91 total 10.00 last 122.54) - open/close 2356 (time 1788906946.92 total 20.01 last 112.93) - open/close 3534 (time 1788906956.92 total 30.02 last 117.72) total: 4500 open/close in 38.64 seconds: 116.45 ops/second - /unlink 1948 (time 1788906978.13 total 10.01 last 194.69) - /unlink 4079 (time 1788906988.13 total 20.01 last 213.10) total: 4500 /unlink in 21.85 seconds: 205.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1512 (time 1788907002.25 total 10.00 last 151.15) - open/close 2805 (time 1788907012.25 total 20.00 last 129.30) - open/close 4164 (time 1788907022.26 total 30.01 last 135.88) total: 4500 open/close in 32.51 seconds: 138.40 ops/second - /unlink 2313 (time 1788907036.99 total 10.00 last 231.25) total: 4500 /unlink in 17.42 seconds: 258.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1279 (time 1788907056.27 total 10.01 last 127.78) - open/close 2415 (time 1788907066.28 total 20.02 last 113.48) - open/close 3619 (time 1788907076.29 total 30.03 last 120.31) total: 4500 open/close in 37.57 seconds: 119.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2372 (time 1788907095.93 total 10.00 last 237.16) total: 4500 /unlink in 18.44 seconds: 244.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1478 (time 1788907116.37 total 10.00 last 147.78) - open/close 2937 (time 1788907126.37 total 20.00 last 145.88) - open/close 4111 (time 1788907136.38 total 30.01 last 117.32) total: 4500 open/close in 33.51 seconds: 134.29 ops/second - /unlink 2124 (time 1788907152.16 total 10.00 last 212.36) - /unlink 4278 (time 1788907162.16 total 20.01 last 215.32) total: 4500 /unlink in 21.34 seconds: 210.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1241 (time 1788907176.06 total 10.01 last 124.03) - open/close 2730 (time 1788907186.06 total 20.01 last 148.87) - open/close 4224 (time 1788907196.06 total 30.01 last 149.40) total: 4500 open/close in 31.94 seconds: 140.88 ops/second - /unlink 2852 (time 1788907209.88 total 10.00 last 285.13) total: 4500 /unlink in 15.87 seconds: 283.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1480 (time 1788907227.88 total 10.00 last 148.00) - open/close 2553 (time 1788907237.88 total 20.01 last 107.24) - open/close 3817 (time 1788907247.89 total 30.01 last 126.35) total: 4500 open/close in 35.84 seconds: 125.56 ops/second - /unlink 2443 (time 1788907265.87 total 10.00 last 244.23) - /unlink 4326 (time 1788907275.87 total 20.01 last 188.22) total: 4500 /unlink in 21.23 seconds: 211.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 455 (time 1788907292.48 total 10.02 last 45.42) - open/close 1018 (time 1788907302.48 total 20.02 last 56.27) - open/close 1452 (time 1788907312.49 total 30.03 last 43.36) - open/close 2032 (time 1788907322.51 total 40.05 last 57.88) - open/close 2584 (time 1788907332.54 total 50.08 last 55.04) - open/close 3223 (time 1788907342.55 total 60.09 last 63.85) - open/close 3755 (time 1788907352.55 total 70.10 last 53.16) - open/close 4413 (time 1788907362.56 total 80.10 last 65.78) total: 4500 open/close in 81.20 seconds: 55.42 ops/second - /unlink 1435 (time 1788907377.83 total 10.00 last 143.49) - /unlink 2752 (time 1788907387.84 total 20.01 last 131.58) - /unlink 4209 (time 1788907397.85 total 30.02 last 145.55) total: 4500 /unlink in 31.98 seconds: 140.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 778 (time 1788907412.67 total 10.01 last 77.74) - open/close 1562 (time 1788907422.67 total 20.01 last 78.39) - open/close 2380 (time 1788907432.67 total 30.01 last 81.78) - open/close 3204 (time 1788907442.68 total 40.02 last 82.36) - open/close 4065 (time 1788907452.68 total 50.02 last 86.09) total: 4500 open/close in 54.89 seconds: 81.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1439 (time 1788907470.48 total 10.01 last 143.78) - /unlink 2781 (time 1788907480.48 total 20.01 last 134.20) - /unlink 4107 (time 1788907490.48 total 30.01 last 132.59) total: 4500 /unlink in 33.16 seconds: 135.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 750 (time 1788907507.74 total 10.02 last 74.84) - open/close 1506 (time 1788907517.74 total 20.02 last 75.58) - open/close 2178 (time 1788907527.75 total 30.03 last 67.15) - open/close 2953 (time 1788907537.75 total 40.04 last 77.47) - open/close 3676 (time 1788907547.76 total 50.04 last 72.26) - open/close 4377 (time 1788907557.77 total 60.06 last 70.00) total: 4500 open/close in 61.56 seconds: 73.10 ops/second - /unlink 1551 (time 1788907572.68 total 10.00 last 155.06) - /unlink 3188 (time 1788907582.68 total 20.01 last 163.66) total: 4500 /unlink in 27.25 seconds: 165.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 904 (time 1788907603.11 total 10.01 last 90.29) - open/close 1929 (time 1788907613.11 total 20.02 last 102.46) - open/close 3149 (time 1788907623.11 total 30.02 last 122.00) - open/close 4342 (time 1788907633.11 total 40.02 last 119.27) total: 4500 open/close in 41.72 seconds: 107.87 ops/second - /unlink 1919 (time 1788907647.11 total 10.00 last 191.87) - /unlink 4391 (time 1788907657.12 total 20.00 last 247.16) total: 4500 /unlink in 20.31 seconds: 221.55 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 94610 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server unloading modules via unload_modules_local on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local oleg322-server: modules unloaded.