== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:55:12 (1787396112) Stopping clients: oleg602-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg602-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg602-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg602-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg602-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg602-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg602-server oleg602-server: oleg602-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg602-server' oleg602-server: oleg602-server.virtnet: executing load_modules_local oleg602-server: Loading modules from /home/green/git/lustre-release/lustre oleg602-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg602-server: detected 4 online CPUs by sysfs oleg602-server: MODOPTS_LIBCFS= oleg602-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg602-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg602-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg602-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg602-server: oleg602-server.virtnet: executing set_default_debug -1 all pdsh@oleg602-client: oleg602-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg602-client.virtnet: -o user_xattr,flock 192.168.206.102@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 793 (time 1787396325.40 total 10.01 last 79.22) - open/close 1527 (time 1787396335.41 total 20.03 last 73.29) - open/close 2203 (time 1787396345.41 total 30.03 last 67.58) - open/close 2852 (time 1787396355.42 total 40.04 last 64.85) - open/close 3569 (time 1787396365.43 total 50.04 last 71.67) - open/close 4253 (time 1787396375.43 total 60.04 last 68.40) total: 4500 open/close in 63.65 seconds: 70.70 ops/second - /unlink 1378 (time 1787396393.37 total 10.00 last 137.80) - /unlink 2688 (time 1787396403.38 total 20.00 last 130.97) - /unlink 4069 (time 1787396413.38 total 30.01 last 138.00) total: 4500 /unlink in 34.12 seconds: 131.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 689 (time 1787396433.31 total 10.01 last 68.84) - open/close 1340 (time 1787396443.32 total 20.02 last 65.06) - open/close 2000 (time 1787396453.32 total 30.02 last 65.98) - open/close 2619 (time 1787396463.32 total 40.02 last 61.89) - open/close 3286 (time 1787396473.37 total 50.06 last 66.40) - open/close 3973 (time 1787396483.37 total 60.06 last 68.70) total: 4500 open/close in 69.71 seconds: 64.55 ops/second - /unlink 1043 (time 1787396508.35 total 10.01 last 104.19) - /unlink 2246 (time 1787396518.35 total 20.01 last 120.27) - /unlink 3656 (time 1787396528.35 total 30.01 last 141.00) total: 4500 /unlink in 36.67 seconds: 122.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 628 (time 1787396550.53 total 10.01 last 62.76) - open/close 1293 (time 1787396560.53 total 20.01 last 66.46) - open/close 1945 (time 1787396570.54 total 30.02 last 65.16) - open/close 2700 (time 1787396580.54 total 40.02 last 75.48) - open/close 3397 (time 1787396590.55 total 50.03 last 69.65) - open/close 4081 (time 1787396600.55 total 60.03 last 68.38) total: 4500 open/close in 66.00 seconds: 68.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1324 (time 1787396620.63 total 10.00 last 132.39) - /unlink 2540 (time 1787396630.64 total 20.01 last 121.49) - /unlink 3709 (time 1787396640.65 total 30.01 last 116.86) total: 4500 /unlink in 36.39 seconds: 123.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 621 (time 1787396662.77 total 10.01 last 62.03) - open/close 1298 (time 1787396672.77 total 20.02 last 67.67) - open/close 1927 (time 1787396682.79 total 30.03 last 62.79) - open/close 2549 (time 1787396692.79 total 40.04 last 62.17) - open/close 3254 (time 1787396702.80 total 50.04 last 70.47) - open/close 3975 (time 1787396712.80 total 60.05 last 72.07) total: 4500 open/close in 67.43 seconds: 66.74 ops/second - /unlink 1344 (time 1787396734.83 total 10.00 last 134.38) - /unlink 2563 (time 1787396744.83 total 20.00 last 121.89) - /unlink 3872 (time 1787396754.84 total 30.00 last 130.88) total: 4500 /unlink in 34.74 seconds: 129.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 656 (time 1787396774.19 total 10.00 last 65.58) - open/close 1322 (time 1787396784.19 total 20.01 last 66.57) - open/close 2060 (time 1787396794.22 total 30.03 last 73.63) - open/close 2796 (time 1787396804.22 total 40.03 last 73.57) - open/close 3520 (time 1787396814.23 total 50.04 last 72.35) - open/close 4279 (time 1787396824.23 total 60.05 last 75.83) total: 4500 open/close in 63.16 seconds: 71.25 ops/second - /unlink 1312 (time 1787396841.80 total 10.00 last 131.19) - /unlink 2614 (time 1787396851.80 total 20.00 last 130.19) - /unlink 3895 (time 1787396861.81 total 30.01 last 128.03) total: 4500 /unlink in 34.74 seconds: 129.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 724 (time 1787396881.17 total 10.01 last 72.34) - open/close 1438 (time 1787396891.17 total 20.01 last 71.39) - open/close 2151 (time 1787396901.17 total 30.01 last 71.26) - open/close 2938 (time 1787396911.18 total 40.02 last 78.67) - open/close 3631 (time 1787396921.19 total 50.03 last 69.18) - open/close 4412 (time 1787396931.20 total 60.04 last 78.07) total: 4500 open/close in 61.21 seconds: 73.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1298 (time 1787396946.97 total 10.01 last 129.62) - /unlink 2610 (time 1787396956.98 total 20.02 last 131.14) - /unlink 3936 (time 1787396966.98 total 30.02 last 132.54) total: 4500 /unlink in 34.20 seconds: 131.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 670 (time 1787396986.26 total 10.01 last 66.95) - open/close 1415 (time 1787396996.27 total 20.02 last 74.40) - open/close 2160 (time 1787397006.27 total 30.02 last 74.49) - open/close 2918 (time 1787397016.29 total 40.04 last 75.69) - open/close 3610 (time 1787397026.30 total 50.05 last 69.14) - open/close 4353 (time 1787397036.30 total 60.05 last 74.29) total: 4500 open/close in 62.01 seconds: 72.56 ops/second - /unlink 1145 (time 1787397053.86 total 10.00 last 114.48) - /unlink 2507 (time 1787397063.87 total 20.01 last 136.14) - /unlink 3856 (time 1787397073.87 total 30.01 last 134.88) total: 4500 /unlink in 36.32 seconds: 123.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 758 (time 1787397095.94 total 10.02 last 75.68) - open/close 1461 (time 1787397105.95 total 20.02 last 70.25) - open/close 2143 (time 1787397115.95 total 30.02 last 68.19) - open/close 2887 (time 1787397125.96 total 40.03 last 74.35) - open/close 3585 (time 1787397135.98 total 50.05 last 69.68) - open/close 4197 (time 1787397145.98 total 60.05 last 61.19) total: 4500 open/close in 64.43 seconds: 69.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1111 (time 1787397165.23 total 10.01 last 110.98) - /unlink 2311 (time 1787397175.24 total 20.01 last 119.95) - /unlink 3565 (time 1787397185.24 total 30.02 last 125.36) total: 4500 /unlink in 37.60 seconds: 119.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 643 (time 1787397208.39 total 10.01 last 64.23) - open/close 1280 (time 1787397218.40 total 20.02 last 63.63) - open/close 2022 (time 1787397228.42 total 30.05 last 74.03) - open/close 2768 (time 1787397238.43 total 40.05 last 74.57) - open/close 3395 (time 1787397248.43 total 50.05 last 62.69) - open/close 4007 (time 1787397258.44 total 60.06 last 61.12) total: 4500 open/close in 67.60 seconds: 66.57 ops/second - /unlink 1295 (time 1787397280.41 total 10.00 last 129.47) - /unlink 2472 (time 1787397290.41 total 20.01 last 117.64) - /unlink 3829 (time 1787397300.42 total 30.02 last 135.57) total: 4500 /unlink in 34.46 seconds: 130.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 736 (time 1787397319.64 total 10.00 last 73.57) - open/close 1471 (time 1787397329.65 total 20.01 last 73.47) - open/close 2193 (time 1787397339.65 total 30.01 last 72.20) - open/close 2924 (time 1787397349.65 total 40.01 last 73.07) - open/close 3666 (time 1787397359.66 total 50.01 last 74.18) - open/close 4422 (time 1787397369.66 total 60.02 last 75.58) total: 4500 open/close in 61.14 seconds: 73.60 ops/second - /unlink 1159 (time 1787397384.99 total 10.01 last 115.75) - /unlink 2455 (time 1787397394.99 total 20.01 last 129.59) - /unlink 3739 (time 1787397404.99 total 30.02 last 128.36) total: 4500 /unlink in 35.89 seconds: 125.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 732 (time 1787397426.19 total 10.00 last 73.18) - open/close 1488 (time 1787397436.21 total 20.02 last 75.49) - open/close 2172 (time 1787397446.21 total 30.02 last 68.36) - open/close 2975 (time 1787397456.22 total 40.03 last 80.24) - open/close 3620 (time 1787397466.23 total 50.04 last 64.46) - open/close 4314 (time 1787397476.23 total 60.04 last 69.39) total: 4500 open/close in 62.58 seconds: 71.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1367 (time 1787397493.76 total 10.00 last 136.65) - /unlink 2643 (time 1787397503.77 total 20.00 last 127.59) - /unlink 4004 (time 1787397513.77 total 30.01 last 136.06) total: 4500 /unlink in 34.02 seconds: 132.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 691 (time 1787397532.81 total 10.00 last 69.08) - open/close 1377 (time 1787397542.83 total 20.02 last 68.51) - open/close 2109 (time 1787397552.83 total 30.02 last 73.17) - open/close 2834 (time 1787397562.84 total 40.03 last 72.42) - open/close 3531 (time 1787397572.86 total 50.05 last 69.60) - open/close 4280 (time 1787397582.86 total 60.05 last 74.87) total: 4500 open/close in 63.68 seconds: 70.66 ops/second - /unlink 1331 (time 1787397601.74 total 10.00 last 133.07) - /unlink 2635 (time 1787397611.75 total 20.01 last 130.36) - /unlink 4045 (time 1787397621.76 total 30.02 last 140.85) total: 4500 /unlink in 33.63 seconds: 133.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 759 (time 1787397639.67 total 10.00 last 75.87) - open/close 1481 (time 1787397649.70 total 20.03 last 72.01) - open/close 2250 (time 1787397659.71 total 30.04 last 76.81) - open/close 3024 (time 1787397669.72 total 40.05 last 77.33) - open/close 3760 (time 1787397679.72 total 50.05 last 73.60) total: 4500 open/close in 59.82 seconds: 75.22 ops/second - /unlink 1417 (time 1787397703.94 total 10.00 last 141.68) - /unlink 2766 (time 1787397713.94 total 20.00 last 134.90) - /unlink 4164 (time 1787397723.94 total 30.01 last 139.76) total: 4500 /unlink in 32.44 seconds: 138.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 794 (time 1787397739.97 total 10.01 last 79.33) - open/close 1433 (time 1787397749.98 total 20.02 last 63.80) - open/close 2074 (time 1787397760.00 total 30.04 last 63.99) - open/close 2737 (time 1787397770.00 total 40.04 last 66.29) - open/close 3411 (time 1787397780.01 total 50.05 last 67.33) - open/close 4137 (time 1787397790.01 total 60.05 last 72.60) total: 4500 open/close in 64.69 seconds: 69.56 ops/second - /unlink 1311 (time 1787397809.02 total 10.00 last 131.03) - /unlink 2714 (time 1787397819.04 total 20.02 last 140.04) - /unlink 4122 (time 1787397829.05 total 30.04 last 140.64) total: 4500 /unlink in 32.99 seconds: 136.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 703 (time 1787397847.09 total 10.01 last 70.26) - open/close 1482 (time 1787397857.10 total 20.01 last 77.88) - open/close 2128 (time 1787397867.11 total 30.02 last 64.53) - open/close 2863 (time 1787397877.12 total 40.03 last 73.43) - open/close 3554 (time 1787397887.12 total 50.03 last 69.07) - open/close 4219 (time 1787397897.12 total 60.03 last 66.50) total: 4500 open/close in 64.71 seconds: 69.54 ops/second - /unlink 1357 (time 1787397916.80 total 10.00 last 135.65) - /unlink 2567 (time 1787397926.80 total 20.01 last 120.97) - /unlink 3941 (time 1787397936.80 total 30.01 last 137.32) total: 4500 /unlink in 34.57 seconds: 130.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 645 (time 1787397956.29 total 10.01 last 64.46) - open/close 1378 (time 1787397966.30 total 20.01 last 73.24) - open/close 2165 (time 1787397976.30 total 30.01 last 78.70) - open/close 2906 (time 1787397986.30 total 40.02 last 74.09) - open/close 3622 (time 1787397996.30 total 50.02 last 71.56) - open/close 4322 (time 1787398006.30 total 60.02 last 70.00) total: 4500 open/close in 62.48 seconds: 72.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1438 (time 1787398023.26 total 10.00 last 143.78) - /unlink 2667 (time 1787398033.27 total 20.00 last 122.86) - /unlink 3919 (time 1787398043.27 total 30.01 last 125.19) total: 4500 /unlink in 34.28 seconds: 131.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 773 (time 1787398062.21 total 10.00 last 77.27) - open/close 1514 (time 1787398072.22 total 20.01 last 74.04) - open/close 2271 (time 1787398082.23 total 30.02 last 75.66) - open/close 2983 (time 1787398092.24 total 40.03 last 71.11) - open/close 3629 (time 1787398102.25 total 50.04 last 64.53) - open/close 4250 (time 1787398112.26 total 60.05 last 62.02) total: 4500 open/close in 63.15 seconds: 71.26 ops/second - /unlink 1205 (time 1787398129.79 total 10.00 last 120.49) - /unlink 2535 (time 1787398139.80 total 20.00 last 132.95) - /unlink 3951 (time 1787398149.80 total 30.01 last 141.56) total: 4500 /unlink in 33.97 seconds: 132.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 718 (time 1787398169.18 total 10.00 last 71.77) - open/close 1458 (time 1787398179.19 total 20.01 last 73.93) - open/close 2166 (time 1787398189.19 total 30.02 last 70.79) - open/close 2905 (time 1787398199.20 total 40.02 last 73.85) - open/close 3638 (time 1787398209.20 total 50.03 last 73.29) - open/close 4375 (time 1787398219.21 total 60.03 last 73.68) total: 4500 open/close in 61.56 seconds: 73.10 ops/second - /unlink 1293 (time 1787398235.44 total 10.00 last 129.26) - /unlink 2640 (time 1787398245.44 total 20.00 last 134.68) - /unlink 3937 (time 1787398255.44 total 30.01 last 129.61) total: 4500 /unlink in 34.32 seconds: 131.12 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95020 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg602-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg602-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg602-server stop mds service on oleg602-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg602-server stop mds service on oleg602-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg602-server unloading modules via unload_modules_local on: 'oleg602-server' oleg602-server: oleg602-server.virtnet: executing unload_modules_local oleg602-server: modules unloaded.