== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:07:54 (1782760074) Stopping clients: oleg220-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg220-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg220-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg220-server oleg220-server: oleg220-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg220-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg220-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg220-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 730 (time 1782760295.57 total 10.00 last 72.98) - open/close 1393 (time 1782760305.58 total 20.01 last 66.24) - open/close 2082 (time 1782760315.58 total 30.01 last 68.90) - open/close 2764 (time 1782760325.59 total 40.02 last 68.12) - open/close 3438 (time 1782760335.60 total 50.03 last 67.35) - open/close 4190 (time 1782760345.60 total 60.04 last 75.17) total: 4500 open/close in 64.08 seconds: 70.22 ops/second - /unlink 1341 (time 1782760364.80 total 10.00 last 134.06) - /unlink 2762 (time 1782760374.80 total 20.01 last 142.05) - /unlink 3957 (time 1782760384.80 total 30.01 last 119.49) total: 4500 /unlink in 35.19 seconds: 127.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 730 (time 1782760404.92 total 10.00 last 72.99) - open/close 1444 (time 1782760414.93 total 20.02 last 71.29) - open/close 2006 (time 1782760424.94 total 30.02 last 56.16) - open/close 2580 (time 1782760434.96 total 40.04 last 57.30) - open/close 3299 (time 1782760444.96 total 50.04 last 71.88) - open/close 4010 (time 1782760454.96 total 60.05 last 71.08) total: 4500 open/close in 66.63 seconds: 67.54 ops/second - /unlink 1417 (time 1782760476.70 total 10.00 last 141.66) - /unlink 2586 (time 1782760486.71 total 20.01 last 116.86) - /unlink 3934 (time 1782760496.71 total 30.01 last 134.72) total: 4500 /unlink in 34.83 seconds: 129.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 702 (time 1782760516.94 total 10.03 last 69.98) - open/close 1276 (time 1782760526.96 total 20.05 last 57.29) - open/close 1905 (time 1782760536.96 total 30.05 last 62.88) - open/close 2538 (time 1782760546.96 total 40.06 last 63.29) - open/close 3176 (time 1782760556.97 total 50.06 last 63.78) - open/close 3809 (time 1782760566.98 total 60.07 last 63.20) - open/close 4457 (time 1782760576.98 total 70.07 last 64.80) total: 4500 open/close in 70.95 seconds: 63.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 974 (time 1782760592.67 total 10.01 last 97.34) - /unlink 2110 (time 1782760602.67 total 20.01 last 113.55) - /unlink 3180 (time 1782760612.68 total 30.02 last 106.94) - /unlink 4190 (time 1782760622.68 total 40.02 last 100.97) total: 4500 /unlink in 42.59 seconds: 105.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 712 (time 1782760641.41 total 10.02 last 71.06) - open/close 1467 (time 1782760651.42 total 20.03 last 75.40) - open/close 2210 (time 1782760661.44 total 30.05 last 74.15) - open/close 2979 (time 1782760671.45 total 40.06 last 76.83) - open/close 3632 (time 1782760681.45 total 50.07 last 65.28) - open/close 4235 (time 1782760691.48 total 60.09 last 60.16) total: 4500 open/close in 64.44 seconds: 69.83 ops/second - /unlink 1479 (time 1782760710.99 total 10.01 last 147.80) - /unlink 2939 (time 1782760720.99 total 20.01 last 145.95) - /unlink 4142 (time 1782760731.00 total 30.02 last 120.19) total: 4500 /unlink in 33.38 seconds: 134.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 720 (time 1782760749.82 total 10.00 last 71.98) - open/close 1441 (time 1782760759.83 total 20.01 last 72.02) - open/close 1939 (time 1782760769.86 total 30.04 last 49.67) - open/close 2505 (time 1782760779.86 total 40.04 last 56.58) - open/close 3197 (time 1782760789.88 total 50.06 last 69.08) - open/close 3904 (time 1782760799.88 total 60.06 last 70.67) total: 4500 open/close in 68.91 seconds: 65.30 ops/second - /unlink 1012 (time 1782760823.92 total 10.01 last 101.15) - /unlink 2345 (time 1782760833.92 total 20.01 last 133.30) - /unlink 3267 (time 1782760843.92 total 30.01 last 92.17) - /unlink 4101 (time 1782760853.93 total 40.02 last 83.32) total: 4500 /unlink in 44.96 seconds: 100.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 713 (time 1782760874.28 total 10.00 last 71.30) - open/close 1371 (time 1782760884.29 total 20.01 last 65.76) - open/close 2126 (time 1782760894.30 total 30.02 last 75.41) - open/close 2840 (time 1782760904.30 total 40.02 last 71.37) - open/close 3585 (time 1782760914.31 total 50.02 last 74.49) - open/close 4313 (time 1782760924.31 total 60.03 last 72.78) total: 4500 open/close in 62.47 seconds: 72.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1473 (time 1782760941.12 total 10.01 last 147.14) - /unlink 3075 (time 1782760951.12 total 20.01 last 160.16) total: 4500 /unlink in 29.85 seconds: 150.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 766 (time 1782760975.57 total 10.01 last 76.50) - open/close 1507 (time 1782760985.57 total 20.02 last 74.08) - open/close 2264 (time 1782760995.59 total 30.04 last 75.52) - open/close 2949 (time 1782761005.60 total 40.04 last 68.49) - open/close 3684 (time 1782761015.60 total 50.05 last 73.46) - open/close 4274 (time 1782761025.60 total 60.05 last 58.98) total: 4500 open/close in 64.79 seconds: 69.45 ops/second - /unlink 1516 (time 1782761045.33 total 10.00 last 151.52) - /unlink 2848 (time 1782761055.33 total 20.01 last 133.17) - /unlink 4170 (time 1782761065.35 total 30.02 last 131.97) total: 4500 /unlink in 32.43 seconds: 138.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 711 (time 1782761081.93 total 10.00 last 71.08) - open/close 1546 (time 1782761091.95 total 20.02 last 83.32) - open/close 2285 (time 1782761101.96 total 30.03 last 73.83) - open/close 3023 (time 1782761111.96 total 40.03 last 73.79) - open/close 3845 (time 1782761121.97 total 50.04 last 82.15) total: 4500 open/close in 57.82 seconds: 77.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1252 (time 1782761144.09 total 10.00 last 125.17) - /unlink 2257 (time 1782761154.10 total 20.01 last 100.45) - /unlink 3326 (time 1782761164.10 total 30.01 last 106.86) - /unlink 4437 (time 1782761174.11 total 40.02 last 110.98) total: 4500 /unlink in 40.75 seconds: 110.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 763 (time 1782761189.71 total 10.01 last 76.24) - open/close 1498 (time 1782761199.72 total 20.01 last 73.46) - open/close 2264 (time 1782761209.73 total 30.02 last 76.52) - open/close 3058 (time 1782761219.74 total 40.03 last 79.32) - open/close 3858 (time 1782761229.75 total 50.04 last 79.95) total: 4500 open/close in 58.28 seconds: 77.21 ops/second - /unlink 1491 (time 1782761252.54 total 10.00 last 149.09) - /unlink 2744 (time 1782761262.54 total 20.00 last 125.27) - /unlink 4023 (time 1782761272.55 total 30.01 last 127.79) total: 4500 /unlink in 34.03 seconds: 132.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 635 (time 1782761292.21 total 10.02 last 63.35) - open/close 1401 (time 1782761302.22 total 20.03 last 76.54) - open/close 2258 (time 1782761312.22 total 30.03 last 85.67) - open/close 2980 (time 1782761322.23 total 40.04 last 72.18) - open/close 3717 (time 1782761332.23 total 50.05 last 73.64) - open/close 4460 (time 1782761342.24 total 60.05 last 74.28) total: 4500 open/close in 60.52 seconds: 74.36 ops/second - /unlink 1444 (time 1782761357.29 total 10.00 last 144.34) - /unlink 2756 (time 1782761367.29 total 20.01 last 131.15) - /unlink 3958 (time 1782761377.30 total 30.02 last 120.07) total: 4500 /unlink in 35.79 seconds: 125.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 757 (time 1782761397.85 total 10.00 last 75.70) - open/close 1430 (time 1782761407.85 total 20.01 last 67.26) - open/close 2161 (time 1782761417.86 total 30.01 last 73.09) - open/close 3015 (time 1782761427.87 total 40.02 last 85.28) - open/close 3784 (time 1782761437.89 total 50.04 last 76.73) total: 4500 open/close in 58.83 seconds: 76.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1307 (time 1782761461.60 total 10.00 last 130.64) - /unlink 2676 (time 1782761471.61 total 20.01 last 136.77) - /unlink 4054 (time 1782761481.61 total 30.02 last 137.78) total: 4500 /unlink in 32.73 seconds: 137.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 749 (time 1782761498.93 total 10.00 last 74.87) - open/close 1444 (time 1782761508.93 total 20.01 last 69.49) - open/close 2143 (time 1782761518.94 total 30.02 last 69.82) - open/close 2898 (time 1782761528.95 total 40.03 last 75.41) - open/close 3557 (time 1782761538.95 total 50.03 last 65.90) - open/close 4317 (time 1782761548.97 total 60.05 last 75.84) total: 4500 open/close in 63.51 seconds: 70.85 ops/second - /unlink 1149 (time 1782761568.34 total 10.01 last 114.82) - /unlink 2189 (time 1782761578.35 total 20.01 last 103.93) - /unlink 3440 (time 1782761588.35 total 30.02 last 125.05) total: 4500 /unlink in 38.78 seconds: 116.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 629 (time 1782761612.65 total 10.00 last 62.88) - open/close 1298 (time 1782761622.65 total 20.00 last 66.90) - open/close 1954 (time 1782761632.65 total 30.00 last 65.60) - open/close 2793 (time 1782761642.65 total 40.01 last 83.89) - open/close 3549 (time 1782761652.65 total 50.01 last 75.57) - open/close 4345 (time 1782761662.66 total 60.01 last 79.58) total: 4500 open/close in 61.62 seconds: 73.03 ops/second - /unlink 1269 (time 1782761678.56 total 10.00 last 126.86) - /unlink 2643 (time 1782761688.56 total 20.01 last 137.37) - /unlink 4080 (time 1782761698.57 total 30.01 last 143.60) total: 4500 /unlink in 33.09 seconds: 136.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 669 (time 1782761717.44 total 10.00 last 66.88) - open/close 1371 (time 1782761727.45 total 20.01 last 70.16) - open/close 2092 (time 1782761737.45 total 30.01 last 72.05) - open/close 2833 (time 1782761747.46 total 40.02 last 74.05) - open/close 3531 (time 1782761757.47 total 50.03 last 69.75) - open/close 4212 (time 1782761767.48 total 60.04 last 68.04) total: 4500 open/close in 64.33 seconds: 69.95 ops/second - /unlink 1443 (time 1782761787.61 total 10.01 last 144.15) - /unlink 2785 (time 1782761797.63 total 20.02 last 134.01) - /unlink 4024 (time 1782761807.64 total 30.03 last 123.80) total: 4500 /unlink in 33.57 seconds: 134.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 683 (time 1782761827.00 total 10.01 last 68.25) - open/close 1468 (time 1782761837.04 total 20.04 last 78.24) - open/close 2213 (time 1782761847.06 total 30.07 last 74.31) - open/close 2874 (time 1782761857.06 total 40.07 last 66.08) - open/close 3490 (time 1782761867.10 total 50.10 last 61.41) - open/close 4186 (time 1782761877.11 total 60.12 last 69.50) total: 4500 open/close in 64.63 seconds: 69.63 ops/second - /unlink 1276 (time 1782761896.67 total 10.00 last 127.58) - /unlink 2764 (time 1782761906.67 total 20.01 last 148.71) - /unlink 4083 (time 1782761916.68 total 30.01 last 131.86) total: 4500 /unlink in 33.55 seconds: 134.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 627 (time 1782761936.32 total 10.02 last 62.60) - open/close 1212 (time 1782761946.32 total 20.02 last 58.48) - open/close 1932 (time 1782761956.33 total 30.03 last 71.91) - open/close 2616 (time 1782761966.34 total 40.04 last 68.37) - open/close 3242 (time 1782761976.34 total 50.04 last 62.57) - open/close 3816 (time 1782761986.36 total 60.06 last 57.27) - open/close 4360 (time 1782761996.38 total 70.08 last 54.29) total: 4500 open/close in 72.77 seconds: 61.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1289 (time 1782762014.53 total 10.01 last 128.81) - /unlink 2830 (time 1782762024.54 total 20.01 last 154.00) - /unlink 4194 (time 1782762034.54 total 30.02 last 136.30) total: 4500 /unlink in 32.33 seconds: 139.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 778 (time 1782762051.91 total 10.02 last 77.61) - open/close 1524 (time 1782762061.92 total 20.03 last 74.58) - open/close 2148 (time 1782762071.93 total 30.04 last 62.33) - open/close 2862 (time 1782762081.93 total 40.04 last 71.36) - open/close 3624 (time 1782762091.95 total 50.06 last 76.11) - open/close 4317 (time 1782762101.99 total 60.10 last 69.01) total: 4500 open/close in 62.48 seconds: 72.02 ops/second - /unlink 1369 (time 1782762118.92 total 10.00 last 136.88) - /unlink 2648 (time 1782762128.92 total 20.00 last 127.87) - /unlink 3971 (time 1782762138.93 total 30.01 last 132.27) total: 4500 /unlink in 33.63 seconds: 133.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 699 (time 1782762157.70 total 10.01 last 69.85) - open/close 1284 (time 1782762167.70 total 20.01 last 58.47) - open/close 2017 (time 1782762177.70 total 30.01 last 73.29) - open/close 2722 (time 1782762187.71 total 40.02 last 70.47) - open/close 3439 (time 1782762197.71 total 50.02 last 71.67) - open/close 4128 (time 1782762207.71 total 60.02 last 68.89) total: 4500 open/close in 65.39 seconds: 68.82 ops/second - /unlink 1148 (time 1782762228.17 total 10.00 last 114.79) - /unlink 2397 (time 1782762238.17 total 20.00 last 124.89) - /unlink 3875 (time 1782762248.18 total 30.01 last 147.73) total: 4500 /unlink in 35.23 seconds: 127.72 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94834 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg220-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server unloading modules via unload_modules_local on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing unload_modules_local oleg220-server: modules unloaded.