== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:36:47 (1783949807) Stopping clients: oleg438-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg438-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg438-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg438-server oleg438-server: oleg438-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg438-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg438-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 566 (time 1783949972.98 total 10.01 last 56.56) - open/close 1150 (time 1783949982.99 total 20.02 last 58.34) - open/close 1694 (time 1783949992.99 total 30.02 last 54.40) - open/close 2230 (time 1783950003.00 total 40.03 last 53.54) - open/close 2867 (time 1783950013.01 total 50.04 last 63.66) - open/close 3450 (time 1783950023.01 total 60.04 last 58.26) - open/close 3989 (time 1783950033.01 total 70.04 last 53.90) - open/close 4491 (time 1783950043.03 total 80.06 last 50.14) total: 4500 open/close in 80.26 seconds: 56.07 ops/second - /unlink 1162 (time 1783950058.03 total 10.01 last 116.12) - /unlink 2221 (time 1783950068.04 total 20.01 last 105.83) - /unlink 3208 (time 1783950078.05 total 30.02 last 98.59) - /unlink 4306 (time 1783950088.06 total 40.03 last 109.75) total: 4500 /unlink in 41.55 seconds: 108.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 545 (time 1783950104.57 total 10.02 last 54.41) - open/close 1131 (time 1783950114.59 total 20.04 last 58.46) - open/close 1828 (time 1783950124.60 total 30.05 last 69.66) - open/close 2418 (time 1783950134.61 total 40.06 last 58.93) - open/close 2999 (time 1783950144.61 total 50.06 last 58.10) - open/close 3534 (time 1783950154.63 total 60.08 last 53.42) - open/close 4152 (time 1783950164.63 total 70.08 last 61.79) total: 4500 open/close in 76.09 seconds: 59.14 ops/second - /unlink 1157 (time 1783950185.29 total 10.00 last 115.67) - /unlink 2228 (time 1783950195.30 total 20.01 last 107.07) - /unlink 3318 (time 1783950205.30 total 30.01 last 108.95) - /unlink 4322 (time 1783950215.31 total 40.02 last 100.31) total: 4500 /unlink in 41.73 seconds: 107.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 584 (time 1783950231.47 total 10.03 last 58.25) - open/close 1156 (time 1783950241.48 total 20.03 last 57.15) - open/close 1696 (time 1783950251.48 total 30.04 last 53.98) - open/close 2295 (time 1783950261.50 total 40.05 last 59.83) - open/close 2893 (time 1783950271.51 total 50.06 last 59.75) - open/close 3528 (time 1783950281.54 total 60.09 last 63.30) - open/close 4133 (time 1783950291.55 total 70.10 last 60.43) total: 4500 open/close in 76.74 seconds: 58.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 931 (time 1783950312.84 total 10.00 last 93.07) - /unlink 1840 (time 1783950322.84 total 20.00 last 90.90) - /unlink 2897 (time 1783950332.85 total 30.01 last 105.68) - /unlink 3960 (time 1783950342.85 total 40.01 last 106.29) total: 4500 /unlink in 45.46 seconds: 99.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 508 (time 1783950364.29 total 10.01 last 50.77) - open/close 1013 (time 1783950374.30 total 20.01 last 50.45) - open/close 1545 (time 1783950384.31 total 30.02 last 53.16) - open/close 2067 (time 1783950394.32 total 40.03 last 52.16) - open/close 2677 (time 1783950404.32 total 50.03 last 60.98) - open/close 3247 (time 1783950414.32 total 60.03 last 57.00) - open/close 3845 (time 1783950424.33 total 70.04 last 59.75) - open/close 4469 (time 1783950434.34 total 80.05 last 62.35) total: 4500 open/close in 80.57 seconds: 55.85 ops/second - /unlink 1026 (time 1783950449.61 total 10.00 last 102.59) - /unlink 1846 (time 1783950459.62 total 20.01 last 81.89) - /unlink 2897 (time 1783950469.63 total 30.02 last 105.07) - /unlink 3836 (time 1783950479.63 total 40.02 last 93.87) total: 4500 /unlink in 47.52 seconds: 94.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 565 (time 1783950502.89 total 10.02 last 56.41) - open/close 1110 (time 1783950512.90 total 20.03 last 54.44) - open/close 1625 (time 1783950522.90 total 30.03 last 51.47) - open/close 2168 (time 1783950532.91 total 40.04 last 54.28) - open/close 2760 (time 1783950542.91 total 50.04 last 59.16) - open/close 3353 (time 1783950552.93 total 60.06 last 59.18) - open/close 3951 (time 1783950562.94 total 70.07 last 59.77) - open/close 4431 (time 1783950572.96 total 80.09 last 47.89) total: 4500 open/close in 81.24 seconds: 55.39 ops/second - /unlink 1036 (time 1783950589.28 total 10.00 last 103.58) - /unlink 2094 (time 1783950599.29 total 20.01 last 105.76) - /unlink 3049 (time 1783950609.29 total 30.01 last 95.44) - /unlink 4148 (time 1783950619.30 total 40.02 last 109.84) total: 4500 /unlink in 43.75 seconds: 102.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 537 (time 1783950638.19 total 10.00 last 53.68) - open/close 1098 (time 1783950648.19 total 20.01 last 56.07) - open/close 1659 (time 1783950658.20 total 30.02 last 56.05) - open/close 2156 (time 1783950668.21 total 40.02 last 49.67) - open/close 2782 (time 1783950678.22 total 50.03 last 62.54) - open/close 3315 (time 1783950688.24 total 60.05 last 53.20) - open/close 3858 (time 1783950698.24 total 70.05 last 54.30) - open/close 4395 (time 1783950708.24 total 80.06 last 53.67) total: 4500 open/close in 81.76 seconds: 55.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 999 (time 1783950724.74 total 10.00 last 99.89) - /unlink 2058 (time 1783950734.74 total 20.00 last 105.90) - /unlink 3079 (time 1783950744.74 total 30.00 last 102.07) - /unlink 4139 (time 1783950754.75 total 40.01 last 105.94) total: 4500 /unlink in 43.41 seconds: 103.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 524 (time 1783950773.25 total 10.04 last 52.19) - open/close 1062 (time 1783950783.28 total 20.07 last 53.67) - open/close 1613 (time 1783950793.29 total 30.08 last 55.03) - open/close 2185 (time 1783950803.30 total 40.09 last 57.12) - open/close 2746 (time 1783950813.31 total 50.10 last 56.05) - open/close 3295 (time 1783950823.33 total 60.11 last 54.84) - open/close 3929 (time 1783950833.33 total 70.11 last 63.40) - open/close 4484 (time 1783950843.33 total 80.12 last 55.46) total: 4500 open/close in 80.45 seconds: 55.94 ops/second - /unlink 1047 (time 1783950859.24 total 10.00 last 104.65) - /unlink 2057 (time 1783950869.25 total 20.01 last 100.90) - /unlink 2928 (time 1783950879.26 total 30.02 last 87.06) - /unlink 3861 (time 1783950889.26 total 40.03 last 93.22) total: 4500 /unlink in 47.37 seconds: 95.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 493 (time 1783950912.13 total 10.01 last 49.23) - open/close 1068 (time 1783950922.14 total 20.03 last 57.44) - open/close 1645 (time 1783950932.15 total 30.04 last 57.63) - open/close 2205 (time 1783950942.16 total 40.05 last 55.95) - open/close 2706 (time 1783950952.18 total 50.06 last 50.03) - open/close 3300 (time 1783950962.18 total 60.06 last 59.40) - open/close 3896 (time 1783950972.18 total 70.07 last 59.58) - open/close 4468 (time 1783950982.20 total 80.09 last 57.08) total: 4500 open/close in 80.53 seconds: 55.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1016 (time 1783950997.28 total 10.01 last 101.46) - /unlink 1955 (time 1783951007.28 total 20.02 last 93.85) - /unlink 2967 (time 1783951017.29 total 30.03 last 101.10) - /unlink 3880 (time 1783951027.30 total 40.04 last 91.21) total: 4500 /unlink in 46.08 seconds: 97.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 558 (time 1783951048.50 total 10.01 last 55.76) - open/close 1061 (time 1783951058.51 total 20.02 last 50.25) - open/close 1616 (time 1783951068.53 total 30.03 last 55.42) - open/close 2182 (time 1783951078.53 total 40.03 last 56.58) - open/close 2707 (time 1783951088.54 total 50.05 last 52.44) - open/close 3158 (time 1783951098.55 total 60.05 last 45.08) - open/close 3752 (time 1783951108.56 total 70.06 last 59.36) - open/close 4317 (time 1783951118.56 total 80.06 last 56.48) total: 4500 open/close in 82.82 seconds: 54.33 ops/second - /unlink 923 (time 1783951136.71 total 10.01 last 92.21) - /unlink 1822 (time 1783951146.72 total 20.02 last 89.81) - /unlink 2659 (time 1783951156.72 total 30.03 last 83.65) - /unlink 3380 (time 1783951166.73 total 40.04 last 72.01) - /unlink 4142 (time 1783951176.74 total 50.04 last 76.17) total: 4500 /unlink in 52.97 seconds: 84.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 499 (time 1783951196.00 total 10.00 last 49.88) - open/close 1015 (time 1783951206.02 total 20.02 last 51.50) - open/close 1529 (time 1783951216.03 total 30.03 last 51.36) - open/close 2065 (time 1783951226.05 total 40.05 last 53.52) - open/close 2585 (time 1783951236.06 total 50.06 last 51.91) - open/close 3085 (time 1783951246.07 total 60.07 last 49.96) - open/close 3590 (time 1783951256.07 total 70.07 last 50.49) - open/close 4093 (time 1783951266.10 total 80.10 last 50.17) total: 4500 open/close in 87.13 seconds: 51.65 ops/second - /unlink 1016 (time 1783951288.07 total 10.01 last 101.52) - /unlink 1993 (time 1783951298.07 total 20.01 last 97.66) - /unlink 2902 (time 1783951308.07 total 30.02 last 90.86) - /unlink 3856 (time 1783951318.08 total 40.02 last 95.38) total: 4500 /unlink in 46.37 seconds: 97.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 532 (time 1783951339.86 total 10.01 last 53.14) - open/close 1039 (time 1783951349.87 total 20.03 last 50.63) - open/close 1553 (time 1783951359.87 total 30.03 last 51.40) - open/close 2142 (time 1783951369.87 total 40.03 last 58.87) - open/close 2709 (time 1783951379.88 total 50.04 last 56.66) - open/close 3231 (time 1783951389.91 total 60.06 last 52.07) - open/close 3746 (time 1783951399.91 total 70.06 last 51.49) - open/close 4166 (time 1783951409.92 total 80.08 last 41.95) total: 4500 open/close in 87.46 seconds: 51.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 823 (time 1783951433.91 total 10.01 last 82.25) - /unlink 1741 (time 1783951443.91 total 20.01 last 91.78) - /unlink 2753 (time 1783951453.91 total 30.01 last 101.15) - /unlink 3873 (time 1783951463.92 total 40.01 last 111.98) total: 4500 /unlink in 45.78 seconds: 98.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 546 (time 1783951484.05 total 10.00 last 54.57) - open/close 1139 (time 1783951494.08 total 20.03 last 59.17) - open/close 1737 (time 1783951504.08 total 30.03 last 59.78) - open/close 2292 (time 1783951514.08 total 40.03 last 55.50) - open/close 2897 (time 1783951524.09 total 50.04 last 60.45) - open/close 3491 (time 1783951534.10 total 60.05 last 59.36) - open/close 4071 (time 1783951544.10 total 70.05 last 57.97) total: 4500 open/close in 77.57 seconds: 58.01 ops/second - /unlink 1015 (time 1783951566.06 total 10.00 last 101.47) - /unlink 2056 (time 1783951576.06 total 20.00 last 104.09) - /unlink 3064 (time 1783951586.06 total 30.01 last 100.75) - /unlink 4061 (time 1783951596.07 total 40.01 last 99.65) total: 4500 /unlink in 43.89 seconds: 102.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 527 (time 1783951615.27 total 10.05 last 52.42) - open/close 1098 (time 1783951625.27 total 20.06 last 57.06) - open/close 1691 (time 1783951635.30 total 30.09 last 59.14) - open/close 2239 (time 1783951645.32 total 40.10 last 54.71) - open/close 2750 (time 1783951655.32 total 50.10 last 51.09) - open/close 3381 (time 1783951665.32 total 60.11 last 63.08) - open/close 3972 (time 1783951675.35 total 70.13 last 58.96) total: 4500 open/close in 78.97 seconds: 56.98 ops/second - /unlink 981 (time 1783951698.94 total 10.00 last 98.10) - /unlink 1923 (time 1783951708.95 total 20.01 last 94.10) - /unlink 2812 (time 1783951718.96 total 30.02 last 88.84) - /unlink 3739 (time 1783951728.96 total 40.02 last 92.65) total: 4500 /unlink in 47.96 seconds: 93.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 528 (time 1783951752.60 total 10.01 last 52.76) - open/close 1097 (time 1783951762.62 total 20.02 last 56.81) - open/close 1741 (time 1783951772.63 total 30.03 last 64.36) - open/close 2368 (time 1783951782.63 total 40.04 last 62.65) - open/close 2934 (time 1783951792.64 total 50.04 last 56.59) - open/close 3515 (time 1783951802.64 total 60.04 last 58.08) - open/close 4057 (time 1783951812.64 total 70.04 last 54.19) total: 4500 open/close in 78.92 seconds: 57.02 ops/second - /unlink 888 (time 1783951837.83 total 10.00 last 88.79) - /unlink 1830 (time 1783951847.84 total 20.01 last 94.11) - /unlink 2854 (time 1783951857.85 total 30.02 last 102.30) - /unlink 3909 (time 1783951867.86 total 40.03 last 105.42) total: 4500 /unlink in 45.55 seconds: 98.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 589 (time 1783951888.45 total 10.01 last 58.85) - open/close 1201 (time 1783951898.46 total 20.02 last 61.16) - open/close 1821 (time 1783951908.46 total 30.02 last 61.96) - open/close 2345 (time 1783951918.47 total 40.03 last 52.37) - open/close 2903 (time 1783951928.50 total 50.06 last 55.64) - open/close 3417 (time 1783951938.51 total 60.07 last 51.34) - open/close 3983 (time 1783951948.53 total 70.09 last 56.50) total: 4500 open/close in 79.11 seconds: 56.89 ops/second - /unlink 1138 (time 1783951971.97 total 10.01 last 113.72) - /unlink 2236 (time 1783951981.98 total 20.02 last 109.69) - /unlink 3344 (time 1783951991.99 total 30.02 last 110.72) - /unlink 4489 (time 1783952002.00 total 40.03 last 114.45) total: 4500 /unlink in 40.12 seconds: 112.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 612 (time 1783952016.81 total 10.01 last 61.16) - open/close 1151 (time 1783952026.82 total 20.02 last 53.83) - open/close 1700 (time 1783952036.84 total 30.03 last 54.82) - open/close 2311 (time 1783952046.85 total 40.05 last 61.03) - open/close 2873 (time 1783952056.87 total 50.06 last 56.10) - open/close 3390 (time 1783952066.88 total 60.07 last 51.65) - open/close 3881 (time 1783952076.89 total 70.08 last 49.05) - open/close 4421 (time 1783952086.89 total 80.09 last 53.98) total: 4500 open/close in 81.30 seconds: 55.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1085 (time 1783952102.58 total 10.00 last 108.47) - /unlink 2206 (time 1783952112.58 total 20.01 last 112.04) - /unlink 3219 (time 1783952122.59 total 30.01 last 101.25) - /unlink 4361 (time 1783952132.59 total 40.02 last 114.15) total: 4500 /unlink in 41.72 seconds: 107.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 579 (time 1783952149.23 total 10.03 last 57.70) - open/close 1176 (time 1783952159.24 total 20.04 last 59.66) - open/close 1746 (time 1783952169.24 total 30.05 last 56.97) - open/close 2323 (time 1783952179.26 total 40.07 last 57.59) - open/close 2971 (time 1783952189.27 total 50.07 last 64.75) - open/close 3547 (time 1783952199.28 total 60.08 last 57.55) - open/close 4103 (time 1783952209.28 total 70.09 last 55.58) total: 4500 open/close in 77.04 seconds: 58.41 ops/second - /unlink 1115 (time 1783952231.26 total 10.00 last 111.48) - /unlink 2154 (time 1783952241.27 total 20.01 last 103.82) - /unlink 3184 (time 1783952251.27 total 30.01 last 102.96) - /unlink 4105 (time 1783952261.27 total 40.02 last 92.08) total: 4500 /unlink in 45.17 seconds: 99.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 518 (time 1783952283.57 total 10.01 last 51.77) - open/close 1121 (time 1783952293.58 total 20.01 last 60.24) - open/close 1717 (time 1783952303.59 total 30.02 last 59.58) - open/close 2288 (time 1783952313.59 total 40.03 last 57.05) - open/close 2892 (time 1783952323.60 total 50.03 last 60.39) - open/close 3485 (time 1783952333.60 total 60.03 last 59.29) - open/close 4081 (time 1783952343.60 total 70.03 last 59.58) total: 4500 open/close in 77.35 seconds: 58.18 ops/second - /unlink 982 (time 1783952366.32 total 10.00 last 98.19) - /unlink 1985 (time 1783952376.33 total 20.02 last 100.15) - /unlink 2956 (time 1783952386.33 total 30.02 last 97.06) - /unlink 3904 (time 1783952396.34 total 40.03 last 94.74) total: 4500 /unlink in 45.84 seconds: 98.17 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 75522 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server unloading modules via unload_modules_local on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: modules unloaded.