== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:06:26 (1779872786) Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg456-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg456-server oleg456-server: oleg456-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg456-server: detected 4 online CPUs by sysfs oleg456-server: MODOPTS_LIBCFS= oleg456-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg456-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg456-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg456-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 618 (time 1779873010.72 total 10.01 last 61.77) - open/close 1253 (time 1779873020.72 total 20.01 last 63.48) - open/close 1911 (time 1779873030.74 total 30.02 last 65.70) - open/close 2557 (time 1779873040.75 total 40.03 last 64.53) - open/close 3249 (time 1779873050.77 total 50.05 last 69.10) - open/close 3853 (time 1779873060.77 total 60.06 last 60.36) - open/close 4478 (time 1779873070.78 total 70.06 last 62.45) total: 4500 open/close in 70.57 seconds: 63.77 ops/second - /unlink 1149 (time 1779873085.84 total 10.01 last 114.84) - /unlink 2276 (time 1779873095.84 total 20.01 last 112.65) - /unlink 3447 (time 1779873105.84 total 30.01 last 117.10) total: 4500 /unlink in 38.56 seconds: 116.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 466 (time 1779873130.05 total 10.00 last 46.60) - open/close 1040 (time 1779873140.06 total 20.01 last 57.34) - open/close 1605 (time 1779873150.08 total 30.03 last 56.37) - open/close 2132 (time 1779873160.11 total 40.06 last 52.58) - open/close 2649 (time 1779873170.11 total 50.06 last 51.66) - open/close 3280 (time 1779873180.12 total 60.07 last 63.08) - open/close 4018 (time 1779873190.12 total 70.08 last 73.74) total: 4500 open/close in 78.89 seconds: 57.04 ops/second - /unlink 1203 (time 1779873213.53 total 10.00 last 120.27) - /unlink 2445 (time 1779873223.53 total 20.01 last 124.17) - /unlink 3615 (time 1779873233.53 total 30.01 last 116.99) total: 4500 /unlink in 37.35 seconds: 120.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 666 (time 1779873255.34 total 10.01 last 66.54) - open/close 1312 (time 1779873265.34 total 20.02 last 64.54) - open/close 1928 (time 1779873275.35 total 30.02 last 61.58) - open/close 2586 (time 1779873285.35 total 40.03 last 65.76) - open/close 3245 (time 1779873295.37 total 50.04 last 65.80) - open/close 3843 (time 1779873305.39 total 60.06 last 59.69) - open/close 4410 (time 1779873315.42 total 70.09 last 56.54) total: 4500 open/close in 71.81 seconds: 62.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1119 (time 1779873333.21 total 10.00 last 111.85) - /unlink 2181 (time 1779873343.22 total 20.01 last 106.13) - /unlink 3292 (time 1779873353.22 total 30.01 last 111.07) total: 4500 /unlink in 39.68 seconds: 113.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 668 (time 1779873377.57 total 10.02 last 66.66) - open/close 1304 (time 1779873387.58 total 20.03 last 63.52) - open/close 1921 (time 1779873397.59 total 30.04 last 61.64) - open/close 2543 (time 1779873407.59 total 40.04 last 62.18) - open/close 3228 (time 1779873417.60 total 50.06 last 68.43) - open/close 3871 (time 1779873427.61 total 60.06 last 64.24) total: 4500 open/close in 69.96 seconds: 64.33 ops/second - /unlink 1192 (time 1779873452.34 total 10.00 last 119.17) - /unlink 2335 (time 1779873462.35 total 20.01 last 114.23) - /unlink 3426 (time 1779873472.35 total 30.01 last 109.07) total: 4500 /unlink in 39.70 seconds: 113.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 577 (time 1779873496.62 total 10.01 last 57.65) - open/close 1210 (time 1779873506.63 total 20.02 last 63.20) - open/close 1810 (time 1779873516.64 total 30.03 last 59.95) - open/close 2467 (time 1779873526.65 total 40.04 last 65.63) - open/close 3103 (time 1779873536.65 total 50.04 last 63.60) - open/close 3763 (time 1779873546.66 total 60.06 last 65.92) - open/close 4433 (time 1779873556.67 total 70.06 last 66.97) total: 4500 open/close in 70.93 seconds: 63.45 ops/second - /unlink 1233 (time 1779873571.89 total 10.01 last 123.15) - /unlink 2466 (time 1779873581.89 total 20.01 last 123.29) - /unlink 3567 (time 1779873591.90 total 30.02 last 110.06) total: 4500 /unlink in 37.62 seconds: 119.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 607 (time 1779873614.27 total 10.02 last 60.60) - open/close 1274 (time 1779873624.29 total 20.03 last 66.61) - open/close 1866 (time 1779873634.29 total 30.03 last 59.19) - open/close 2439 (time 1779873644.31 total 40.05 last 57.18) - open/close 3104 (time 1779873654.31 total 50.06 last 66.47) - open/close 3740 (time 1779873664.32 total 60.06 last 63.57) - open/close 4426 (time 1779873674.33 total 70.07 last 68.52) total: 4500 open/close in 71.25 seconds: 63.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 909 (time 1779873690.56 total 10.01 last 90.84) - /unlink 1919 (time 1779873700.57 total 20.01 last 100.94) - /unlink 2918 (time 1779873710.57 total 30.01 last 99.89) - /unlink 3998 (time 1779873720.57 total 40.02 last 107.98) total: 4500 /unlink in 44.92 seconds: 100.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 639 (time 1779873740.36 total 10.01 last 63.83) - open/close 1248 (time 1779873750.38 total 20.03 last 60.81) - open/close 1889 (time 1779873760.38 total 30.03 last 64.07) - open/close 2493 (time 1779873770.39 total 40.04 last 60.33) - open/close 3071 (time 1779873780.40 total 50.05 last 57.78) - open/close 3660 (time 1779873790.40 total 60.05 last 58.89) - open/close 4309 (time 1779873800.40 total 70.05 last 64.87) total: 4500 open/close in 72.79 seconds: 61.83 ops/second - /unlink 1213 (time 1779873817.11 total 10.00 last 121.27) - /unlink 2361 (time 1779873827.12 total 20.01 last 114.68) - /unlink 3549 (time 1779873837.13 total 30.02 last 118.76) total: 4500 /unlink in 37.91 seconds: 118.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 641 (time 1779873859.18 total 10.01 last 64.05) - open/close 1291 (time 1779873869.19 total 20.02 last 64.93) - open/close 1916 (time 1779873879.20 total 30.03 last 62.43) - open/close 2603 (time 1779873889.21 total 40.03 last 68.68) - open/close 3210 (time 1779873899.21 total 50.03 last 60.70) - open/close 3866 (time 1779873909.21 total 60.04 last 65.59) total: 4500 open/close in 69.54 seconds: 64.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1209 (time 1779873932.64 total 10.00 last 120.85) - /unlink 2580 (time 1779873942.64 total 20.01 last 137.07) - /unlink 3822 (time 1779873952.64 total 30.01 last 124.18) total: 4500 /unlink in 35.19 seconds: 127.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 697 (time 1779873972.23 total 10.00 last 69.68) - open/close 1331 (time 1779873982.24 total 20.01 last 63.34) - open/close 2002 (time 1779873992.27 total 30.04 last 66.89) - open/close 2653 (time 1779874002.28 total 40.05 last 65.03) - open/close 3334 (time 1779874012.29 total 50.06 last 68.06) - open/close 3977 (time 1779874022.29 total 60.06 last 64.28) total: 4500 open/close in 67.06 seconds: 67.11 ops/second - /unlink 1275 (time 1779874042.91 total 10.00 last 127.45) - /unlink 2655 (time 1779874052.91 total 20.01 last 137.97) - /unlink 4009 (time 1779874062.92 total 30.01 last 135.37) total: 4500 /unlink in 33.30 seconds: 135.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 726 (time 1779874080.13 total 10.02 last 72.43) - open/close 1331 (time 1779874090.14 total 20.03 last 60.48) - open/close 1928 (time 1779874100.15 total 30.04 last 59.64) - open/close 2365 (time 1779874110.20 total 40.09 last 43.46) - open/close 2745 (time 1779874120.22 total 50.11 last 37.94) - open/close 3193 (time 1779874130.23 total 60.12 last 44.74) - open/close 3781 (time 1779874140.23 total 70.12 last 58.79) - open/close 4326 (time 1779874150.26 total 80.14 last 54.39) total: 4500 open/close in 84.20 seconds: 53.44 ops/second - /unlink 1029 (time 1779874170.21 total 10.00 last 102.89) - /unlink 2102 (time 1779874180.22 total 20.01 last 107.23) - /unlink 3223 (time 1779874190.23 total 30.02 last 112.01) - /unlink 4458 (time 1779874200.23 total 40.02 last 123.40) total: 4500 /unlink in 40.45 seconds: 111.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 650 (time 1779874214.67 total 10.00 last 64.97) - open/close 1311 (time 1779874224.68 total 20.02 last 65.99) - open/close 1975 (time 1779874234.69 total 30.03 last 66.34) - open/close 2616 (time 1779874244.70 total 40.04 last 64.04) - open/close 3187 (time 1779874254.71 total 50.04 last 57.07) - open/close 3711 (time 1779874264.71 total 60.05 last 52.35) - open/close 4197 (time 1779874274.72 total 70.06 last 48.57) total: 4500 open/close in 74.89 seconds: 60.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1199 (time 1779874295.36 total 10.00 last 119.86) - /unlink 2223 (time 1779874305.37 total 20.01 last 102.36) - /unlink 3324 (time 1779874315.37 total 30.01 last 110.10) - /unlink 4422 (time 1779874325.37 total 40.01 last 109.76) total: 4500 /unlink in 40.67 seconds: 110.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 596 (time 1779874341.21 total 10.03 last 59.44) - open/close 1174 (time 1779874351.22 total 20.03 last 57.76) - open/close 1677 (time 1779874361.23 total 30.05 last 50.24) - open/close 2208 (time 1779874371.24 total 40.05 last 53.08) - open/close 2752 (time 1779874381.24 total 50.06 last 54.37) - open/close 3378 (time 1779874391.24 total 60.06 last 62.60) - open/close 3925 (time 1779874401.25 total 70.06 last 54.68) total: 4500 open/close in 78.82 seconds: 57.09 ops/second - /unlink 1203 (time 1779874424.44 total 10.01 last 120.24) - /unlink 2339 (time 1779874434.44 total 20.01 last 113.58) - /unlink 3455 (time 1779874444.45 total 30.01 last 111.59) total: 4500 /unlink in 39.26 seconds: 114.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 625 (time 1779874468.51 total 10.01 last 62.47) - open/close 1252 (time 1779874478.51 total 20.01 last 62.69) - open/close 1890 (time 1779874488.54 total 30.03 last 63.63) - open/close 2535 (time 1779874498.54 total 40.04 last 64.48) - open/close 3129 (time 1779874508.54 total 50.04 last 59.39) - open/close 3745 (time 1779874518.56 total 60.05 last 61.52) - open/close 4320 (time 1779874528.56 total 70.05 last 57.49) total: 4500 open/close in 72.99 seconds: 61.65 ops/second - /unlink 1034 (time 1779874547.30 total 10.00 last 103.38) - /unlink 1987 (time 1779874557.30 total 20.00 last 95.28) - /unlink 3116 (time 1779874567.31 total 30.01 last 112.82) - /unlink 4288 (time 1779874577.31 total 40.01 last 117.19) total: 4500 /unlink in 42.67 seconds: 105.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 539 (time 1779874596.81 total 10.00 last 53.88) - open/close 1094 (time 1779874606.82 total 20.02 last 55.43) - open/close 1657 (time 1779874616.83 total 30.03 last 56.24) - open/close 2321 (time 1779874626.84 total 40.03 last 66.35) - open/close 2982 (time 1779874636.84 total 50.04 last 66.09) - open/close 3607 (time 1779874646.85 total 60.05 last 62.44) - open/close 4264 (time 1779874656.85 total 70.05 last 65.67) total: 4500 open/close in 73.56 seconds: 61.17 ops/second - /unlink 1056 (time 1779874674.69 total 10.00 last 105.59) - /unlink 2190 (time 1779874684.70 total 20.01 last 113.32) - /unlink 3336 (time 1779874694.72 total 30.02 last 114.43) - /unlink 4403 (time 1779874704.72 total 40.03 last 106.62) total: 4500 /unlink in 40.96 seconds: 109.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 696 (time 1779874720.19 total 10.01 last 69.51) - open/close 1326 (time 1779874730.20 total 20.02 last 62.94) - open/close 1956 (time 1779874740.21 total 30.04 last 62.91) - open/close 2591 (time 1779874750.23 total 40.05 last 63.39) - open/close 3185 (time 1779874760.24 total 50.06 last 59.35) - open/close 3748 (time 1779874770.24 total 60.07 last 56.26) - open/close 4258 (time 1779874780.25 total 70.08 last 50.96) total: 4500 open/close in 74.15 seconds: 60.69 ops/second - /unlink 1068 (time 1779874799.44 total 10.00 last 106.75) - /unlink 2187 (time 1779874809.44 total 20.01 last 111.83) - /unlink 3369 (time 1779874819.45 total 30.01 last 118.18) total: 4500 /unlink in 39.76 seconds: 113.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 647 (time 1779874844.07 total 10.01 last 64.64) - open/close 1229 (time 1779874854.07 total 20.01 last 58.19) - open/close 1795 (time 1779874864.08 total 30.02 last 56.53) - open/close 2418 (time 1779874874.11 total 40.06 last 62.09) - open/close 3085 (time 1779874884.11 total 50.06 last 66.68) - open/close 3649 (time 1779874894.14 total 60.08 last 56.26) - open/close 4256 (time 1779874904.15 total 70.09 last 60.64) total: 4500 open/close in 74.15 seconds: 60.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1223 (time 1779874923.25 total 10.01 last 122.21) - /unlink 2343 (time 1779874933.25 total 20.01 last 111.95) - /unlink 3540 (time 1779874943.26 total 30.01 last 119.68) total: 4500 /unlink in 38.20 seconds: 117.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 650 (time 1779874966.90 total 10.02 last 64.89) - open/close 1328 (time 1779874976.91 total 20.03 last 67.70) - open/close 1927 (time 1779874986.91 total 30.03 last 59.89) - open/close 2592 (time 1779874996.92 total 40.04 last 66.45) - open/close 3208 (time 1779875006.93 total 50.05 last 61.55) - open/close 3774 (time 1779875016.94 total 60.06 last 56.57) - open/close 4388 (time 1779875026.95 total 70.07 last 61.34) total: 4500 open/close in 72.09 seconds: 62.42 ops/second - /unlink 1269 (time 1779875043.20 total 10.00 last 126.86) - /unlink 2440 (time 1779875053.21 total 20.01 last 117.03) - /unlink 3686 (time 1779875063.23 total 30.03 last 124.34) total: 4500 /unlink in 37.32 seconds: 120.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 496 (time 1779875085.09 total 10.00 last 49.59) - open/close 904 (time 1779875095.10 total 20.01 last 40.78) - open/close 1449 (time 1779875105.10 total 30.01 last 54.48) - open/close 2082 (time 1779875115.10 total 40.01 last 63.30) - open/close 2715 (time 1779875125.11 total 50.02 last 63.21) - open/close 3288 (time 1779875135.12 total 60.03 last 57.25) - open/close 3896 (time 1779875145.13 total 70.04 last 60.75) - open/close 4431 (time 1779875155.13 total 80.04 last 53.49) total: 4500 open/close in 80.92 seconds: 55.61 ops/second - /unlink 1266 (time 1779875170.52 total 10.00 last 126.59) - /unlink 2503 (time 1779875180.53 total 20.01 last 123.58) - /unlink 3624 (time 1779875190.53 total 30.01 last 112.08) total: 4500 /unlink in 36.63 seconds: 122.84 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93828 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg456-server unloading modules via unload_modules_local on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local oleg456-server: modules unloaded.