== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:57:19 (1786330639) Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg328-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg328-server oleg328-server: oleg328-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg328-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg328-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg328-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 720 (time 1786330838.33 total 10.01 last 71.95) - open/close 1490 (time 1786330848.35 total 20.03 last 76.83) - open/close 2183 (time 1786330858.36 total 30.04 last 69.25) - open/close 2841 (time 1786330868.37 total 40.04 last 65.75) - open/close 3569 (time 1786330878.38 total 50.06 last 72.71) - open/close 4339 (time 1786330888.38 total 60.06 last 76.96) total: 4500 open/close in 62.53 seconds: 71.97 ops/second - /unlink 1290 (time 1786330906.98 total 10.00 last 128.99) - /unlink 2585 (time 1786330916.99 total 20.02 last 129.31) - /unlink 3889 (time 1786330927.00 total 30.03 last 130.26) total: 4500 /unlink in 34.20 seconds: 131.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 744 (time 1786330948.42 total 10.02 last 74.29) - open/close 1488 (time 1786330958.42 total 20.02 last 74.36) - open/close 2212 (time 1786330968.43 total 30.03 last 72.34) - open/close 2927 (time 1786330978.44 total 40.04 last 71.43) - open/close 3642 (time 1786330988.47 total 50.07 last 71.31) - open/close 4322 (time 1786330998.48 total 60.08 last 67.92) total: 4500 open/close in 62.52 seconds: 71.98 ops/second - /unlink 1274 (time 1786331015.23 total 10.00 last 127.40) - /unlink 2503 (time 1786331025.24 total 20.01 last 122.79) - /unlink 3753 (time 1786331035.24 total 30.01 last 124.98) total: 4500 /unlink in 35.33 seconds: 127.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 770 (time 1786331054.75 total 10.01 last 76.92) - open/close 1543 (time 1786331064.76 total 20.02 last 77.23) - open/close 2360 (time 1786331074.76 total 30.02 last 81.66) - open/close 3143 (time 1786331084.77 total 40.03 last 78.25) - open/close 3919 (time 1786331094.79 total 50.05 last 77.41) total: 4500 open/close in 58.03 seconds: 77.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1285 (time 1786331116.75 total 10.02 last 128.29) - /unlink 2488 (time 1786331126.76 total 20.02 last 120.24) - /unlink 3669 (time 1786331136.79 total 30.05 last 117.77) total: 4500 /unlink in 36.75 seconds: 122.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 724 (time 1786331159.16 total 10.00 last 72.40) - open/close 1355 (time 1786331169.16 total 20.00 last 63.10) - open/close 2143 (time 1786331179.17 total 30.01 last 78.76) - open/close 2927 (time 1786331189.28 total 40.11 last 77.57) - open/close 3641 (time 1786331199.29 total 50.13 last 71.31) - open/close 4330 (time 1786331209.29 total 60.13 last 68.89) total: 4500 open/close in 62.27 seconds: 72.26 ops/second - /unlink 1264 (time 1786331225.99 total 10.00 last 126.37) - /unlink 2516 (time 1786331236.00 total 20.01 last 125.07) - /unlink 3767 (time 1786331246.01 total 30.02 last 125.00) total: 4500 /unlink in 36.07 seconds: 124.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 701 (time 1786331268.31 total 10.00 last 70.09) - open/close 1439 (time 1786331278.31 total 20.01 last 73.77) - open/close 2184 (time 1786331288.32 total 30.02 last 74.43) - open/close 2912 (time 1786331298.32 total 40.02 last 72.80) - open/close 3615 (time 1786331308.33 total 50.03 last 70.22) - open/close 4318 (time 1786331318.33 total 60.03 last 70.28) total: 4500 open/close in 62.18 seconds: 72.37 ops/second - /unlink 1210 (time 1786331335.97 total 10.00 last 120.98) - /unlink 2426 (time 1786331345.97 total 20.00 last 121.59) - /unlink 3619 (time 1786331356.00 total 30.03 last 118.95) total: 4500 /unlink in 36.78 seconds: 122.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 692 (time 1786331378.64 total 10.01 last 69.14) - open/close 1403 (time 1786331388.66 total 20.03 last 70.98) - open/close 2085 (time 1786331398.66 total 30.03 last 68.15) - open/close 2750 (time 1786331408.68 total 40.04 last 66.43) - open/close 3483 (time 1786331418.68 total 50.05 last 73.29) - open/close 4096 (time 1786331428.68 total 60.05 last 61.27) total: 4500 open/close in 65.22 seconds: 69.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1136 (time 1786331449.06 total 10.00 last 113.56) - /unlink 2445 (time 1786331459.09 total 20.03 last 130.61) - /unlink 3491 (time 1786331469.09 total 30.03 last 104.60) total: 4500 /unlink in 37.65 seconds: 119.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 739 (time 1786331493.23 total 10.00 last 73.87) - open/close 1449 (time 1786331503.23 total 20.00 last 71.00) - open/close 2216 (time 1786331513.25 total 30.02 last 76.60) - open/close 2909 (time 1786331523.25 total 40.02 last 69.29) - open/close 3579 (time 1786331533.25 total 50.02 last 66.98) - open/close 4326 (time 1786331543.26 total 60.03 last 74.64) total: 4500 open/close in 62.66 seconds: 71.81 ops/second - /unlink 1334 (time 1786331561.70 total 10.01 last 133.25) - /unlink 2503 (time 1786331571.70 total 20.01 last 116.89) - /unlink 3812 (time 1786331581.71 total 30.02 last 130.86) total: 4500 /unlink in 35.92 seconds: 125.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 688 (time 1786331602.93 total 10.00 last 68.78) - open/close 1358 (time 1786331612.94 total 20.01 last 66.95) - open/close 2117 (time 1786331622.94 total 30.01 last 75.88) - open/close 2881 (time 1786331632.94 total 40.01 last 76.39) - open/close 3615 (time 1786331642.95 total 50.02 last 73.35) - open/close 4370 (time 1786331652.95 total 60.03 last 75.47) total: 4500 open/close in 61.98 seconds: 72.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1271 (time 1786331670.46 total 10.00 last 127.05) - /unlink 2369 (time 1786331680.48 total 20.03 last 109.50) - /unlink 3667 (time 1786331690.49 total 30.03 last 129.76) total: 4500 /unlink in 36.74 seconds: 122.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 702 (time 1786331713.14 total 10.01 last 70.10) - open/close 1409 (time 1786331723.18 total 20.05 last 70.43) - open/close 2163 (time 1786331733.19 total 30.06 last 75.31) - open/close 2977 (time 1786331743.20 total 40.07 last 81.35) - open/close 3770 (time 1786331753.20 total 50.08 last 79.27) total: 4500 open/close in 59.91 seconds: 75.11 ops/second - /unlink 1248 (time 1786331779.31 total 10.00 last 124.77) - /unlink 2450 (time 1786331789.31 total 20.01 last 120.12) - /unlink 3707 (time 1786331799.33 total 30.03 last 125.46) total: 4500 /unlink in 35.77 seconds: 125.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 779 (time 1786331819.70 total 10.00 last 77.87) - open/close 1531 (time 1786331829.71 total 20.01 last 75.17) - open/close 2330 (time 1786331839.71 total 30.01 last 79.88) - open/close 3104 (time 1786331849.71 total 40.01 last 77.38) - open/close 3845 (time 1786331859.71 total 50.01 last 74.10) total: 4500 open/close in 59.49 seconds: 75.64 ops/second - /unlink 1178 (time 1786331883.90 total 10.00 last 117.79) - /unlink 2310 (time 1786331893.90 total 20.00 last 113.19) - /unlink 3583 (time 1786331903.90 total 30.01 last 127.23) total: 4500 /unlink in 37.12 seconds: 121.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 670 (time 1786331926.63 total 10.01 last 66.92) - open/close 1412 (time 1786331936.64 total 20.02 last 74.11) - open/close 2106 (time 1786331946.64 total 30.03 last 69.39) - open/close 2818 (time 1786331956.65 total 40.03 last 71.19) - open/close 3518 (time 1786331966.65 total 50.03 last 69.97) - open/close 4301 (time 1786331976.66 total 60.05 last 78.20) total: 4500 open/close in 62.65 seconds: 71.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1345 (time 1786331993.47 total 10.00 last 134.46) - /unlink 2769 (time 1786332003.48 total 20.01 last 142.36) - /unlink 4149 (time 1786332013.48 total 30.01 last 137.98) total: 4500 /unlink in 32.63 seconds: 137.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 793 (time 1786332031.03 total 10.00 last 79.30) - open/close 1550 (time 1786332041.03 total 20.01 last 75.66) - open/close 2330 (time 1786332051.04 total 30.01 last 77.94) - open/close 3046 (time 1786332061.05 total 40.03 last 71.50) - open/close 3815 (time 1786332071.06 total 50.03 last 76.89) total: 4500 open/close in 58.58 seconds: 76.81 ops/second - /unlink 1480 (time 1786332093.41 total 10.00 last 147.98) - /unlink 2903 (time 1786332103.42 total 20.00 last 142.28) - /unlink 4041 (time 1786332113.43 total 30.02 last 113.63) total: 4500 /unlink in 34.31 seconds: 131.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 773 (time 1786332133.02 total 10.00 last 77.28) - open/close 1454 (time 1786332143.02 total 20.00 last 68.10) - open/close 2189 (time 1786332153.04 total 30.02 last 73.36) - open/close 2949 (time 1786332163.05 total 40.03 last 75.95) - open/close 3761 (time 1786332173.06 total 50.04 last 81.09) - open/close 4473 (time 1786332183.07 total 60.05 last 71.14) total: 4500 open/close in 60.36 seconds: 74.56 ops/second - /unlink 1329 (time 1786332199.19 total 10.00 last 132.85) - /unlink 2606 (time 1786332209.20 total 20.01 last 127.59) - /unlink 3881 (time 1786332219.20 total 30.01 last 127.49) total: 4500 /unlink in 34.61 seconds: 130.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 755 (time 1786332238.18 total 10.00 last 75.49) - open/close 1455 (time 1786332248.20 total 20.02 last 69.84) - open/close 2178 (time 1786332258.21 total 30.03 last 72.23) - open/close 2898 (time 1786332268.22 total 40.04 last 71.94) - open/close 3675 (time 1786332278.22 total 50.04 last 77.70) - open/close 4443 (time 1786332288.23 total 60.05 last 76.76) total: 4500 open/close in 60.76 seconds: 74.06 ops/second - /unlink 1438 (time 1786332303.17 total 10.00 last 143.78) - /unlink 2717 (time 1786332313.17 total 20.01 last 127.85) - /unlink 3910 (time 1786332323.17 total 30.01 last 119.23) total: 4500 /unlink in 34.41 seconds: 130.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 728 (time 1786332342.99 total 10.00 last 72.78) - open/close 1387 (time 1786332352.99 total 20.00 last 65.90) - open/close 2091 (time 1786332363.00 total 30.01 last 70.32) - open/close 2855 (time 1786332373.01 total 40.02 last 76.37) - open/close 3651 (time 1786332383.02 total 50.03 last 79.52) - open/close 4478 (time 1786332393.02 total 60.03 last 82.70) total: 4500 open/close in 60.27 seconds: 74.67 ops/second - /unlink 1415 (time 1786332407.25 total 10.01 last 141.41) - /unlink 2766 (time 1786332417.26 total 20.01 last 135.09) - /unlink 4028 (time 1786332427.26 total 30.01 last 126.16) total: 4500 /unlink in 34.02 seconds: 132.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 740 (time 1786332446.20 total 10.01 last 73.92) - open/close 1407 (time 1786332456.21 total 20.02 last 66.64) - open/close 2063 (time 1786332466.21 total 30.02 last 65.58) - open/close 2733 (time 1786332476.27 total 40.08 last 66.60) - open/close 3456 (time 1786332486.27 total 50.08 last 72.29) - open/close 4137 (time 1786332496.29 total 60.10 last 68.00) total: 4500 open/close in 64.98 seconds: 69.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1336 (time 1786332516.53 total 10.00 last 133.55) - /unlink 2557 (time 1786332526.54 total 20.01 last 122.04) - /unlink 3844 (time 1786332536.54 total 30.01 last 128.62) total: 4500 /unlink in 34.52 seconds: 130.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 721 (time 1786332556.50 total 10.00 last 72.09) - open/close 1370 (time 1786332566.64 total 20.14 last 64.02) - open/close 2052 (time 1786332576.65 total 30.14 last 68.16) - open/close 2792 (time 1786332586.65 total 40.15 last 73.94) - open/close 3559 (time 1786332596.66 total 50.16 last 76.66) - open/close 4327 (time 1786332606.67 total 60.17 last 76.73) total: 4500 open/close in 62.52 seconds: 71.97 ops/second - /unlink 1242 (time 1786332623.51 total 10.00 last 124.18) - /unlink 2473 (time 1786332633.51 total 20.00 last 123.10) - /unlink 3738 (time 1786332643.51 total 30.00 last 126.48) total: 4500 /unlink in 35.70 seconds: 126.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 735 (time 1786332663.33 total 10.04 last 73.21) - open/close 1461 (time 1786332673.34 total 20.05 last 72.53) - open/close 2148 (time 1786332683.36 total 30.07 last 68.57) - open/close 2860 (time 1786332693.36 total 40.08 last 71.14) - open/close 3614 (time 1786332703.37 total 50.08 last 75.36) - open/close 4370 (time 1786332713.37 total 60.09 last 75.58) total: 4500 open/close in 61.73 seconds: 72.89 ops/second - /unlink 1317 (time 1786332728.88 total 10.00 last 131.66) - /unlink 2700 (time 1786332738.88 total 20.00 last 138.29) - /unlink 3888 (time 1786332748.88 total 30.01 last 118.78) total: 4500 /unlink in 34.55 seconds: 130.23 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95318 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg328-server unloading modules via unload_modules_local on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local oleg328-server: modules unloaded.