== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:15:01 (1773609301) Stopping clients: oleg329-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg329-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg329-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg329-server oleg329-server: oleg329-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg329-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg329-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg329-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 603 (time 1773609513.76 total 10.02 last 60.17) - open/close 1339 (time 1773609523.78 total 20.04 last 73.44) - open/close 2153 (time 1773609533.80 total 30.06 last 81.28) - open/close 2901 (time 1773609543.83 total 40.09 last 74.56) - open/close 3618 (time 1773609553.84 total 50.10 last 71.64) - open/close 4340 (time 1773609563.87 total 60.12 last 72.02) total: 4500 open/close in 62.74 seconds: 71.73 ops/second - /unlink 1360 (time 1773609581.27 total 10.00 last 135.99) - /unlink 2666 (time 1773609591.28 total 20.00 last 130.57) - /unlink 3865 (time 1773609601.28 total 30.01 last 119.85) total: 4500 /unlink in 34.87 seconds: 129.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 658 (time 1773609621.16 total 10.00 last 65.77) - open/close 1291 (time 1773609631.18 total 20.02 last 63.21) - open/close 1957 (time 1773609641.20 total 30.04 last 66.47) - open/close 2584 (time 1773609651.21 total 40.05 last 62.60) - open/close 3110 (time 1773609661.21 total 50.06 last 52.58) - open/close 4005 (time 1773609671.22 total 60.06 last 89.46) total: 4500 open/close in 67.27 seconds: 66.89 ops/second - /unlink 1285 (time 1773609693.91 total 10.00 last 128.46) - /unlink 2677 (time 1773609703.91 total 20.01 last 139.17) - /unlink 4092 (time 1773609713.92 total 30.01 last 141.44) total: 4500 /unlink in 33.31 seconds: 135.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 744 (time 1773609733.76 total 10.00 last 74.39) - open/close 1439 (time 1773609743.77 total 20.00 last 69.49) - open/close 2116 (time 1773609753.78 total 30.02 last 67.58) - open/close 2749 (time 1773609763.78 total 40.02 last 63.29) - open/close 3448 (time 1773609773.79 total 50.02 last 69.88) - open/close 4249 (time 1773609783.80 total 60.04 last 80.00) total: 4500 open/close in 64.55 seconds: 69.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1241 (time 1773609802.78 total 10.01 last 124.01) - /unlink 2550 (time 1773609812.79 total 20.01 last 130.82) - /unlink 3753 (time 1773609822.79 total 30.02 last 120.26) total: 4500 /unlink in 35.58 seconds: 126.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 828 (time 1773609843.70 total 10.03 last 82.59) - open/close 1614 (time 1773609853.73 total 20.06 last 78.33) - open/close 2385 (time 1773609863.76 total 30.08 last 76.93) - open/close 3140 (time 1773609873.77 total 40.10 last 75.38) - open/close 3884 (time 1773609883.79 total 50.12 last 74.23) total: 4500 open/close in 57.66 seconds: 78.05 ops/second - /unlink 1361 (time 1773609906.03 total 10.00 last 136.04) - /unlink 2793 (time 1773609916.03 total 20.01 last 143.16) - /unlink 4183 (time 1773609926.03 total 30.01 last 138.94) total: 4500 /unlink in 32.89 seconds: 136.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 643 (time 1773609944.81 total 10.02 last 64.19) - open/close 1339 (time 1773609954.82 total 20.03 last 69.51) - open/close 1908 (time 1773609964.83 total 30.03 last 56.87) - open/close 2313 (time 1773609974.87 total 40.07 last 40.35) - open/close 2735 (time 1773609984.90 total 50.10 last 42.08) - open/close 3391 (time 1773609994.91 total 60.11 last 65.52) - open/close 4008 (time 1773610004.92 total 70.13 last 61.63) total: 4500 open/close in 77.19 seconds: 58.30 ops/second - /unlink 1022 (time 1773610027.51 total 10.00 last 102.18) - /unlink 2528 (time 1773610037.51 total 20.00 last 150.59) - /unlink 3893 (time 1773610047.52 total 30.01 last 136.45) total: 4500 /unlink in 33.92 seconds: 132.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 732 (time 1773610067.45 total 10.00 last 73.19) - open/close 1531 (time 1773610077.45 total 20.01 last 79.84) - open/close 2247 (time 1773610087.46 total 30.02 last 71.54) - open/close 3017 (time 1773610097.47 total 40.03 last 76.93) - open/close 3707 (time 1773610107.48 total 50.03 last 68.95) total: 4500 open/close in 58.90 seconds: 76.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1174 (time 1773610131.01 total 10.00 last 117.37) - /unlink 2590 (time 1773610141.01 total 20.00 last 141.59) - /unlink 3894 (time 1773610151.02 total 30.01 last 130.34) total: 4500 /unlink in 35.01 seconds: 128.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 590 (time 1773610171.36 total 10.01 last 58.96) - open/close 1372 (time 1773610181.37 total 20.01 last 78.16) - open/close 2068 (time 1773610191.38 total 30.02 last 69.55) - open/close 2885 (time 1773610201.38 total 40.02 last 81.69) - open/close 3568 (time 1773610211.39 total 50.03 last 68.22) - open/close 4329 (time 1773610221.41 total 60.05 last 75.97) total: 4500 open/close in 63.26 seconds: 71.14 ops/second - /unlink 985 (time 1773610240.33 total 10.01 last 98.43) - /unlink 2008 (time 1773610250.34 total 20.01 last 102.25) - /unlink 3005 (time 1773610260.35 total 30.02 last 99.60) - /unlink 4329 (time 1773610270.35 total 40.02 last 132.37) total: 4500 /unlink in 41.64 seconds: 108.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 558 (time 1773610287.99 total 10.01 last 55.75) - open/close 1137 (time 1773610298.00 total 20.02 last 57.84) - open/close 1894 (time 1773610308.01 total 30.03 last 75.61) - open/close 2575 (time 1773610318.03 total 40.05 last 67.97) - open/close 3214 (time 1773610328.04 total 50.07 last 63.80) - open/close 3966 (time 1773610338.05 total 60.07 last 75.17) total: 4500 open/close in 66.95 seconds: 67.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1185 (time 1773610359.28 total 10.00 last 118.50) - /unlink 2640 (time 1773610369.28 total 20.00 last 145.48) - /unlink 3978 (time 1773610379.29 total 30.01 last 133.74) total: 4500 /unlink in 33.88 seconds: 132.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 601 (time 1773610400.63 total 10.00 last 60.09) - open/close 1231 (time 1773610410.64 total 20.01 last 62.97) - open/close 1916 (time 1773610420.64 total 30.01 last 68.46) - open/close 2624 (time 1773610430.68 total 40.05 last 70.54) - open/close 3333 (time 1773610440.69 total 50.06 last 70.83) - open/close 4053 (time 1773610450.70 total 60.07 last 71.90) total: 4500 open/close in 67.12 seconds: 67.04 ops/second - /unlink 1275 (time 1773610472.58 total 10.00 last 127.48) - /unlink 2795 (time 1773610482.59 total 20.01 last 151.94) - /unlink 3939 (time 1773610492.59 total 30.01 last 114.38) total: 4500 /unlink in 34.54 seconds: 130.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 758 (time 1773610512.15 total 10.02 last 75.68) - open/close 1513 (time 1773610522.16 total 20.02 last 75.46) - open/close 2275 (time 1773610532.16 total 30.02 last 76.17) - open/close 2998 (time 1773610542.17 total 40.03 last 72.23) - open/close 3750 (time 1773610552.19 total 50.06 last 75.03) - open/close 4401 (time 1773610562.21 total 60.08 last 64.97) total: 4500 open/close in 61.90 seconds: 72.70 ops/second - /unlink 1204 (time 1773610579.02 total 10.00 last 120.35) - /unlink 2562 (time 1773610589.02 total 20.01 last 135.78) - /unlink 4087 (time 1773610599.02 total 30.01 last 152.46) total: 4500 /unlink in 32.96 seconds: 136.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 752 (time 1773610617.66 total 10.01 last 75.15) - open/close 1560 (time 1773610627.67 total 20.01 last 80.77) - open/close 2262 (time 1773610637.68 total 30.02 last 70.11) - open/close 2899 (time 1773610647.69 total 40.04 last 63.59) - open/close 3564 (time 1773610657.70 total 50.05 last 66.45) - open/close 4163 (time 1773610667.71 total 60.06 last 59.84) total: 4500 open/close in 66.07 seconds: 68.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1107 (time 1773610689.37 total 10.00 last 110.66) - /unlink 2261 (time 1773610699.38 total 20.02 last 115.25) - /unlink 3288 (time 1773610709.39 total 30.02 last 102.66) total: 4500 /unlink in 39.03 seconds: 115.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 576 (time 1773610732.96 total 10.00 last 57.58) - open/close 1272 (time 1773610742.96 total 20.01 last 69.56) - open/close 1971 (time 1773610752.97 total 30.01 last 69.87) - open/close 2786 (time 1773610762.98 total 40.02 last 81.44) - open/close 3586 (time 1773610772.99 total 50.03 last 79.92) - open/close 4357 (time 1773610782.99 total 60.03 last 77.07) total: 4500 open/close in 62.32 seconds: 72.20 ops/second - /unlink 1168 (time 1773610800.60 total 10.00 last 116.77) - /unlink 2431 (time 1773610810.61 total 20.01 last 126.25) - /unlink 3709 (time 1773610820.61 total 30.01 last 127.75) total: 4500 /unlink in 37.21 seconds: 120.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 777 (time 1773610842.89 total 10.01 last 77.65) - open/close 1532 (time 1773610852.89 total 20.01 last 75.49) - open/close 2283 (time 1773610862.94 total 30.05 last 74.79) - open/close 3031 (time 1773610872.94 total 40.06 last 74.76) - open/close 3807 (time 1773610882.94 total 50.06 last 77.59) total: 4500 open/close in 59.88 seconds: 75.15 ops/second - /unlink 1477 (time 1773610907.14 total 10.00 last 147.66) - /unlink 2720 (time 1773610917.15 total 20.01 last 124.19) - /unlink 3944 (time 1773610927.16 total 30.02 last 122.27) total: 4500 /unlink in 34.41 seconds: 130.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 783 (time 1773610948.18 total 10.00 last 78.27) - open/close 1526 (time 1773610958.18 total 20.00 last 74.29) - open/close 2343 (time 1773610968.19 total 30.01 last 81.64) - open/close 3115 (time 1773610978.20 total 40.02 last 77.10) - open/close 3866 (time 1773610988.20 total 50.02 last 75.10) total: 4500 open/close in 59.06 seconds: 76.20 ops/second - /unlink 1221 (time 1773611012.18 total 10.00 last 122.08) - /unlink 2405 (time 1773611022.18 total 20.01 last 118.36) - /unlink 3485 (time 1773611032.18 total 30.01 last 107.96) total: 4500 /unlink in 37.73 seconds: 119.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 531 (time 1773611056.17 total 10.01 last 53.06) - open/close 1196 (time 1773611066.18 total 20.01 last 66.47) - open/close 2058 (time 1773611076.20 total 30.04 last 86.00) - open/close 2795 (time 1773611086.21 total 40.04 last 73.66) - open/close 3655 (time 1773611096.21 total 50.05 last 85.93) total: 4500 open/close in 59.28 seconds: 75.91 ops/second - /unlink 1421 (time 1773611119.71 total 10.00 last 142.06) - /unlink 2762 (time 1773611129.72 total 20.01 last 133.98) - /unlink 4086 (time 1773611139.73 total 30.02 last 132.35) total: 4500 /unlink in 33.27 seconds: 135.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 671 (time 1773611158.12 total 10.01 last 67.05) - open/close 1308 (time 1773611168.12 total 20.01 last 63.69) - open/close 2114 (time 1773611178.13 total 30.02 last 80.55) - open/close 2860 (time 1773611188.13 total 40.02 last 74.56) - open/close 3669 (time 1773611198.17 total 50.06 last 80.59) total: 4500 open/close in 58.78 seconds: 76.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1310 (time 1773611221.21 total 10.01 last 130.91) - /unlink 2639 (time 1773611231.22 total 20.01 last 132.86) - /unlink 3982 (time 1773611241.23 total 30.02 last 134.16) total: 4500 /unlink in 33.76 seconds: 133.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 702 (time 1773611260.67 total 10.01 last 70.12) - open/close 1608 (time 1773611270.67 total 20.02 last 90.53) - open/close 2501 (time 1773611280.68 total 30.03 last 89.23) - open/close 3336 (time 1773611290.71 total 40.05 last 83.26) - open/close 4190 (time 1773611300.73 total 50.07 last 85.26) total: 4500 open/close in 53.69 seconds: 83.81 ops/second - /unlink 1424 (time 1773611319.19 total 10.00 last 142.37) - /unlink 3083 (time 1773611329.19 total 20.00 last 165.90) total: 4500 /unlink in 29.95 seconds: 150.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 884 (time 1773611353.64 total 10.00 last 88.36) - open/close 1706 (time 1773611363.64 total 20.01 last 82.17) - open/close 2467 (time 1773611373.65 total 30.01 last 76.08) - open/close 2990 (time 1773611383.66 total 40.03 last 52.22) - open/close 3577 (time 1773611393.66 total 50.03 last 58.69) - open/close 4400 (time 1773611403.68 total 60.05 last 82.14) total: 4500 open/close in 61.23 seconds: 73.50 ops/second - /unlink 1564 (time 1773611419.55 total 10.00 last 156.33) - /unlink 3046 (time 1773611429.56 total 20.01 last 148.13) - /unlink 4248 (time 1773611439.56 total 30.01 last 120.15) total: 4500 /unlink in 32.11 seconds: 140.14 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11170: 94238 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local oleg329-server: modules unloaded.