== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:32:51 (1783971171) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg160-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server oleg160-server: oleg160-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: MODOPTS_LIBCFS= oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg160-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg160-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 677 (time 1783971411.54 total 10.01 last 67.65) - open/close 1280 (time 1783971421.54 total 20.01 last 60.28) - open/close 1847 (time 1783971431.54 total 30.01 last 56.70) - open/close 2405 (time 1783971441.55 total 40.02 last 55.77) - open/close 2986 (time 1783971451.55 total 50.02 last 58.07) - open/close 3462 (time 1783971461.55 total 60.02 last 47.60) - open/close 3980 (time 1783971471.55 total 70.02 last 51.80) - open/close 4472 (time 1783971481.58 total 80.05 last 49.07) total: 4500 open/close in 80.83 seconds: 55.67 ops/second - /unlink 1021 (time 1783971498.70 total 10.01 last 102.04) - /unlink 1824 (time 1783971508.70 total 20.01 last 80.27) - /unlink 2795 (time 1783971518.70 total 30.01 last 97.07) - /unlink 3644 (time 1783971528.70 total 40.01 last 84.90) total: 4500 /unlink in 49.19 seconds: 91.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 563 (time 1783971554.79 total 10.00 last 56.30) - open/close 1135 (time 1783971564.82 total 20.02 last 57.08) - open/close 1636 (time 1783971574.82 total 30.03 last 50.08) - open/close 2208 (time 1783971584.82 total 40.03 last 57.17) - open/close 2780 (time 1783971594.84 total 50.05 last 57.08) - open/close 3307 (time 1783971604.85 total 60.06 last 52.65) - open/close 3802 (time 1783971614.86 total 70.07 last 49.47) - open/close 4327 (time 1783971624.87 total 80.08 last 52.44) total: 4500 open/close in 83.70 seconds: 53.76 ops/second - /unlink 1086 (time 1783971644.54 total 10.01 last 108.53) - /unlink 2023 (time 1783971654.54 total 20.01 last 93.67) - /unlink 3099 (time 1783971664.54 total 30.01 last 107.57) - /unlink 4303 (time 1783971674.55 total 40.02 last 120.33) total: 4500 /unlink in 42.07 seconds: 106.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 513 (time 1783971692.69 total 10.02 last 51.20) - open/close 1034 (time 1783971702.70 total 20.03 last 52.03) - open/close 1638 (time 1783971712.71 total 30.04 last 60.37) - open/close 2168 (time 1783971722.72 total 40.04 last 52.96) - open/close 2729 (time 1783971732.73 total 50.06 last 55.99) - open/close 3282 (time 1783971742.74 total 60.07 last 55.28) - open/close 3916 (time 1783971752.75 total 70.08 last 63.30) - open/close 4430 (time 1783971762.77 total 80.10 last 51.30) total: 4500 open/close in 81.55 seconds: 55.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1102 (time 1783971780.27 total 10.01 last 110.14) - /unlink 2084 (time 1783971790.28 total 20.01 last 98.12) - /unlink 3075 (time 1783971800.28 total 30.02 last 99.06) - /unlink 4199 (time 1783971810.29 total 40.02 last 112.37) total: 4500 /unlink in 42.77 seconds: 105.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 626 (time 1783971828.85 total 10.00 last 62.57) - open/close 1270 (time 1783971838.85 total 20.01 last 64.36) - open/close 1855 (time 1783971848.86 total 30.01 last 58.48) - open/close 2404 (time 1783971858.88 total 40.03 last 54.79) - open/close 3027 (time 1783971868.88 total 50.04 last 62.29) - open/close 3673 (time 1783971878.89 total 60.04 last 64.56) - open/close 4343 (time 1783971888.89 total 70.05 last 66.96) total: 4500 open/close in 71.88 seconds: 62.60 ops/second - /unlink 1042 (time 1783971906.29 total 10.01 last 104.12) - /unlink 2014 (time 1783971916.30 total 20.01 last 97.16) - /unlink 3233 (time 1783971926.30 total 30.02 last 121.82) - /unlink 4408 (time 1783971936.31 total 40.02 last 117.43) total: 4500 /unlink in 40.75 seconds: 110.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 593 (time 1783971952.66 total 10.01 last 59.26) - open/close 1116 (time 1783971962.68 total 20.02 last 52.22) - open/close 1774 (time 1783971972.68 total 30.03 last 65.78) - open/close 2422 (time 1783971982.68 total 40.03 last 64.79) - open/close 3006 (time 1783971992.69 total 50.03 last 58.38) - open/close 3659 (time 1783972002.69 total 60.03 last 65.28) - open/close 4266 (time 1783972012.71 total 70.05 last 60.59) total: 4500 open/close in 73.65 seconds: 61.10 ops/second - /unlink 1180 (time 1783972031.49 total 10.01 last 117.93) - /unlink 2065 (time 1783972041.49 total 20.01 last 88.48) - /unlink 2852 (time 1783972051.49 total 30.01 last 78.67) - /unlink 4015 (time 1783972061.50 total 40.02 last 116.16) total: 4500 /unlink in 43.67 seconds: 103.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 664 (time 1783972081.29 total 10.02 last 66.26) - open/close 1347 (time 1783972091.30 total 20.03 last 68.24) - open/close 2074 (time 1783972101.31 total 30.04 last 72.64) - open/close 2813 (time 1783972111.31 total 40.04 last 73.88) - open/close 3435 (time 1783972121.33 total 50.06 last 62.07) - open/close 3995 (time 1783972131.33 total 60.06 last 55.98) total: 4500 open/close in 67.46 seconds: 66.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1239 (time 1783972155.00 total 10.03 last 123.58) - /unlink 2482 (time 1783972165.01 total 20.03 last 124.19) - /unlink 3644 (time 1783972175.01 total 30.03 last 116.20) total: 4500 /unlink in 38.25 seconds: 117.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 633 (time 1783972199.05 total 10.01 last 63.21) - open/close 1379 (time 1783972209.06 total 20.02 last 74.53) - open/close 2050 (time 1783972219.07 total 30.03 last 67.07) - open/close 2647 (time 1783972229.07 total 40.03 last 59.66) - open/close 3216 (time 1783972239.07 total 50.03 last 56.90) - open/close 3826 (time 1783972249.08 total 60.04 last 60.96) total: 4500 open/close in 69.58 seconds: 64.68 ops/second - /unlink 1198 (time 1783972274.45 total 10.01 last 119.72) - /unlink 2387 (time 1783972284.45 total 20.01 last 118.88) - /unlink 3647 (time 1783972294.45 total 30.01 last 125.99) total: 4500 /unlink in 36.89 seconds: 121.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 559 (time 1783972317.17 total 10.01 last 55.83) - open/close 1085 (time 1783972327.17 total 20.01 last 52.60) - open/close 1637 (time 1783972337.19 total 30.03 last 55.10) - open/close 2241 (time 1783972347.20 total 40.04 last 60.32) - open/close 2918 (time 1783972357.21 total 50.05 last 67.63) - open/close 3472 (time 1783972367.21 total 60.06 last 55.39) - open/close 4109 (time 1783972377.25 total 70.10 last 63.44) total: 4500 open/close in 75.69 seconds: 59.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1200 (time 1783972398.68 total 10.00 last 119.95) - /unlink 2497 (time 1783972408.68 total 20.01 last 129.68) - /unlink 3738 (time 1783972418.69 total 30.01 last 124.05) total: 4500 /unlink in 36.54 seconds: 123.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 627 (time 1783972440.98 total 10.00 last 62.68) - open/close 1343 (time 1783972450.99 total 20.01 last 71.56) - open/close 2110 (time 1783972460.99 total 30.02 last 76.64) - open/close 2829 (time 1783972471.00 total 40.03 last 71.82) - open/close 3402 (time 1783972481.01 total 50.03 last 57.28) - open/close 4077 (time 1783972491.01 total 60.03 last 67.47) total: 4500 open/close in 66.41 seconds: 67.76 ops/second - /unlink 1185 (time 1783972513.50 total 10.00 last 118.49) - /unlink 2507 (time 1783972523.51 total 20.01 last 132.10) - /unlink 3765 (time 1783972533.52 total 30.02 last 125.69) total: 4500 /unlink in 35.41 seconds: 127.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 619 (time 1783972554.00 total 10.01 last 61.84) - open/close 1229 (time 1783972564.01 total 20.01 last 60.98) - open/close 1899 (time 1783972574.02 total 30.03 last 66.90) - open/close 2562 (time 1783972584.03 total 40.04 last 66.24) - open/close 3192 (time 1783972594.05 total 50.05 last 62.91) - open/close 3907 (time 1783972604.06 total 60.07 last 71.41) total: 4500 open/close in 68.30 seconds: 65.89 ops/second - /unlink 1279 (time 1783972627.17 total 10.00 last 127.89) - /unlink 2559 (time 1783972637.17 total 20.01 last 127.94) - /unlink 3770 (time 1783972647.17 total 30.01 last 121.09) total: 4500 /unlink in 37.37 seconds: 120.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 744 (time 1783972669.90 total 10.00 last 74.38) - open/close 1382 (time 1783972679.91 total 20.01 last 63.76) - open/close 2067 (time 1783972689.91 total 30.01 last 68.48) - open/close 2747 (time 1783972699.91 total 40.01 last 68.00) - open/close 3528 (time 1783972709.92 total 50.02 last 78.03) - open/close 4229 (time 1783972719.93 total 60.04 last 69.99) total: 4500 open/close in 64.21 seconds: 70.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1399 (time 1783972739.50 total 10.00 last 139.87) - /unlink 2593 (time 1783972749.51 total 20.01 last 119.28) - /unlink 3790 (time 1783972759.51 total 30.01 last 119.68) total: 4500 /unlink in 36.14 seconds: 124.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 641 (time 1783972781.92 total 10.01 last 64.02) - open/close 1308 (time 1783972791.92 total 20.01 last 66.69) - open/close 2050 (time 1783972801.93 total 30.02 last 74.15) - open/close 2721 (time 1783972811.93 total 40.03 last 67.06) - open/close 3475 (time 1783972821.94 total 50.03 last 75.36) - open/close 4194 (time 1783972831.94 total 60.04 last 71.87) total: 4500 open/close in 64.25 seconds: 70.04 ops/second - /unlink 1259 (time 1783972851.91 total 10.00 last 125.88) - /unlink 2594 (time 1783972861.91 total 20.00 last 133.50) - /unlink 3827 (time 1783972871.92 total 30.00 last 123.30) total: 4500 /unlink in 35.60 seconds: 126.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 680 (time 1783972892.19 total 10.00 last 67.99) - open/close 1391 (time 1783972902.19 total 20.00 last 71.08) - open/close 2040 (time 1783972912.20 total 30.01 last 64.84) - open/close 2738 (time 1783972922.21 total 40.02 last 69.73) - open/close 3413 (time 1783972932.22 total 50.03 last 67.45) - open/close 4159 (time 1783972942.23 total 60.04 last 74.55) total: 4500 open/close in 64.48 seconds: 69.79 ops/second - /unlink 1147 (time 1783972961.61 total 10.01 last 114.57) - /unlink 2497 (time 1783972971.61 total 20.01 last 134.98) - /unlink 3838 (time 1783972981.61 total 30.02 last 134.07) total: 4500 /unlink in 36.42 seconds: 123.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 639 (time 1783973003.69 total 10.00 last 63.89) - open/close 1473 (time 1783973013.69 total 20.00 last 83.39) - open/close 2178 (time 1783973023.72 total 30.03 last 70.33) - open/close 2974 (time 1783973033.74 total 40.06 last 79.38) - open/close 3657 (time 1783973043.75 total 50.06 last 68.27) - open/close 4337 (time 1783973053.75 total 60.07 last 67.95) total: 4500 open/close in 62.00 seconds: 72.58 ops/second - /unlink 1131 (time 1783973070.35 total 10.00 last 113.06) - /unlink 2472 (time 1783973080.35 total 20.00 last 134.10) - /unlink 3676 (time 1783973090.36 total 30.01 last 120.36) total: 4500 /unlink in 36.11 seconds: 124.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 670 (time 1783973111.70 total 10.02 last 66.84) - open/close 1357 (time 1783973121.71 total 20.03 last 68.68) - open/close 2199 (time 1783973131.71 total 30.03 last 84.16) - open/close 2975 (time 1783973141.71 total 40.03 last 77.60) - open/close 3739 (time 1783973151.72 total 50.04 last 76.35) total: 4500 open/close in 59.20 seconds: 76.01 ops/second - /unlink 1290 (time 1783973175.72 total 10.02 last 128.70) - /unlink 2528 (time 1783973185.72 total 20.03 last 123.77) - /unlink 3881 (time 1783973195.73 total 30.04 last 135.14) total: 4500 /unlink in 34.87 seconds: 129.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 669 (time 1783973216.18 total 10.03 last 66.73) - open/close 1363 (time 1783973226.19 total 20.04 last 69.31) - open/close 2202 (time 1783973236.20 total 30.05 last 83.83) - open/close 3035 (time 1783973246.20 total 40.05 last 83.27) - open/close 3713 (time 1783973256.20 total 50.05 last 67.78) - open/close 4441 (time 1783973266.21 total 60.06 last 72.75) total: 4500 open/close in 61.10 seconds: 73.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1147 (time 1783973281.97 total 10.00 last 114.66) - /unlink 2420 (time 1783973291.98 total 20.01 last 127.24) - /unlink 3688 (time 1783973301.98 total 30.01 last 126.76) total: 4500 /unlink in 37.62 seconds: 119.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 655 (time 1783973326.53 total 10.03 last 65.30) - open/close 1393 (time 1783973336.54 total 20.04 last 73.73) - open/close 2124 (time 1783973346.56 total 30.06 last 72.97) - open/close 2712 (time 1783973356.57 total 40.07 last 58.75) - open/close 3294 (time 1783973366.61 total 50.11 last 57.93) - open/close 3917 (time 1783973376.62 total 60.11 last 62.29) total: 4500 open/close in 69.31 seconds: 64.92 ops/second - /unlink 1084 (time 1783973402.24 total 10.01 last 108.29) - /unlink 2281 (time 1783973412.24 total 20.01 last 119.68) - /unlink 3538 (time 1783973422.24 total 30.02 last 125.66) total: 4500 /unlink in 37.74 seconds: 119.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 676 (time 1783973445.43 total 10.01 last 67.54) - open/close 1414 (time 1783973455.44 total 20.02 last 73.69) - open/close 2104 (time 1783973465.46 total 30.04 last 68.90) - open/close 2833 (time 1783973475.46 total 40.04 last 72.88) - open/close 3538 (time 1783973485.47 total 50.05 last 70.41) - open/close 4150 (time 1783973495.48 total 60.06 last 61.17) total: 4500 open/close in 64.43 seconds: 69.85 ops/second - /unlink 1214 (time 1783973514.19 total 10.01 last 121.29) - /unlink 2519 (time 1783973524.19 total 20.01 last 130.42) - /unlink 3884 (time 1783973534.19 total 30.02 last 136.46) total: 4500 /unlink in 34.71 seconds: 129.65 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95427 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: modules unloaded.