== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:25:09 (1788445509) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg160-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server oleg160-server: oleg160-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: MODOPTS_LIBCFS= oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg160-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg160-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 599 (time 1788445725.56 total 10.01 last 59.82) - open/close 1129 (time 1788445735.56 total 20.02 last 52.99) - open/close 1753 (time 1788445745.56 total 30.02 last 62.39) - open/close 2515 (time 1788445755.57 total 40.02 last 76.14) - open/close 3183 (time 1788445765.58 total 50.03 last 66.75) - open/close 3820 (time 1788445775.59 total 60.04 last 63.65) - open/close 4480 (time 1788445785.59 total 70.05 last 65.95) total: 4500 open/close in 70.34 seconds: 63.98 ops/second - /unlink 1319 (time 1788445801.54 total 10.00 last 131.89) - /unlink 2593 (time 1788445811.54 total 20.00 last 127.35) - /unlink 3802 (time 1788445821.54 total 30.01 last 120.89) total: 4500 /unlink in 36.34 seconds: 123.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 663 (time 1788445842.75 total 10.01 last 66.24) - open/close 1346 (time 1788445852.76 total 20.02 last 68.25) - open/close 2029 (time 1788445862.76 total 30.02 last 68.25) - open/close 2735 (time 1788445872.83 total 40.09 last 70.13) - open/close 3424 (time 1788445882.85 total 50.12 last 68.73) - open/close 4127 (time 1788445892.87 total 60.13 last 70.22) total: 4500 open/close in 65.76 seconds: 68.43 ops/second - /unlink 1222 (time 1788445913.03 total 10.00 last 122.18) - /unlink 2503 (time 1788445923.03 total 20.00 last 128.08) - /unlink 3743 (time 1788445933.04 total 30.01 last 123.95) total: 4500 /unlink in 35.66 seconds: 126.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 667 (time 1788445953.93 total 10.00 last 66.68) - open/close 1467 (time 1788445963.95 total 20.01 last 79.90) - open/close 2161 (time 1788445973.97 total 30.03 last 69.27) - open/close 2760 (time 1788445983.97 total 40.04 last 59.86) - open/close 3309 (time 1788445993.98 total 50.05 last 54.86) - open/close 3912 (time 1788446003.98 total 60.05 last 60.28) - open/close 4390 (time 1788446013.99 total 70.05 last 47.79) total: 4500 open/close in 71.39 seconds: 63.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1027 (time 1788446031.33 total 10.02 last 102.51) - /unlink 2327 (time 1788446041.34 total 20.03 last 129.91) - /unlink 3592 (time 1788446051.34 total 30.03 last 126.44) total: 4500 /unlink in 38.42 seconds: 117.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 597 (time 1788446077.00 total 10.01 last 59.63) - open/close 1263 (time 1788446087.00 total 20.01 last 66.59) - open/close 1894 (time 1788446097.01 total 30.02 last 63.05) - open/close 2637 (time 1788446107.02 total 40.03 last 74.25) - open/close 3331 (time 1788446117.02 total 50.03 last 69.36) - open/close 4098 (time 1788446127.04 total 60.06 last 76.53) total: 4500 open/close in 65.58 seconds: 68.62 ops/second - /unlink 1299 (time 1788446147.21 total 10.00 last 129.88) - /unlink 2678 (time 1788446157.21 total 20.00 last 137.87) - /unlink 3996 (time 1788446167.21 total 30.01 last 131.74) total: 4500 /unlink in 33.52 seconds: 134.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 724 (time 1788446185.74 total 10.01 last 72.33) - open/close 1490 (time 1788446195.75 total 20.01 last 76.57) - open/close 2152 (time 1788446205.76 total 30.02 last 66.14) - open/close 2863 (time 1788446215.76 total 40.03 last 71.05) - open/close 3592 (time 1788446225.77 total 50.04 last 72.82) - open/close 4432 (time 1788446235.78 total 60.04 last 83.96) total: 4500 open/close in 60.96 seconds: 73.82 ops/second - /unlink 1363 (time 1788446250.75 total 10.01 last 136.23) - /unlink 2663 (time 1788446260.75 total 20.01 last 129.99) - /unlink 3989 (time 1788446270.76 total 30.01 last 132.51) total: 4500 /unlink in 33.48 seconds: 134.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 821 (time 1788446289.44 total 10.00 last 82.10) - open/close 1589 (time 1788446299.44 total 20.00 last 76.77) - open/close 2341 (time 1788446309.44 total 30.00 last 75.20) - open/close 3118 (time 1788446319.46 total 40.02 last 77.56) - open/close 3844 (time 1788446329.47 total 50.03 last 72.55) total: 4500 open/close in 59.68 seconds: 75.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1186 (time 1788446354.35 total 10.00 last 118.54) - /unlink 2341 (time 1788446364.37 total 20.02 last 115.36) - /unlink 3765 (time 1788446374.37 total 30.02 last 142.32) total: 4500 /unlink in 35.28 seconds: 127.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 738 (time 1788446394.33 total 10.01 last 73.73) - open/close 1481 (time 1788446404.33 total 20.02 last 74.26) - open/close 2170 (time 1788446414.35 total 30.03 last 68.82) - open/close 2947 (time 1788446424.35 total 40.03 last 77.67) - open/close 3680 (time 1788446434.36 total 50.04 last 73.25) - open/close 4412 (time 1788446444.36 total 60.04 last 73.18) total: 4500 open/close in 61.34 seconds: 73.37 ops/second - /unlink 1333 (time 1788446460.43 total 10.01 last 133.22) - /unlink 2769 (time 1788446470.44 total 20.01 last 143.51) - /unlink 4315 (time 1788446480.44 total 30.02 last 154.55) total: 4500 /unlink in 31.54 seconds: 142.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 686 (time 1788446496.59 total 10.00 last 68.59) - open/close 1431 (time 1788446506.60 total 20.01 last 74.45) - open/close 2106 (time 1788446516.62 total 30.04 last 67.32) - open/close 2802 (time 1788446526.63 total 40.04 last 69.59) - open/close 3553 (time 1788446536.64 total 50.06 last 74.96) - open/close 4287 (time 1788446546.66 total 60.07 last 73.29) total: 4500 open/close in 62.91 seconds: 71.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1414 (time 1788446563.56 total 10.00 last 141.34) - /unlink 2724 (time 1788446573.56 total 20.00 last 131.00) - /unlink 3989 (time 1788446583.56 total 30.01 last 126.49) total: 4500 /unlink in 34.37 seconds: 130.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 746 (time 1788446603.00 total 10.00 last 74.59) - open/close 1436 (time 1788446613.00 total 20.00 last 68.98) - open/close 2268 (time 1788446623.01 total 30.01 last 83.12) - open/close 2979 (time 1788446633.03 total 40.04 last 70.93) - open/close 3616 (time 1788446643.04 total 50.04 last 63.66) - open/close 4397 (time 1788446653.05 total 60.06 last 77.98) total: 4500 open/close in 61.59 seconds: 73.06 ops/second - /unlink 1327 (time 1788446669.45 total 10.00 last 132.64) - /unlink 2651 (time 1788446679.46 total 20.01 last 132.38) - /unlink 4187 (time 1788446689.46 total 30.01 last 153.57) total: 4500 /unlink in 33.00 seconds: 136.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 720 (time 1788446708.30 total 10.00 last 71.99) - open/close 1518 (time 1788446718.30 total 20.00 last 79.78) - open/close 2329 (time 1788446728.30 total 30.01 last 81.09) - open/close 3097 (time 1788446738.32 total 40.02 last 76.69) - open/close 3838 (time 1788446748.33 total 50.03 last 74.03) total: 4500 open/close in 59.43 seconds: 75.71 ops/second - /unlink 1347 (time 1788446772.63 total 10.01 last 134.56) - /unlink 2817 (time 1788446782.63 total 20.01 last 146.99) - /unlink 4295 (time 1788446792.64 total 30.02 last 147.64) total: 4500 /unlink in 31.97 seconds: 140.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 799 (time 1788446809.85 total 10.00 last 79.88) - open/close 1578 (time 1788446819.85 total 20.00 last 77.88) - open/close 2451 (time 1788446829.87 total 30.03 last 87.10) - open/close 3263 (time 1788446839.88 total 40.03 last 81.17) - open/close 4026 (time 1788446849.89 total 50.04 last 76.21) total: 4500 open/close in 56.15 seconds: 80.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1408 (time 1788446870.20 total 10.00 last 140.76) - /unlink 2894 (time 1788446880.21 total 20.01 last 148.47) - /unlink 4410 (time 1788446890.21 total 30.02 last 151.54) total: 4500 /unlink in 30.78 seconds: 146.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 762 (time 1788446905.43 total 10.00 last 76.19) - open/close 1522 (time 1788446915.43 total 20.00 last 75.98) - open/close 2326 (time 1788446925.44 total 30.01 last 80.30) - open/close 2966 (time 1788446935.45 total 40.02 last 63.99) - open/close 3749 (time 1788446945.45 total 50.02 last 78.29) - open/close 4481 (time 1788446955.45 total 60.02 last 73.20) total: 4500 open/close in 60.22 seconds: 74.73 ops/second - /unlink 1420 (time 1788446970.21 total 10.00 last 141.98) - /unlink 2907 (time 1788446980.21 total 20.00 last 148.70) - /unlink 4417 (time 1788446990.21 total 30.01 last 150.91) total: 4500 /unlink in 30.76 seconds: 146.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 829 (time 1788447005.27 total 10.01 last 82.78) - open/close 1601 (time 1788447015.29 total 20.03 last 77.08) - open/close 2405 (time 1788447025.30 total 30.04 last 80.30) - open/close 3205 (time 1788447035.31 total 40.06 last 79.89) - open/close 3995 (time 1788447045.32 total 50.06 last 78.95) total: 4500 open/close in 56.59 seconds: 79.53 ops/second - /unlink 1486 (time 1788447066.61 total 10.00 last 148.57) - /unlink 2897 (time 1788447076.62 total 20.00 last 141.09) - /unlink 4229 (time 1788447086.62 total 30.00 last 133.20) total: 4500 /unlink in 32.32 seconds: 139.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 774 (time 1788447104.50 total 10.00 last 77.38) - open/close 1541 (time 1788447114.51 total 20.01 last 76.65) - open/close 2261 (time 1788447124.51 total 30.01 last 71.98) - open/close 3062 (time 1788447134.51 total 40.02 last 80.06) - open/close 3803 (time 1788447144.52 total 50.03 last 74.03) - open/close 4365 (time 1788447154.54 total 60.05 last 56.10) total: 4500 open/close in 62.06 seconds: 72.51 ops/second - /unlink 1211 (time 1788447172.62 total 10.00 last 121.05) - /unlink 2574 (time 1788447182.62 total 20.01 last 136.26) - /unlink 4107 (time 1788447192.63 total 30.01 last 153.22) total: 4500 /unlink in 33.39 seconds: 134.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 751 (time 1788447211.23 total 10.00 last 75.06) - open/close 1521 (time 1788447221.23 total 20.01 last 77.00) - open/close 2303 (time 1788447231.24 total 30.01 last 78.13) - open/close 3051 (time 1788447241.24 total 40.02 last 74.74) - open/close 3841 (time 1788447251.25 total 50.03 last 78.97) total: 4500 open/close in 59.38 seconds: 75.78 ops/second - /unlink 1280 (time 1788447275.87 total 10.00 last 127.98) - /unlink 2578 (time 1788447285.87 total 20.00 last 129.80) - /unlink 3863 (time 1788447295.87 total 30.00 last 128.47) total: 4500 /unlink in 35.24 seconds: 127.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 728 (time 1788447315.67 total 10.01 last 72.70) - open/close 1400 (time 1788447325.67 total 20.01 last 67.20) - open/close 2076 (time 1788447335.69 total 30.03 last 67.48) - open/close 2806 (time 1788447345.70 total 40.04 last 72.93) - open/close 3447 (time 1788447355.73 total 50.07 last 63.89) - open/close 4125 (time 1788447365.74 total 60.08 last 67.77) total: 4500 open/close in 64.70 seconds: 69.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1257 (time 1788447384.93 total 10.01 last 125.60) - /unlink 2651 (time 1788447394.93 total 20.01 last 139.38) - /unlink 4052 (time 1788447404.93 total 30.01 last 140.07) total: 4500 /unlink in 33.80 seconds: 133.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 833 (time 1788447423.19 total 10.00 last 83.27) - open/close 1614 (time 1788447433.19 total 20.01 last 78.09) - open/close 2324 (time 1788447443.19 total 30.01 last 70.98) - open/close 3069 (time 1788447453.20 total 40.01 last 74.49) - open/close 3828 (time 1788447463.21 total 50.03 last 75.77) total: 4500 open/close in 59.31 seconds: 75.87 ops/second - /unlink 1302 (time 1788447487.16 total 10.01 last 130.04) - /unlink 2595 (time 1788447497.17 total 20.02 last 129.16) - /unlink 3952 (time 1788447507.18 total 30.03 last 135.65) total: 4500 /unlink in 33.86 seconds: 132.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 710 (time 1788447526.01 total 10.00 last 70.99) - open/close 1430 (time 1788447536.01 total 20.01 last 71.97) - open/close 2305 (time 1788447546.02 total 30.01 last 87.44) - open/close 3041 (time 1788447556.03 total 40.02 last 73.52) - open/close 3819 (time 1788447566.04 total 50.03 last 77.75) total: 4500 open/close in 58.65 seconds: 76.73 ops/second - /unlink 1308 (time 1788447590.06 total 10.01 last 130.64) - /unlink 2403 (time 1788447600.06 total 20.02 last 109.42) - /unlink 3588 (time 1788447610.07 total 30.02 last 118.44) total: 4500 /unlink in 36.90 seconds: 121.94 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95343 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: modules unloaded.