== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:42:14 (1789677734) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg132-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server oleg132-server: oleg132-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: kmemleak disabled oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 677 (time 1789677947.81 total 10.01 last 67.65) - open/close 1325 (time 1789677957.82 total 20.02 last 64.74) - open/close 1934 (time 1789677967.83 total 30.02 last 60.89) - open/close 2633 (time 1789677977.84 total 40.03 last 69.81) - open/close 3422 (time 1789677987.84 total 50.03 last 78.89) - open/close 4138 (time 1789677997.85 total 60.04 last 71.52) total: 4500 open/close in 65.94 seconds: 68.25 ops/second - /unlink 1235 (time 1789678018.18 total 10.00 last 123.46) - /unlink 2526 (time 1789678028.19 total 20.02 last 128.89) - /unlink 3795 (time 1789678038.19 total 30.02 last 126.90) total: 4500 /unlink in 34.10 seconds: 131.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 698 (time 1789678057.57 total 10.00 last 69.79) - open/close 1465 (time 1789678067.58 total 20.01 last 76.65) - open/close 2275 (time 1789678077.61 total 30.04 last 80.76) - open/close 2989 (time 1789678087.61 total 40.04 last 71.39) - open/close 3799 (time 1789678097.61 total 50.04 last 80.99) total: 4500 open/close in 59.28 seconds: 75.91 ops/second - /unlink 1383 (time 1789678121.18 total 10.00 last 138.25) - /unlink 2774 (time 1789678131.18 total 20.01 last 139.03) - /unlink 4093 (time 1789678141.19 total 30.01 last 131.89) total: 4500 /unlink in 32.93 seconds: 136.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 733 (time 1789678158.51 total 10.01 last 73.23) - open/close 1399 (time 1789678168.51 total 20.01 last 66.58) - open/close 2155 (time 1789678178.53 total 30.03 last 75.46) - open/close 2808 (time 1789678188.55 total 40.05 last 65.17) - open/close 3483 (time 1789678198.56 total 50.06 last 67.45) - open/close 4138 (time 1789678208.57 total 60.06 last 65.47) total: 4500 open/close in 66.30 seconds: 67.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 867 (time 1789678229.76 total 10.00 last 86.70) - /unlink 2052 (time 1789678239.77 total 20.01 last 118.44) - /unlink 3308 (time 1789678249.78 total 30.01 last 125.50) - /unlink 4153 (time 1789678259.78 total 40.02 last 84.47) total: 4500 /unlink in 43.52 seconds: 103.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 731 (time 1789678279.37 total 10.02 last 72.98) - open/close 1455 (time 1789678289.37 total 20.02 last 72.39) - open/close 2265 (time 1789678299.37 total 30.03 last 80.93) - open/close 2996 (time 1789678309.38 total 40.03 last 73.05) - open/close 3652 (time 1789678319.39 total 50.04 last 65.54) - open/close 4372 (time 1789678329.40 total 60.05 last 71.91) total: 4500 open/close in 61.67 seconds: 72.97 ops/second - /unlink 1473 (time 1789678346.19 total 10.00 last 147.27) - /unlink 2865 (time 1789678356.19 total 20.01 last 139.10) - /unlink 4130 (time 1789678366.19 total 30.01 last 126.50) total: 4500 /unlink in 33.69 seconds: 133.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 710 (time 1789678385.35 total 10.03 last 70.81) - open/close 1416 (time 1789678395.35 total 20.03 last 70.56) - open/close 2172 (time 1789678405.37 total 30.05 last 75.49) - open/close 3028 (time 1789678415.38 total 40.05 last 85.54) - open/close 3912 (time 1789678425.38 total 50.06 last 88.37) total: 4500 open/close in 57.75 seconds: 77.93 ops/second - /unlink 1405 (time 1789678447.12 total 10.00 last 140.49) - /unlink 2779 (time 1789678457.13 total 20.01 last 137.33) total: 4500 /unlink in 29.86 seconds: 150.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 892 (time 1789678482.29 total 10.01 last 89.12) - open/close 1567 (time 1789678492.29 total 20.01 last 67.46) - open/close 2183 (time 1789678502.30 total 30.02 last 61.55) - open/close 2669 (time 1789678512.32 total 40.04 last 48.51) - open/close 3106 (time 1789678522.32 total 50.04 last 43.69) - open/close 3767 (time 1789678532.33 total 60.06 last 66.01) - open/close 4414 (time 1789678542.37 total 70.09 last 64.46) total: 4500 open/close in 71.37 seconds: 63.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1459 (time 1789678558.48 total 10.01 last 145.77) - /unlink 2921 (time 1789678568.48 total 20.01 last 146.19) total: 4500 /unlink in 29.19 seconds: 154.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 728 (time 1789678592.56 total 10.01 last 72.72) - open/close 1433 (time 1789678602.56 total 20.01 last 70.49) - open/close 2165 (time 1789678612.57 total 30.01 last 73.19) - open/close 2839 (time 1789678622.58 total 40.03 last 67.32) - open/close 3612 (time 1789678632.59 total 50.03 last 77.24) - open/close 4363 (time 1789678642.60 total 60.04 last 75.01) total: 4500 open/close in 62.02 seconds: 72.56 ops/second - /unlink 1447 (time 1789678659.04 total 10.01 last 144.60) - /unlink 2783 (time 1789678669.04 total 20.02 last 133.48) - /unlink 3835 (time 1789678679.05 total 30.02 last 105.18) total: 4500 /unlink in 36.65 seconds: 122.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 441 (time 1789678702.67 total 10.02 last 44.01) - open/close 1024 (time 1789678712.69 total 20.04 last 58.19) - open/close 1606 (time 1789678722.69 total 30.05 last 58.15) - open/close 2268 (time 1789678732.71 total 40.06 last 66.12) - open/close 2974 (time 1789678742.74 total 50.09 last 70.37) - open/close 3758 (time 1789678752.74 total 60.10 last 78.37) total: 4500 open/close in 70.09 seconds: 64.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1227 (time 1789678777.86 total 10.01 last 122.63) - /unlink 2541 (time 1789678787.87 total 20.01 last 131.32) - /unlink 3911 (time 1789678797.87 total 30.02 last 136.95) total: 4500 /unlink in 33.76 seconds: 133.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 687 (time 1789678817.83 total 10.01 last 68.62) - open/close 1245 (time 1789678827.84 total 20.02 last 55.75) - open/close 1982 (time 1789678837.84 total 30.02 last 73.68) - open/close 2585 (time 1789678847.86 total 40.04 last 60.21) - open/close 3268 (time 1789678857.86 total 50.04 last 68.29) - open/close 3854 (time 1789678867.87 total 60.04 last 58.57) total: 4500 open/close in 68.50 seconds: 65.70 ops/second - /unlink 1224 (time 1789678890.83 total 10.01 last 122.27) - /unlink 2491 (time 1789678900.83 total 20.01 last 126.69) - /unlink 3830 (time 1789678910.83 total 30.01 last 133.88) total: 4500 /unlink in 34.91 seconds: 128.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 598 (time 1789678931.21 total 10.02 last 59.70) - open/close 1166 (time 1789678941.22 total 20.03 last 56.72) - open/close 1782 (time 1789678951.23 total 30.04 last 61.56) - open/close 2523 (time 1789678961.23 total 40.04 last 74.07) - open/close 3381 (time 1789678971.24 total 50.05 last 85.76) - open/close 4150 (time 1789678981.26 total 60.07 last 76.76) total: 4500 open/close in 65.54 seconds: 68.66 ops/second - /unlink 1322 (time 1789679002.60 total 10.00 last 132.17) - /unlink 2869 (time 1789679012.61 total 20.01 last 154.63) - /unlink 4150 (time 1789679022.61 total 30.01 last 128.04) total: 4500 /unlink in 32.14 seconds: 140.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 591 (time 1789679040.85 total 10.01 last 59.05) - open/close 1260 (time 1789679050.87 total 20.02 last 66.79) - open/close 1988 (time 1789679060.88 total 30.04 last 72.72) - open/close 2721 (time 1789679070.88 total 40.04 last 73.29) - open/close 3494 (time 1789679080.88 total 50.04 last 77.26) - open/close 4084 (time 1789679090.90 total 60.06 last 58.88) total: 4500 open/close in 67.70 seconds: 66.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1200 (time 1789679113.23 total 10.00 last 119.99) - /unlink 2445 (time 1789679123.25 total 20.02 last 124.24) - /unlink 3909 (time 1789679133.25 total 30.02 last 146.37) total: 4500 /unlink in 33.77 seconds: 133.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 747 (time 1789679150.89 total 10.00 last 74.67) - open/close 1487 (time 1789679160.90 total 20.01 last 73.93) - open/close 2232 (time 1789679170.91 total 30.03 last 74.41) - open/close 2966 (time 1789679180.92 total 40.03 last 73.33) - open/close 3731 (time 1789679190.92 total 50.03 last 76.50) - open/close 4447 (time 1789679200.93 total 60.04 last 71.54) total: 4500 open/close in 60.67 seconds: 74.18 ops/second - /unlink 1307 (time 1789679216.47 total 10.01 last 130.59) - /unlink 2180 (time 1789679226.49 total 20.03 last 87.15) - /unlink 3008 (time 1789679236.49 total 30.03 last 82.79) - /unlink 4019 (time 1789679246.50 total 40.03 last 101.04) total: 4500 /unlink in 44.15 seconds: 101.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 560 (time 1789679265.91 total 10.02 last 55.90) - open/close 1311 (time 1789679275.92 total 20.03 last 75.01) - open/close 1828 (time 1789679285.93 total 30.04 last 51.63) - open/close 2320 (time 1789679295.93 total 40.05 last 49.18) - open/close 3012 (time 1789679305.94 total 50.05 last 69.15) - open/close 3619 (time 1789679315.95 total 60.06 last 60.64) - open/close 4269 (time 1789679325.96 total 70.07 last 64.96) total: 4500 open/close in 75.06 seconds: 59.95 ops/second - /unlink 1266 (time 1789679347.52 total 10.01 last 126.47) - /unlink 2503 (time 1789679357.53 total 20.02 last 123.60) - /unlink 3607 (time 1789679367.53 total 30.02 last 110.40) total: 4500 /unlink in 37.85 seconds: 118.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 718 (time 1789679392.20 total 10.00 last 71.79) - open/close 1471 (time 1789679402.21 total 20.01 last 75.23) - open/close 2130 (time 1789679412.22 total 30.02 last 65.83) - open/close 2735 (time 1789679422.22 total 40.02 last 60.49) - open/close 3349 (time 1789679432.23 total 50.03 last 61.37) - open/close 4001 (time 1789679442.23 total 60.03 last 65.16) total: 4500 open/close in 67.80 seconds: 66.37 ops/second - /unlink 1129 (time 1789679464.79 total 10.01 last 112.83) - /unlink 2340 (time 1789679474.79 total 20.01 last 121.04) - /unlink 3323 (time 1789679484.79 total 30.01 last 98.29) total: 4500 /unlink in 39.61 seconds: 113.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 704 (time 1789679509.01 total 10.02 last 70.27) - open/close 1399 (time 1789679519.02 total 20.03 last 69.41) - open/close 2177 (time 1789679529.02 total 30.03 last 77.77) - open/close 2992 (time 1789679539.03 total 40.04 last 81.48) - open/close 3772 (time 1789679549.03 total 50.04 last 77.94) - open/close 4413 (time 1789679559.04 total 60.05 last 64.08) total: 4500 open/close in 60.97 seconds: 73.81 ops/second - /unlink 1273 (time 1789679575.86 total 10.00 last 127.24) - /unlink 2558 (time 1789679585.86 total 20.01 last 128.48) - /unlink 3675 (time 1789679595.86 total 30.01 last 111.69) total: 4500 /unlink in 39.39 seconds: 114.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 418 (time 1789679621.58 total 10.03 last 41.69) - open/close 1063 (time 1789679631.59 total 20.03 last 64.44) - open/close 1844 (time 1789679641.59 total 30.04 last 78.07) - open/close 2571 (time 1789679651.60 total 40.05 last 72.63) - open/close 3298 (time 1789679661.61 total 50.06 last 72.64) - open/close 4061 (time 1789679671.62 total 60.06 last 76.27) total: 4500 open/close in 66.26 seconds: 67.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1491 (time 1789679692.33 total 10.00 last 149.07) - /unlink 2966 (time 1789679702.33 total 20.01 last 147.43) - /unlink 4306 (time 1789679712.34 total 30.01 last 133.97) total: 4500 /unlink in 31.54 seconds: 142.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 435 (time 1789679729.87 total 10.03 last 43.38) - open/close 956 (time 1789679739.88 total 20.04 last 52.03) - open/close 1725 (time 1789679749.88 total 30.04 last 76.90) - open/close 2461 (time 1789679759.90 total 40.06 last 73.50) - open/close 3136 (time 1789679769.90 total 50.06 last 67.46) - open/close 3829 (time 1789679779.91 total 60.06 last 69.29) - open/close 4447 (time 1789679789.94 total 70.09 last 61.62) total: 4500 open/close in 71.08 seconds: 63.31 ops/second - /unlink 1607 (time 1789679806.09 total 10.00 last 160.69) - /unlink 2936 (time 1789679816.09 total 20.00 last 132.86) - /unlink 4320 (time 1789679826.10 total 30.01 last 138.34) total: 4500 /unlink in 31.43 seconds: 143.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 701 (time 1789679843.05 total 10.00 last 70.09) - open/close 1330 (time 1789679853.07 total 20.02 last 62.79) - open/close 1957 (time 1789679863.08 total 30.03 last 62.65) - open/close 2668 (time 1789679873.08 total 40.03 last 71.10) - open/close 3410 (time 1789679883.09 total 50.04 last 74.07) - open/close 4040 (time 1789679893.10 total 60.06 last 62.93) total: 4500 open/close in 67.85 seconds: 66.32 ops/second - /unlink 1271 (time 1789679915.40 total 10.01 last 126.98) - /unlink 2711 (time 1789679925.40 total 20.01 last 143.96) - /unlink 4087 (time 1789679935.40 total 30.02 last 137.54) total: 4500 /unlink in 33.40 seconds: 134.74 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95332 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: kmemleak disabled modules unloaded.