== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:09:56 (1788448196) Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg450-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg450-server oleg450-server: oleg450-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg450-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg450-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 759 (time 1788448399.72 total 10.01 last 75.79) - open/close 1514 (time 1788448409.73 total 20.02 last 75.49) - open/close 2211 (time 1788448419.73 total 30.02 last 69.66) - open/close 2905 (time 1788448429.73 total 40.02 last 69.39) - open/close 3700 (time 1788448439.74 total 50.03 last 79.48) total: 4500 open/close in 60.01 seconds: 74.99 ops/second - /unlink 1484 (time 1788448464.07 total 10.00 last 148.37) - /unlink 2979 (time 1788448474.08 total 20.00 last 149.47) total: 4500 /unlink in 29.77 seconds: 151.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 771 (time 1788448497.96 total 10.01 last 77.04) - open/close 1473 (time 1788448507.96 total 20.01 last 70.17) - open/close 2171 (time 1788448517.98 total 30.02 last 69.71) - open/close 2814 (time 1788448527.99 total 40.03 last 64.24) - open/close 3447 (time 1788448538.02 total 50.07 last 63.10) - open/close 4121 (time 1788448548.04 total 60.08 last 67.29) total: 4500 open/close in 66.00 seconds: 68.18 ops/second - /unlink 1276 (time 1788448568.85 total 10.01 last 127.44) - /unlink 2421 (time 1788448578.86 total 20.02 last 114.41) - /unlink 3675 (time 1788448588.86 total 30.02 last 125.39) total: 4500 /unlink in 36.75 seconds: 122.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 740 (time 1788448611.02 total 10.01 last 73.94) - open/close 1427 (time 1788448621.03 total 20.02 last 68.64) - open/close 2254 (time 1788448631.03 total 30.02 last 82.64) - open/close 2991 (time 1788448641.04 total 40.03 last 73.63) - open/close 3800 (time 1788448651.06 total 50.05 last 80.79) total: 4500 open/close in 59.08 seconds: 76.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1470 (time 1788448673.98 total 10.00 last 146.98) - /unlink 2880 (time 1788448683.98 total 20.00 last 140.95) - /unlink 4396 (time 1788448693.98 total 30.01 last 151.57) total: 4500 /unlink in 30.82 seconds: 145.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 714 (time 1788448709.52 total 10.03 last 71.15) - open/close 1485 (time 1788448719.53 total 20.05 last 77.02) - open/close 2335 (time 1788448729.53 total 30.05 last 84.98) - open/close 3138 (time 1788448739.53 total 40.05 last 80.27) - open/close 3988 (time 1788448749.54 total 50.05 last 84.99) total: 4500 open/close in 56.12 seconds: 80.18 ops/second - /unlink 1454 (time 1788448769.88 total 10.01 last 145.32) - /unlink 2928 (time 1788448779.89 total 20.01 last 147.32) - /unlink 4285 (time 1788448789.89 total 30.01 last 135.67) total: 4500 /unlink in 31.63 seconds: 142.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 763 (time 1788448805.72 total 10.03 last 76.09) - open/close 1489 (time 1788448815.73 total 20.04 last 72.52) - open/close 2333 (time 1788448825.74 total 30.05 last 84.30) - open/close 3173 (time 1788448835.75 total 40.05 last 83.98) - open/close 3989 (time 1788448845.76 total 50.07 last 81.45) total: 4500 open/close in 56.14 seconds: 80.15 ops/second - /unlink 1373 (time 1788448865.94 total 10.01 last 137.20) - /unlink 2594 (time 1788448875.94 total 20.01 last 122.07) - /unlink 3970 (time 1788448885.95 total 30.02 last 137.47) total: 4500 /unlink in 33.90 seconds: 132.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 678 (time 1788448904.02 total 10.04 last 67.54) - open/close 1400 (time 1788448914.03 total 20.05 last 72.12) - open/close 2021 (time 1788448924.04 total 30.06 last 62.06) - open/close 2791 (time 1788448934.04 total 40.06 last 77.00) - open/close 3629 (time 1788448944.04 total 50.06 last 83.80) - open/close 4404 (time 1788448954.04 total 60.06 last 77.49) total: 4500 open/close in 61.29 seconds: 73.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1484 (time 1788448970.03 total 10.01 last 148.32) - /unlink 2959 (time 1788448980.03 total 20.01 last 147.47) - /unlink 4375 (time 1788448990.05 total 30.02 last 141.39) total: 4500 /unlink in 30.84 seconds: 145.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 803 (time 1788449005.39 total 10.00 last 80.27) - open/close 1497 (time 1788449015.40 total 20.01 last 69.33) - open/close 2254 (time 1788449025.40 total 30.02 last 75.69) - open/close 3042 (time 1788449035.41 total 40.02 last 78.76) - open/close 3693 (time 1788449045.42 total 50.03 last 65.05) - open/close 4420 (time 1788449055.42 total 60.04 last 72.66) total: 4500 open/close in 60.97 seconds: 73.81 ops/second - /unlink 1418 (time 1788449070.84 total 10.00 last 141.79) - /unlink 2789 (time 1788449080.84 total 20.00 last 137.09) - /unlink 4123 (time 1788449090.85 total 30.01 last 133.34) total: 4500 /unlink in 33.11 seconds: 135.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 695 (time 1788449109.05 total 10.01 last 69.46) - open/close 1519 (time 1788449119.07 total 20.02 last 82.27) - open/close 2275 (time 1788449129.08 total 30.03 last 75.55) - open/close 3068 (time 1788449139.08 total 40.03 last 79.29) - open/close 3846 (time 1788449149.08 total 50.04 last 77.75) total: 4500 open/close in 58.18 seconds: 77.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1370 (time 1788449171.78 total 10.00 last 136.99) - /unlink 2662 (time 1788449181.78 total 20.00 last 129.20) - /unlink 4030 (time 1788449191.80 total 30.02 last 136.51) total: 4500 /unlink in 33.78 seconds: 133.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 676 (time 1788449210.27 total 10.01 last 67.55) - open/close 1426 (time 1788449220.27 total 20.01 last 74.98) - open/close 2267 (time 1788449230.28 total 30.02 last 84.02) - open/close 3086 (time 1788449240.28 total 40.02 last 81.90) - open/close 3869 (time 1788449250.29 total 50.04 last 78.18) total: 4500 open/close in 57.69 seconds: 78.00 ops/second - /unlink 1416 (time 1788449271.87 total 10.00 last 141.56) - /unlink 2556 (time 1788449281.87 total 20.00 last 113.98) - /unlink 3824 (time 1788449291.87 total 30.01 last 126.74) total: 4500 /unlink in 34.66 seconds: 129.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 758 (time 1788449312.06 total 10.00 last 75.79) - open/close 1516 (time 1788449322.07 total 20.00 last 75.78) - open/close 2272 (time 1788449332.08 total 30.01 last 75.52) - open/close 3022 (time 1788449342.09 total 40.03 last 74.91) - open/close 3786 (time 1788449352.09 total 50.03 last 76.37) - open/close 4499 (time 1788449362.11 total 60.05 last 71.17) total: 4500 open/close in 60.05 seconds: 74.94 ops/second - /unlink 1195 (time 1788449376.88 total 10.01 last 119.42) - /unlink 2572 (time 1788449386.88 total 20.01 last 137.66) - /unlink 3788 (time 1788449396.89 total 30.01 last 121.56) total: 4500 /unlink in 34.93 seconds: 128.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 773 (time 1788449415.82 total 10.01 last 77.23) - open/close 1557 (time 1788449425.83 total 20.02 last 78.34) - open/close 2320 (time 1788449435.83 total 30.02 last 76.28) - open/close 3086 (time 1788449445.84 total 40.03 last 76.52) - open/close 3859 (time 1788449455.84 total 50.03 last 77.28) total: 4500 open/close in 58.47 seconds: 76.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1344 (time 1788449478.36 total 10.00 last 134.36) - /unlink 2800 (time 1788449488.36 total 20.00 last 145.59) - /unlink 4112 (time 1788449498.37 total 30.01 last 131.12) total: 4500 /unlink in 33.70 seconds: 133.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 667 (time 1788449517.51 total 10.01 last 66.64) - open/close 1405 (time 1788449527.51 total 20.01 last 73.77) - open/close 2199 (time 1788449537.52 total 30.02 last 79.35) - open/close 2992 (time 1788449547.52 total 40.02 last 79.27) - open/close 3712 (time 1788449557.53 total 50.04 last 71.92) - open/close 4409 (time 1788449567.53 total 60.04 last 69.70) total: 4500 open/close in 61.44 seconds: 73.24 ops/second - /unlink 1407 (time 1788449583.48 total 10.01 last 140.60) - /unlink 2645 (time 1788449593.49 total 20.01 last 123.71) - /unlink 4186 (time 1788449603.49 total 30.02 last 154.03) total: 4500 /unlink in 32.13 seconds: 140.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 837 (time 1788449620.62 total 10.01 last 83.65) - open/close 1583 (time 1788449630.63 total 20.01 last 74.57) - open/close 2332 (time 1788449640.64 total 30.03 last 74.76) - open/close 3008 (time 1788449650.66 total 40.04 last 67.51) - open/close 3690 (time 1788449660.66 total 50.05 last 68.16) - open/close 4247 (time 1788449670.68 total 60.07 last 55.59) total: 4500 open/close in 65.14 seconds: 69.08 ops/second - /unlink 1146 (time 1788449692.98 total 10.00 last 114.60) - /unlink 2448 (time 1788449702.99 total 20.01 last 130.11) - /unlink 3745 (time 1788449712.99 total 30.01 last 129.69) total: 4500 /unlink in 35.90 seconds: 125.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 777 (time 1788449733.26 total 10.00 last 77.70) - open/close 1534 (time 1788449743.27 total 20.02 last 75.58) - open/close 2337 (time 1788449753.29 total 30.03 last 80.20) - open/close 3103 (time 1788449763.30 total 40.05 last 76.47) - open/close 3883 (time 1788449773.31 total 50.05 last 77.97) total: 4500 open/close in 58.48 seconds: 76.95 ops/second - /unlink 1292 (time 1788449795.25 total 10.00 last 129.19) - /unlink 2571 (time 1788449805.26 total 20.00 last 127.89) - /unlink 3830 (time 1788449815.26 total 30.00 last 125.89) total: 4500 /unlink in 34.88 seconds: 129.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 795 (time 1788449834.50 total 10.01 last 79.42) - open/close 1546 (time 1788449844.52 total 20.03 last 74.97) - open/close 2328 (time 1788449854.53 total 30.04 last 78.11) - open/close 3114 (time 1788449864.53 total 40.04 last 78.56) - open/close 3902 (time 1788449874.54 total 50.05 last 78.75) total: 4500 open/close in 57.71 seconds: 77.98 ops/second - /unlink 1383 (time 1788449897.17 total 10.00 last 138.26) - /unlink 2808 (time 1788449907.18 total 20.01 last 142.45) - /unlink 4227 (time 1788449917.18 total 30.01 last 141.81) total: 4500 /unlink in 32.21 seconds: 139.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 825 (time 1788449932.81 total 10.00 last 82.47) - open/close 1602 (time 1788449942.83 total 20.02 last 77.59) - open/close 2385 (time 1788449952.85 total 30.04 last 78.14) - open/close 3146 (time 1788449962.87 total 40.06 last 75.91) - open/close 3944 (time 1788449972.88 total 50.07 last 79.76) total: 4500 open/close in 57.10 seconds: 78.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1454 (time 1788449993.71 total 10.01 last 145.32) - /unlink 2830 (time 1788450003.71 total 20.01 last 137.59) - /unlink 4149 (time 1788450013.72 total 30.01 last 131.87) total: 4500 /unlink in 32.98 seconds: 136.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 763 (time 1788450032.44 total 10.00 last 76.27) - open/close 1518 (time 1788450042.45 total 20.01 last 75.46) - open/close 2323 (time 1788450052.46 total 30.02 last 80.44) - open/close 3110 (time 1788450062.46 total 40.02 last 78.70) - open/close 3915 (time 1788450072.47 total 50.03 last 80.38) total: 4500 open/close in 57.80 seconds: 77.86 ops/second - /unlink 1394 (time 1788450094.04 total 10.00 last 139.39) - /unlink 2742 (time 1788450104.05 total 20.00 last 134.77) - /unlink 4030 (time 1788450114.06 total 30.01 last 128.67) total: 4500 /unlink in 33.71 seconds: 133.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 750 (time 1788450132.94 total 10.01 last 74.95) - open/close 1483 (time 1788450142.95 total 20.01 last 73.27) - open/close 2208 (time 1788450152.95 total 30.01 last 72.49) - open/close 2882 (time 1788450162.95 total 40.01 last 67.39) - open/close 3698 (time 1788450172.96 total 50.02 last 81.53) - open/close 4387 (time 1788450182.97 total 60.04 last 68.81) total: 4500 open/close in 61.69 seconds: 72.95 ops/second - /unlink 1384 (time 1788450199.88 total 10.00 last 138.36) - /unlink 2679 (time 1788450209.88 total 20.01 last 129.44) - /unlink 4133 (time 1788450219.89 total 30.02 last 145.24) total: 4500 /unlink in 32.76 seconds: 137.38 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95220 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server unloading modules via unload_modules_local on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local oleg450-server: modules unloaded.