== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:27:42 (1788445662) Stopping clients: oleg126-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg126-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg126-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg126-server oleg126-server: oleg126-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing load_modules_local oleg126-server: Loading modules from /home/green/git/lustre-release/lustre oleg126-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg126-server: detected 4 online CPUs by sysfs oleg126-server: MODOPTS_LIBCFS= oleg126-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg126-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg126-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg126-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg126-server: oleg126-server.virtnet: executing set_default_debug -1 all pdsh@oleg126-client: oleg126-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg126-client.virtnet: -o user_xattr,flock 192.168.201.126@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 906 (time 1788445871.39 total 10.00 last 90.59) - open/close 1823 (time 1788445881.39 total 20.01 last 91.66) - open/close 2741 (time 1788445891.39 total 30.01 last 91.79) - open/close 3612 (time 1788445901.40 total 40.01 last 87.08) - open/close 4424 (time 1788445911.40 total 50.02 last 81.14) total: 4500 open/close in 50.92 seconds: 88.37 ops/second - /unlink 1535 (time 1788445927.72 total 10.00 last 153.45) - /unlink 3287 (time 1788445937.72 total 20.01 last 175.13) total: 4500 /unlink in 27.93 seconds: 161.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 923 (time 1788445961.03 total 10.01 last 92.24) - open/close 1729 (time 1788445971.04 total 20.01 last 80.55) - open/close 2499 (time 1788445981.05 total 30.02 last 76.95) - open/close 3417 (time 1788445991.05 total 40.02 last 91.80) - open/close 4158 (time 1788446001.05 total 50.02 last 74.08) total: 4500 open/close in 53.63 seconds: 83.92 ops/second - /unlink 1561 (time 1788446020.23 total 10.00 last 156.10) - /unlink 2862 (time 1788446030.25 total 20.02 last 129.86) - /unlink 4306 (time 1788446040.25 total 30.02 last 144.38) total: 4500 /unlink in 31.51 seconds: 142.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 816 (time 1788446056.27 total 10.00 last 81.59) - open/close 1534 (time 1788446066.28 total 20.01 last 71.72) - open/close 2377 (time 1788446076.28 total 30.01 last 84.28) - open/close 3243 (time 1788446086.31 total 40.04 last 86.37) - open/close 4104 (time 1788446096.32 total 50.05 last 86.00) total: 4500 open/close in 54.90 seconds: 81.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1726 (time 1788446115.77 total 10.00 last 172.56) - /unlink 3428 (time 1788446125.77 total 20.00 last 170.17) total: 4500 /unlink in 27.59 seconds: 163.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 926 (time 1788446148.08 total 10.01 last 92.54) - open/close 1817 (time 1788446158.11 total 20.04 last 88.81) - open/close 2725 (time 1788446168.12 total 30.05 last 90.72) - open/close 3589 (time 1788446178.12 total 40.05 last 86.38) - open/close 4420 (time 1788446188.13 total 50.05 last 83.08) total: 4500 open/close in 51.27 seconds: 87.77 ops/second - /unlink 1260 (time 1788446204.79 total 10.00 last 125.95) - /unlink 2380 (time 1788446214.79 total 20.01 last 111.96) - /unlink 3945 (time 1788446224.79 total 30.01 last 156.47) total: 4500 /unlink in 33.97 seconds: 132.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 937 (time 1788446243.76 total 10.00 last 93.69) - open/close 1784 (time 1788446253.76 total 20.00 last 84.69) - open/close 2697 (time 1788446263.76 total 30.00 last 91.29) - open/close 3647 (time 1788446273.77 total 40.01 last 94.95) total: 4500 open/close in 48.47 seconds: 92.84 ops/second - /unlink 1669 (time 1788446295.89 total 10.00 last 166.83) - /unlink 3158 (time 1788446305.89 total 20.01 last 148.87) total: 4500 /unlink in 28.76 seconds: 156.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 882 (time 1788446329.81 total 10.00 last 88.18) - open/close 1714 (time 1788446339.82 total 20.01 last 83.12) - open/close 2582 (time 1788446349.83 total 30.02 last 86.70) - open/close 3345 (time 1788446359.85 total 40.04 last 76.16) - open/close 4172 (time 1788446369.88 total 50.07 last 82.43) total: 4500 open/close in 53.78 seconds: 83.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1782 (time 1788446388.18 total 10.00 last 178.20) - /unlink 3409 (time 1788446398.18 total 20.01 last 162.60) total: 4500 /unlink in 26.88 seconds: 167.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 875 (time 1788446419.67 total 10.00 last 87.46) - open/close 1751 (time 1788446429.67 total 20.01 last 87.57) - open/close 2622 (time 1788446439.68 total 30.02 last 87.03) - open/close 3627 (time 1788446449.69 total 40.02 last 100.46) total: 4500 open/close in 49.84 seconds: 90.30 ops/second - /unlink 1758 (time 1788446473.68 total 10.01 last 175.71) - /unlink 3372 (time 1788446483.69 total 20.01 last 161.33) total: 4500 /unlink in 27.03 seconds: 166.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 983 (time 1788446505.69 total 10.00 last 98.26) - open/close 1884 (time 1788446515.70 total 20.02 last 90.00) - open/close 2803 (time 1788446525.71 total 30.03 last 91.78) - open/close 3746 (time 1788446535.72 total 40.04 last 94.24) total: 4500 open/close in 47.38 seconds: 94.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1798 (time 1788446557.09 total 10.01 last 179.65) - /unlink 3509 (time 1788446567.09 total 20.01 last 171.02) total: 4500 /unlink in 26.80 seconds: 167.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 752 (time 1788446588.45 total 10.00 last 75.18) - open/close 1629 (time 1788446598.46 total 20.01 last 87.66) - open/close 2528 (time 1788446608.47 total 30.02 last 89.83) - open/close 3449 (time 1788446618.47 total 40.02 last 92.09) - open/close 4336 (time 1788446628.49 total 50.04 last 88.52) total: 4500 open/close in 51.69 seconds: 87.05 ops/second - /unlink 1695 (time 1788446645.06 total 10.00 last 169.45) - /unlink 3320 (time 1788446655.06 total 20.00 last 162.48) total: 4500 /unlink in 27.84 seconds: 161.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 811 (time 1788446679.28 total 10.00 last 81.09) - open/close 1755 (time 1788446689.31 total 20.03 last 94.13) - open/close 2473 (time 1788446699.32 total 30.04 last 71.74) - open/close 3353 (time 1788446709.32 total 40.04 last 87.98) - open/close 4387 (time 1788446719.33 total 50.04 last 103.36) total: 4500 open/close in 51.23 seconds: 87.85 ops/second - /unlink 1972 (time 1788446734.51 total 10.01 last 197.09) - /unlink 3774 (time 1788446744.52 total 20.01 last 180.11) total: 4500 /unlink in 24.30 seconds: 185.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 842 (time 1788446763.07 total 10.02 last 84.06) - open/close 1814 (time 1788446773.07 total 20.02 last 97.16) - open/close 2788 (time 1788446783.08 total 30.03 last 97.32) - open/close 3681 (time 1788446793.09 total 40.04 last 89.23) - open/close 4414 (time 1788446803.09 total 50.04 last 73.29) total: 4500 open/close in 50.93 seconds: 88.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1883 (time 1788446818.15 total 10.00 last 188.23) - /unlink 3475 (time 1788446828.16 total 20.01 last 159.13) total: 4500 /unlink in 26.16 seconds: 172.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 981 (time 1788446848.34 total 10.01 last 98.04) - open/close 1985 (time 1788446858.35 total 20.01 last 100.32) - open/close 2967 (time 1788446868.35 total 30.02 last 98.18) - open/close 3864 (time 1788446878.36 total 40.03 last 89.61) total: 4500 open/close in 47.52 seconds: 94.69 ops/second - /unlink 1714 (time 1788446899.70 total 10.01 last 171.24) - /unlink 3408 (time 1788446909.71 total 20.02 last 169.19) total: 4500 /unlink in 25.89 seconds: 173.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 893 (time 1788446929.65 total 10.00 last 89.30) - open/close 1799 (time 1788446939.66 total 20.01 last 90.48) - open/close 2763 (time 1788446949.67 total 30.02 last 96.33) - open/close 3712 (time 1788446959.68 total 40.03 last 94.84) total: 4500 open/close in 47.22 seconds: 95.30 ops/second - /unlink 2001 (time 1788446980.75 total 10.01 last 199.98) - /unlink 3835 (time 1788446990.75 total 20.01 last 183.29) total: 4500 /unlink in 24.12 seconds: 186.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 960 (time 1788447008.89 total 10.00 last 95.97) - open/close 1854 (time 1788447018.89 total 20.01 last 89.37) - open/close 2880 (time 1788447028.91 total 30.03 last 102.37) - open/close 3849 (time 1788447038.91 total 40.03 last 96.90) total: 4500 open/close in 46.83 seconds: 96.09 ops/second - /unlink 1700 (time 1788447059.43 total 10.00 last 169.98) - /unlink 3260 (time 1788447069.43 total 20.00 last 155.96) total: 4500 /unlink in 27.17 seconds: 165.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 914 (time 1788447090.25 total 10.00 last 91.38) - open/close 1669 (time 1788447100.27 total 20.02 last 75.36) - open/close 2463 (time 1788447110.29 total 30.04 last 79.24) - open/close 3424 (time 1788447120.30 total 40.05 last 95.97) - open/close 4308 (time 1788447130.31 total 50.06 last 88.38) total: 4500 open/close in 51.74 seconds: 86.97 ops/second - /unlink 1812 (time 1788447146.41 total 10.01 last 181.07) - /unlink 3353 (time 1788447156.41 total 20.01 last 154.07) total: 4500 /unlink in 28.45 seconds: 158.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 766 (time 1788447180.67 total 10.01 last 76.54) - open/close 1730 (time 1788447190.67 total 20.01 last 96.37) - open/close 2659 (time 1788447200.67 total 30.01 last 92.87) - open/close 3744 (time 1788447210.68 total 40.02 last 108.43) total: 4500 open/close in 47.23 seconds: 95.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1623 (time 1788447231.82 total 10.00 last 162.28) - /unlink 3411 (time 1788447241.82 total 20.00 last 178.78) total: 4500 /unlink in 26.13 seconds: 172.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 776 (time 1788447262.78 total 10.01 last 77.55) - open/close 1485 (time 1788447272.78 total 20.01 last 70.88) - open/close 2323 (time 1788447282.80 total 30.03 last 83.66) - open/close 3298 (time 1788447292.80 total 40.03 last 97.47) - open/close 4187 (time 1788447302.80 total 50.03 last 88.88) total: 4500 open/close in 53.09 seconds: 84.76 ops/second - /unlink 1702 (time 1788447320.01 total 10.00 last 170.16) - /unlink 3439 (time 1788447330.01 total 20.00 last 173.66) total: 4500 /unlink in 27.06 seconds: 166.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 845 (time 1788447351.80 total 10.02 last 84.34) - open/close 1834 (time 1788447361.80 total 20.03 last 98.83) - open/close 2815 (time 1788447371.81 total 30.04 last 98.02) - open/close 3780 (time 1788447381.82 total 40.05 last 96.38) total: 4500 open/close in 47.85 seconds: 94.05 ops/second - /unlink 1539 (time 1788447404.05 total 10.00 last 153.89) - /unlink 3298 (time 1788447414.05 total 20.00 last 175.90) total: 4500 /unlink in 27.08 seconds: 166.19 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95219 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg126-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg126-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg126-server stop mds service on oleg126-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg126-server unloading modules via unload_modules_local on: 'oleg126-server' oleg126-server: oleg126-server.virtnet: executing unload_modules_local oleg126-server: modules unloaded.