== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:09:11 (1786093751) Stopping clients: oleg106-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg106-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg106-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg106-server oleg106-server: oleg106-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg106-server: detected 4 online CPUs by sysfs oleg106-server: MODOPTS_LIBCFS= oleg106-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg106-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg106-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 802 (time 1786093947.24 total 10.00 last 80.20) - open/close 1667 (time 1786093957.24 total 20.00 last 86.48) - open/close 2459 (time 1786093967.25 total 30.01 last 79.15) - open/close 3097 (time 1786093977.25 total 40.02 last 63.76) - open/close 3654 (time 1786093987.25 total 50.02 last 55.69) - open/close 4363 (time 1786093997.28 total 60.04 last 70.73) total: 4500 open/close in 61.88 seconds: 72.72 ops/second - /unlink 1641 (time 1786094014.35 total 10.00 last 164.04) - /unlink 3039 (time 1786094024.35 total 20.00 last 139.79) - /unlink 4450 (time 1786094034.35 total 30.01 last 141.09) total: 4500 /unlink in 30.45 seconds: 147.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 601 (time 1786094050.40 total 10.01 last 60.04) - open/close 1339 (time 1786094060.41 total 20.02 last 73.75) - open/close 2025 (time 1786094070.41 total 30.02 last 68.58) - open/close 2626 (time 1786094080.42 total 40.03 last 60.05) - open/close 3251 (time 1786094090.42 total 50.03 last 62.48) - open/close 4201 (time 1786094100.43 total 60.03 last 94.96) total: 4500 open/close in 63.88 seconds: 70.45 ops/second - /unlink 1608 (time 1786094119.05 total 10.00 last 160.75) - /unlink 3168 (time 1786094129.06 total 20.01 last 155.93) total: 4500 /unlink in 29.37 seconds: 153.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 637 (time 1786094153.76 total 10.01 last 63.63) - open/close 1271 (time 1786094163.76 total 20.01 last 63.39) - open/close 1983 (time 1786094173.76 total 30.02 last 71.19) - open/close 2705 (time 1786094183.76 total 40.02 last 72.17) - open/close 3616 (time 1786094193.77 total 50.02 last 91.10) - open/close 4480 (time 1786094203.77 total 60.03 last 86.35) total: 4500 open/close in 60.30 seconds: 74.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1653 (time 1786094218.37 total 10.01 last 165.17) - /unlink 3180 (time 1786094228.37 total 20.01 last 152.65) total: 4500 /unlink in 28.60 seconds: 157.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 870 (time 1786094251.99 total 10.00 last 87.00) - open/close 1781 (time 1786094262.01 total 20.02 last 90.94) - open/close 2659 (time 1786094272.02 total 30.03 last 87.68) - open/close 3443 (time 1786094282.03 total 40.04 last 78.34) - open/close 4402 (time 1786094292.03 total 50.04 last 95.90) total: 4500 open/close in 50.88 seconds: 88.45 ops/second - /unlink 1243 (time 1786094307.00 total 10.00 last 124.26) - /unlink 2435 (time 1786094317.00 total 20.01 last 119.17) - /unlink 4080 (time 1786094327.01 total 30.01 last 164.43) total: 4500 /unlink in 32.77 seconds: 137.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 887 (time 1786094343.87 total 10.00 last 88.69) - open/close 1722 (time 1786094353.87 total 20.00 last 83.50) - open/close 2639 (time 1786094363.88 total 30.01 last 91.67) - open/close 3569 (time 1786094373.89 total 40.01 last 92.92) - open/close 4439 (time 1786094383.89 total 50.01 last 87.00) total: 4500 open/close in 50.79 seconds: 88.61 ops/second - /unlink 1301 (time 1786094399.84 total 10.01 last 130.02) - /unlink 2632 (time 1786094409.84 total 20.01 last 133.04) - /unlink 4264 (time 1786094419.85 total 30.02 last 163.02) total: 4500 /unlink in 32.17 seconds: 139.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 746 (time 1786094436.94 total 10.01 last 74.56) - open/close 1618 (time 1786094446.94 total 20.01 last 87.14) - open/close 2359 (time 1786094456.94 total 30.01 last 74.10) - open/close 3153 (time 1786094466.97 total 40.04 last 79.19) - open/close 3938 (time 1786094476.98 total 50.05 last 78.44) total: 4500 open/close in 58.61 seconds: 76.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1420 (time 1786094500.00 total 10.00 last 141.97) - /unlink 2735 (time 1786094510.02 total 20.01 last 131.34) - /unlink 4260 (time 1786094520.02 total 30.02 last 152.49) total: 4500 /unlink in 32.19 seconds: 139.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 829 (time 1786094536.21 total 10.00 last 82.90) - open/close 1620 (time 1786094546.21 total 20.00 last 79.08) - open/close 2283 (time 1786094556.21 total 30.00 last 66.29) - open/close 2877 (time 1786094566.22 total 40.01 last 59.39) - open/close 3679 (time 1786094576.22 total 50.01 last 80.19) - open/close 4373 (time 1786094586.22 total 60.01 last 69.39) total: 4500 open/close in 62.00 seconds: 72.58 ops/second - /unlink 1236 (time 1786094603.07 total 10.01 last 123.52) - /unlink 2851 (time 1786094613.07 total 20.01 last 161.45) - /unlink 4227 (time 1786094623.08 total 30.02 last 137.50) total: 4500 /unlink in 32.36 seconds: 139.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 831 (time 1786094640.50 total 10.01 last 83.05) - open/close 1550 (time 1786094650.51 total 20.01 last 71.84) - open/close 2197 (time 1786094660.51 total 30.02 last 64.66) - open/close 2810 (time 1786094670.52 total 40.03 last 61.25) - open/close 3435 (time 1786094680.53 total 50.04 last 62.43) - open/close 4127 (time 1786094690.54 total 60.04 last 69.18) total: 4500 open/close in 66.31 seconds: 67.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1434 (time 1786094712.60 total 10.00 last 143.33) - /unlink 2504 (time 1786094722.61 total 20.01 last 106.94) - /unlink 3691 (time 1786094732.61 total 30.02 last 118.63) total: 4500 /unlink in 35.88 seconds: 125.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 871 (time 1786094752.53 total 10.00 last 87.06) - open/close 1788 (time 1786094762.53 total 20.01 last 91.68) - open/close 2584 (time 1786094772.55 total 30.02 last 79.49) - open/close 3345 (time 1786094782.55 total 40.03 last 76.05) - open/close 4085 (time 1786094792.58 total 50.05 last 73.82) total: 4500 open/close in 55.53 seconds: 81.03 ops/second - /unlink 1364 (time 1786094812.65 total 10.00 last 136.37) - /unlink 2787 (time 1786094822.66 total 20.01 last 142.26) - /unlink 4289 (time 1786094832.66 total 30.01 last 150.15) total: 4500 /unlink in 31.83 seconds: 141.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 698 (time 1786094848.94 total 10.02 last 69.68) - open/close 1427 (time 1786094858.94 total 20.02 last 72.88) - open/close 2046 (time 1786094868.97 total 30.04 last 61.76) - open/close 2827 (time 1786094878.98 total 40.06 last 78.00) - open/close 3628 (time 1786094888.99 total 50.07 last 79.99) total: 4500 open/close in 59.39 seconds: 75.77 ops/second - /unlink 1506 (time 1786094912.84 total 10.00 last 150.57) - /unlink 3171 (time 1786094922.85 total 20.01 last 166.38) total: 4500 /unlink in 28.47 seconds: 158.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 613 (time 1786094946.35 total 10.00 last 61.29) - open/close 1284 (time 1786094956.36 total 20.01 last 67.02) - open/close 1981 (time 1786094966.39 total 30.04 last 69.49) - open/close 2775 (time 1786094976.40 total 40.05 last 79.36) - open/close 3620 (time 1786094986.42 total 50.07 last 84.35) - open/close 4415 (time 1786094996.42 total 60.07 last 79.47) total: 4500 open/close in 61.08 seconds: 73.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1544 (time 1786095012.59 total 10.01 last 154.19) - /unlink 2810 (time 1786095022.60 total 20.02 last 126.47) - /unlink 4203 (time 1786095032.61 total 30.02 last 139.29) total: 4500 /unlink in 32.19 seconds: 139.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 742 (time 1786095050.05 total 10.00 last 74.19) - open/close 1607 (time 1786095060.05 total 20.00 last 86.47) - open/close 2446 (time 1786095070.07 total 30.02 last 83.79) - open/close 3254 (time 1786095080.08 total 40.03 last 80.66) - open/close 4043 (time 1786095090.09 total 50.04 last 78.85) total: 4500 open/close in 54.94 seconds: 81.90 ops/second - /unlink 1304 (time 1786095109.63 total 10.00 last 130.34) - /unlink 2508 (time 1786095119.63 total 20.01 last 120.36) - /unlink 3579 (time 1786095129.65 total 30.02 last 106.95) total: 4500 /unlink in 37.67 seconds: 119.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 884 (time 1786095153.13 total 10.00 last 88.36) - open/close 1804 (time 1786095163.13 total 20.00 last 91.99) - open/close 2654 (time 1786095173.14 total 30.01 last 84.95) - open/close 3423 (time 1786095183.15 total 40.02 last 76.82) - open/close 4147 (time 1786095193.16 total 50.03 last 72.34) total: 4500 open/close in 53.79 seconds: 83.66 ops/second - /unlink 1449 (time 1786095212.46 total 10.00 last 144.83) - /unlink 2967 (time 1786095222.46 total 20.01 last 151.76) total: 4500 /unlink in 29.27 seconds: 153.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 985 (time 1786095246.21 total 10.01 last 98.44) - open/close 1864 (time 1786095256.21 total 20.01 last 87.87) - open/close 2859 (time 1786095266.22 total 30.02 last 99.43) - open/close 3671 (time 1786095276.22 total 40.02 last 81.14) total: 4500 open/close in 49.51 seconds: 90.90 ops/second - /unlink 1732 (time 1786095300.22 total 10.00 last 173.14) - /unlink 3334 (time 1786095310.22 total 20.01 last 160.15) total: 4500 /unlink in 26.64 seconds: 168.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 836 (time 1786095332.34 total 10.00 last 83.60) - open/close 1760 (time 1786095342.35 total 20.01 last 92.29) - open/close 2676 (time 1786095352.35 total 30.02 last 91.56) - open/close 3629 (time 1786095362.35 total 40.02 last 95.28) total: 4500 open/close in 49.18 seconds: 91.50 ops/second - /unlink 1900 (time 1786095385.37 total 10.00 last 189.96) - /unlink 3406 (time 1786095395.37 total 20.01 last 150.53) total: 4500 /unlink in 27.50 seconds: 163.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 869 (time 1786095416.94 total 10.01 last 86.82) - open/close 1738 (time 1786095426.94 total 20.01 last 86.85) - open/close 2567 (time 1786095436.95 total 30.02 last 82.86) - open/close 3380 (time 1786095446.96 total 40.03 last 81.22) - open/close 4345 (time 1786095456.96 total 50.03 last 96.50) total: 4500 open/close in 51.88 seconds: 86.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1580 (time 1786095473.04 total 10.00 last 157.95) - /unlink 3106 (time 1786095483.05 total 20.01 last 152.55) - /unlink 4433 (time 1786095493.05 total 30.01 last 132.67) total: 4500 /unlink in 30.60 seconds: 147.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 739 (time 1786095509.82 total 10.00 last 73.86) - open/close 1518 (time 1786095519.82 total 20.01 last 77.87) - open/close 2476 (time 1786095529.83 total 30.01 last 95.78) - open/close 3341 (time 1786095539.83 total 40.02 last 86.45) - open/close 4264 (time 1786095549.83 total 50.02 last 92.30) total: 4500 open/close in 52.28 seconds: 86.07 ops/second - /unlink 1487 (time 1786095566.07 total 10.00 last 148.69) - /unlink 2812 (time 1786095576.07 total 20.01 last 132.44) - /unlink 4361 (time 1786095586.08 total 30.01 last 154.85) total: 4500 /unlink in 31.04 seconds: 144.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 888 (time 1786095602.06 total 10.01 last 88.71) - open/close 1838 (time 1786095612.07 total 20.01 last 94.96) - open/close 2734 (time 1786095622.07 total 30.02 last 89.56) - open/close 3680 (time 1786095632.09 total 40.04 last 94.39) total: 4500 open/close in 49.22 seconds: 91.42 ops/second - /unlink 1745 (time 1786095655.61 total 10.01 last 174.33) - /unlink 3098 (time 1786095665.62 total 20.02 last 135.16) - /unlink 4267 (time 1786095675.62 total 30.02 last 116.85) total: 4500 /unlink in 31.99 seconds: 140.65 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95261 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg106-server unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local oleg106-server: modules unloaded.