== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:27:38 (1787390858) Stopping clients: oleg612-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg612-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg612-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg612-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg612-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg612-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg612-server oleg612-server: oleg612-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing load_modules_local oleg612-server: Loading modules from /home/green/git/lustre-release/lustre oleg612-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg612-server: detected 4 online CPUs by sysfs oleg612-server: MODOPTS_LIBCFS= oleg612-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg612-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg612-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg612-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg612-server: oleg612-server.virtnet: executing set_default_debug -1 all pdsh@oleg612-client: oleg612-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg612-client.virtnet: -o user_xattr,flock 192.168.206.112@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 747 (time 1787391048.42 total 10.00 last 74.68) - open/close 1672 (time 1787391058.43 total 20.01 last 92.39) - open/close 2323 (time 1787391068.43 total 30.01 last 65.10) - open/close 3197 (time 1787391078.45 total 40.04 last 87.22) - open/close 4086 (time 1787391088.45 total 50.04 last 88.84) total: 4500 open/close in 54.57 seconds: 82.47 ops/second - /unlink 1822 (time 1787391107.91 total 10.00 last 182.14) - /unlink 3259 (time 1787391117.91 total 20.01 last 143.65) total: 4500 /unlink in 29.06 seconds: 154.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 822 (time 1787391141.73 total 10.00 last 82.19) - open/close 1720 (time 1787391151.73 total 20.00 last 89.79) - open/close 2702 (time 1787391161.73 total 30.01 last 98.17) - open/close 3633 (time 1787391171.75 total 40.03 last 92.92) total: 4500 open/close in 49.10 seconds: 91.65 ops/second - /unlink 1821 (time 1787391194.39 total 10.00 last 182.05) - /unlink 3452 (time 1787391204.39 total 20.00 last 163.06) total: 4500 /unlink in 25.92 seconds: 173.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 889 (time 1787391224.16 total 10.00 last 88.89) - open/close 1780 (time 1787391234.17 total 20.01 last 89.04) - open/close 2710 (time 1787391244.17 total 30.01 last 92.96) - open/close 3611 (time 1787391254.18 total 40.02 last 90.05) total: 4500 open/close in 49.95 seconds: 90.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1635 (time 1787391278.42 total 10.00 last 163.43) - /unlink 3200 (time 1787391288.42 total 20.01 last 156.48) total: 4500 /unlink in 27.75 seconds: 162.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 915 (time 1787391311.19 total 10.01 last 91.39) - open/close 1801 (time 1787391321.21 total 20.03 last 88.48) - open/close 2628 (time 1787391331.22 total 30.04 last 82.60) - open/close 3455 (time 1787391341.23 total 40.05 last 82.61) - open/close 4448 (time 1787391351.24 total 50.05 last 99.26) total: 4500 open/close in 50.55 seconds: 89.02 ops/second - /unlink 1594 (time 1787391365.93 total 10.00 last 159.35) - /unlink 3155 (time 1787391375.93 total 20.01 last 156.05) total: 4500 /unlink in 29.00 seconds: 155.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 679 (time 1787391399.83 total 10.01 last 67.86) - open/close 1410 (time 1787391409.83 total 20.01 last 73.06) - open/close 2233 (time 1787391419.85 total 30.03 last 82.18) - open/close 2999 (time 1787391429.86 total 40.04 last 76.51) - open/close 3803 (time 1787391439.86 total 50.04 last 80.40) total: 4500 open/close in 57.56 seconds: 78.18 ops/second - /unlink 1621 (time 1787391462.14 total 10.00 last 162.02) - /unlink 3285 (time 1787391472.15 total 20.01 last 166.32) total: 4500 /unlink in 27.78 seconds: 162.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 754 (time 1787391494.55 total 10.00 last 75.38) - open/close 1588 (time 1787391504.57 total 20.01 last 83.31) - open/close 2398 (time 1787391514.57 total 30.02 last 80.93) - open/close 3325 (time 1787391524.57 total 40.02 last 92.70) - open/close 4268 (time 1787391534.58 total 50.03 last 94.23) total: 4500 open/close in 52.37 seconds: 85.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1560 (time 1787391551.48 total 10.01 last 155.88) - /unlink 3141 (time 1787391561.49 total 20.01 last 158.04) total: 4500 /unlink in 28.14 seconds: 159.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 832 (time 1787391584.95 total 10.00 last 83.20) - open/close 1653 (time 1787391594.95 total 20.00 last 82.07) - open/close 2506 (time 1787391604.95 total 30.01 last 85.27) - open/close 3352 (time 1787391614.96 total 40.01 last 84.57) - open/close 4197 (time 1787391624.96 total 50.01 last 84.49) total: 4500 open/close in 53.44 seconds: 84.21 ops/second - /unlink 1443 (time 1787391641.65 total 10.00 last 144.25) - /unlink 2651 (time 1787391651.65 total 20.00 last 120.78) - /unlink 3878 (time 1787391661.65 total 30.01 last 122.67) total: 4500 /unlink in 33.69 seconds: 133.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 805 (time 1787391679.75 total 10.02 last 80.35) - open/close 1546 (time 1787391689.78 total 20.05 last 73.88) - open/close 2374 (time 1787391699.78 total 30.05 last 82.75) - open/close 3262 (time 1787391709.79 total 40.07 last 88.70) - open/close 4078 (time 1787391719.80 total 50.07 last 81.59) total: 4500 open/close in 55.98 seconds: 80.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1527 (time 1787391739.94 total 10.01 last 152.48) - /unlink 3039 (time 1787391749.94 total 20.02 last 151.13) total: 4500 /unlink in 29.62 seconds: 151.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 855 (time 1787391774.41 total 10.00 last 85.49) - open/close 1821 (time 1787391784.42 total 20.00 last 96.58) - open/close 2635 (time 1787391794.42 total 30.00 last 81.40) - open/close 3405 (time 1787391804.43 total 40.01 last 76.93) - open/close 4241 (time 1787391814.43 total 50.01 last 83.58) total: 4500 open/close in 52.74 seconds: 85.33 ops/second - /unlink 1709 (time 1787391831.68 total 10.00 last 170.88) - /unlink 3417 (time 1787391841.69 total 20.00 last 170.77) total: 4500 /unlink in 26.63 seconds: 169.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 869 (time 1787391862.78 total 10.00 last 86.89) - open/close 1758 (time 1787391872.79 total 20.01 last 88.82) - open/close 2402 (time 1787391882.79 total 30.02 last 64.36) - open/close 3148 (time 1787391892.80 total 40.02 last 74.54) - open/close 3928 (time 1787391902.81 total 50.03 last 77.93) total: 4500 open/close in 58.42 seconds: 77.03 ops/second - /unlink 1665 (time 1787391924.69 total 10.00 last 166.49) - /unlink 3605 (time 1787391934.69 total 20.00 last 193.99) total: 4500 /unlink in 24.22 seconds: 185.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1011 (time 1787391952.86 total 10.00 last 101.07) - open/close 1920 (time 1787391962.87 total 20.01 last 90.79) - open/close 2837 (time 1787391972.88 total 30.02 last 91.64) - open/close 3731 (time 1787391982.88 total 40.03 last 89.34) total: 4500 open/close in 48.84 seconds: 92.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1712 (time 1787392005.84 total 10.00 last 171.16) - /unlink 3123 (time 1787392015.85 total 20.01 last 141.03) total: 4500 /unlink in 27.77 seconds: 162.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 932 (time 1787392038.13 total 10.01 last 93.11) - open/close 1836 (time 1787392048.15 total 20.03 last 90.21) - open/close 2723 (time 1787392058.15 total 30.03 last 88.67) - open/close 3646 (time 1787392068.17 total 40.05 last 92.17) total: 4500 open/close in 49.69 seconds: 90.56 ops/second - /unlink 1687 (time 1787392091.70 total 10.01 last 168.57) - /unlink 3218 (time 1787392101.70 total 20.01 last 153.10) total: 4500 /unlink in 28.32 seconds: 158.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 847 (time 1787392124.36 total 10.02 last 84.53) - open/close 1693 (time 1787392134.37 total 20.03 last 84.54) - open/close 2622 (time 1787392144.37 total 30.03 last 92.86) - open/close 3422 (time 1787392154.38 total 40.04 last 79.97) - open/close 4285 (time 1787392164.38 total 50.04 last 86.24) total: 4500 open/close in 52.82 seconds: 85.19 ops/second - /unlink 1426 (time 1787392181.38 total 10.00 last 142.57) - /unlink 2997 (time 1787392191.38 total 20.00 last 157.08) total: 4500 /unlink in 29.15 seconds: 154.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 847 (time 1787392215.53 total 10.01 last 84.64) - open/close 1701 (time 1787392225.53 total 20.01 last 85.38) - open/close 2603 (time 1787392235.55 total 30.03 last 90.05) - open/close 3461 (time 1787392245.55 total 40.03 last 85.80) - open/close 4263 (time 1787392255.57 total 50.05 last 80.01) total: 4500 open/close in 52.87 seconds: 85.11 ops/second - /unlink 1163 (time 1787392274.10 total 10.01 last 116.23) - /unlink 2906 (time 1787392284.10 total 20.01 last 174.27) total: 4500 /unlink in 28.79 seconds: 156.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 867 (time 1787392307.78 total 10.00 last 86.66) - open/close 1758 (time 1787392317.79 total 20.01 last 89.08) - open/close 2618 (time 1787392327.80 total 30.02 last 85.87) - open/close 3374 (time 1787392337.82 total 40.04 last 75.48) - open/close 4209 (time 1787392347.82 total 50.04 last 83.46) total: 4500 open/close in 53.05 seconds: 84.83 ops/second - /unlink 1780 (time 1787392365.24 total 10.00 last 177.92) - /unlink 3380 (time 1787392375.24 total 20.01 last 159.97) total: 4500 /unlink in 27.56 seconds: 163.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 923 (time 1787392397.49 total 10.02 last 92.16) - open/close 1781 (time 1787392407.50 total 20.02 last 85.73) - open/close 2698 (time 1787392417.50 total 30.03 last 91.67) - open/close 3535 (time 1787392427.51 total 40.03 last 83.65) - open/close 4233 (time 1787392437.54 total 50.06 last 69.60) total: 4500 open/close in 52.88 seconds: 85.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1624 (time 1787392455.52 total 10.01 last 162.29) - /unlink 3261 (time 1787392465.53 total 20.02 last 163.54) total: 4500 /unlink in 28.25 seconds: 159.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 860 (time 1787392488.49 total 10.00 last 85.97) - open/close 1774 (time 1787392498.50 total 20.01 last 91.37) - open/close 2666 (time 1787392508.50 total 30.01 last 89.16) - open/close 3515 (time 1787392518.51 total 40.02 last 84.83) - open/close 4383 (time 1787392528.51 total 50.02 last 86.79) total: 4500 open/close in 51.25 seconds: 87.80 ops/second - /unlink 1649 (time 1787392543.72 total 10.00 last 164.84) - /unlink 3224 (time 1787392553.72 total 20.00 last 157.49) total: 4500 /unlink in 28.57 seconds: 157.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 904 (time 1787392576.97 total 10.00 last 90.38) - open/close 1764 (time 1787392586.97 total 20.01 last 85.96) - open/close 2706 (time 1787392596.98 total 30.01 last 94.13) - open/close 3574 (time 1787392606.99 total 40.02 last 86.72) - open/close 4479 (time 1787392617.00 total 50.03 last 90.40) total: 4500 open/close in 50.24 seconds: 89.57 ops/second - /unlink 1706 (time 1787392631.01 total 10.00 last 170.57) - /unlink 3351 (time 1787392641.01 total 20.01 last 164.42) total: 4500 /unlink in 27.13 seconds: 165.87 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95256 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg612-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg612-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg612-server stop mds service on oleg612-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg612-server stop mds service on oleg612-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg612-server unloading modules via unload_modules_local on: 'oleg612-server' oleg612-server: oleg612-server.virtnet: executing unload_modules_local oleg612-server: modules unloaded.