== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:36:55 (1786181815) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg360-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg360-server oleg360-server: oleg360-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg360-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg360-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg360-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 952 (time 1786181986.80 total 10.00 last 95.19) - open/close 1902 (time 1786181996.81 total 20.00 last 94.96) - open/close 2830 (time 1786182006.81 total 30.01 last 92.75) - open/close 3764 (time 1786182016.83 total 40.03 last 93.24) total: 4500 open/close in 48.20 seconds: 93.36 ops/second - /unlink 1672 (time 1786182039.61 total 10.00 last 167.15) - /unlink 3180 (time 1786182049.61 total 20.00 last 150.79) total: 4500 /unlink in 27.47 seconds: 163.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 923 (time 1786182071.78 total 10.01 last 92.23) - open/close 1789 (time 1786182081.78 total 20.01 last 86.60) - open/close 2755 (time 1786182091.79 total 30.02 last 96.51) - open/close 3740 (time 1786182101.80 total 40.03 last 98.41) total: 4500 open/close in 48.39 seconds: 92.99 ops/second - /unlink 1689 (time 1786182123.63 total 10.00 last 168.87) - /unlink 3312 (time 1786182133.63 total 20.00 last 162.27) total: 4500 /unlink in 27.08 seconds: 166.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 818 (time 1786182155.20 total 10.01 last 81.72) - open/close 1730 (time 1786182165.21 total 20.02 last 91.11) - open/close 2631 (time 1786182175.22 total 30.03 last 90.02) - open/close 3556 (time 1786182185.28 total 40.09 last 91.95) - open/close 4486 (time 1786182195.28 total 50.09 last 92.96) total: 4500 open/close in 50.22 seconds: 89.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1520 (time 1786182210.05 total 10.00 last 151.98) - /unlink 2765 (time 1786182220.05 total 20.00 last 124.50) - /unlink 4367 (time 1786182230.05 total 30.00 last 160.19) total: 4500 /unlink in 30.99 seconds: 145.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 818 (time 1786182245.76 total 10.00 last 81.78) - open/close 1758 (time 1786182255.76 total 20.01 last 93.95) - open/close 2725 (time 1786182265.77 total 30.01 last 96.64) - open/close 3642 (time 1786182275.77 total 40.02 last 91.67) total: 4500 open/close in 49.01 seconds: 91.82 ops/second - /unlink 1651 (time 1786182298.67 total 10.00 last 165.07) - /unlink 2956 (time 1786182308.67 total 20.00 last 130.49) total: 4500 /unlink in 29.80 seconds: 151.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 902 (time 1786182332.93 total 10.00 last 90.18) - open/close 1835 (time 1786182342.94 total 20.01 last 93.19) - open/close 2725 (time 1786182352.95 total 30.03 last 88.88) - open/close 3706 (time 1786182362.95 total 40.03 last 98.07) total: 4500 open/close in 48.84 seconds: 92.13 ops/second - /unlink 1614 (time 1786182385.82 total 10.00 last 161.35) - /unlink 3170 (time 1786182395.82 total 20.01 last 155.55) total: 4500 /unlink in 28.35 seconds: 158.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 904 (time 1786182420.44 total 10.00 last 90.38) - open/close 1749 (time 1786182430.45 total 20.02 last 84.39) - open/close 2628 (time 1786182440.46 total 30.02 last 87.82) - open/close 3617 (time 1786182450.46 total 40.03 last 98.89) - open/close 4455 (time 1786182460.47 total 50.03 last 83.74) total: 4500 open/close in 50.41 seconds: 89.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1450 (time 1786182474.64 total 10.00 last 144.95) - /unlink 2961 (time 1786182484.65 total 20.01 last 150.98) total: 4500 /unlink in 29.36 seconds: 153.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 950 (time 1786182509.59 total 10.00 last 94.96) - open/close 1859 (time 1786182519.60 total 20.01 last 90.83) - open/close 2781 (time 1786182529.61 total 30.03 last 92.07) - open/close 3717 (time 1786182539.62 total 40.03 last 93.55) total: 4500 open/close in 48.31 seconds: 93.15 ops/second - /unlink 1671 (time 1786182561.96 total 10.01 last 166.87) - /unlink 3115 (time 1786182571.96 total 20.02 last 144.35) total: 4500 /unlink in 28.98 seconds: 155.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 842 (time 1786182595.90 total 10.02 last 84.04) - open/close 1700 (time 1786182605.93 total 20.05 last 85.55) - open/close 2622 (time 1786182615.94 total 30.05 last 92.16) - open/close 3578 (time 1786182625.94 total 40.06 last 95.56) total: 4500 open/close in 49.59 seconds: 90.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1689 (time 1786182648.94 total 10.00 last 168.88) - /unlink 3271 (time 1786182658.95 total 20.00 last 158.15) total: 4500 /unlink in 27.74 seconds: 162.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 975 (time 1786182680.21 total 10.00 last 97.48) - open/close 1888 (time 1786182690.23 total 20.02 last 91.14) - open/close 2801 (time 1786182700.24 total 30.03 last 91.19) - open/close 3722 (time 1786182710.24 total 40.03 last 92.09) total: 4500 open/close in 48.31 seconds: 93.15 ops/second - /unlink 1497 (time 1786182732.33 total 10.01 last 149.49) - /unlink 3020 (time 1786182742.34 total 20.02 last 152.19) total: 4500 /unlink in 29.06 seconds: 154.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 895 (time 1786182765.81 total 10.00 last 89.49) - open/close 1798 (time 1786182775.81 total 20.01 last 90.26) - open/close 2728 (time 1786182785.82 total 30.01 last 92.92) - open/close 3708 (time 1786182795.84 total 40.03 last 97.83) total: 4500 open/close in 47.99 seconds: 93.77 ops/second - /unlink 1653 (time 1786182817.51 total 10.00 last 165.27) - /unlink 3188 (time 1786182827.51 total 20.01 last 153.44) total: 4500 /unlink in 28.35 seconds: 158.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 896 (time 1786182850.85 total 10.00 last 89.56) - open/close 1777 (time 1786182860.86 total 20.01 last 88.01) - open/close 2666 (time 1786182870.86 total 30.02 last 88.88) - open/close 3614 (time 1786182880.87 total 40.02 last 94.73) total: 4500 open/close in 49.77 seconds: 90.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1541 (time 1786182905.44 total 10.00 last 154.09) - /unlink 3159 (time 1786182915.44 total 20.00 last 161.73) total: 4500 /unlink in 29.66 seconds: 151.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 820 (time 1786182941.13 total 10.17 last 80.62) - open/close 1688 (time 1786182951.13 total 20.17 last 86.77) - open/close 2444 (time 1786182961.14 total 30.18 last 75.58) - open/close 3316 (time 1786182971.16 total 40.20 last 86.98) - open/close 4087 (time 1786182981.17 total 50.21 last 77.05) total: 4500 open/close in 54.63 seconds: 82.37 ops/second - /unlink 1417 (time 1786182999.69 total 10.00 last 141.64) - /unlink 3071 (time 1786183009.69 total 20.00 last 165.40) total: 4500 /unlink in 28.43 seconds: 158.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 883 (time 1786183032.55 total 10.01 last 88.25) - open/close 1724 (time 1786183042.55 total 20.01 last 84.05) - open/close 2637 (time 1786183052.58 total 30.04 last 91.03) - open/close 3581 (time 1786183062.60 total 40.05 last 94.28) - open/close 4462 (time 1786183072.61 total 50.07 last 88.00) total: 4500 open/close in 50.49 seconds: 89.13 ops/second - /unlink 1504 (time 1786183087.73 total 10.01 last 150.25) - /unlink 3023 (time 1786183097.73 total 20.01 last 151.86) total: 4500 /unlink in 29.66 seconds: 151.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 917 (time 1786183122.44 total 10.00 last 91.69) - open/close 1761 (time 1786183132.44 total 20.01 last 84.36) - open/close 2660 (time 1786183142.45 total 30.01 last 89.89) - open/close 3580 (time 1786183152.45 total 40.01 last 91.98) - open/close 4499 (time 1786183162.46 total 50.02 last 91.80) total: 4500 open/close in 50.02 seconds: 89.96 ops/second - /unlink 1719 (time 1786183176.73 total 10.00 last 171.88) - /unlink 3006 (time 1786183186.76 total 20.03 last 128.31) total: 4500 /unlink in 30.01 seconds: 149.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 874 (time 1786183210.72 total 10.01 last 87.30) - open/close 1767 (time 1786183220.73 total 20.02 last 89.20) - open/close 2644 (time 1786183230.73 total 30.03 last 87.66) - open/close 3604 (time 1786183240.74 total 40.03 last 95.95) total: 4500 open/close in 49.57 seconds: 90.79 ops/second - /unlink 1671 (time 1786183264.30 total 10.00 last 167.09) - /unlink 3295 (time 1786183274.30 total 20.00 last 162.35) total: 4500 /unlink in 27.11 seconds: 166.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 877 (time 1786183296.73 total 10.00 last 87.67) - open/close 1814 (time 1786183306.73 total 20.00 last 93.70) - open/close 2686 (time 1786183316.73 total 30.00 last 87.19) - open/close 3565 (time 1786183326.73 total 40.01 last 87.89) - open/close 4302 (time 1786183336.74 total 50.01 last 73.67) total: 4500 open/close in 52.57 seconds: 85.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1396 (time 1786183354.35 total 10.01 last 139.42) - /unlink 3051 (time 1786183364.36 total 20.02 last 165.43) total: 4500 /unlink in 29.22 seconds: 154.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 884 (time 1786183390.10 total 10.01 last 88.31) - open/close 1650 (time 1786183400.12 total 20.03 last 76.48) - open/close 2544 (time 1786183410.12 total 30.03 last 89.36) - open/close 3375 (time 1786183420.13 total 40.04 last 83.04) - open/close 4254 (time 1786183430.14 total 50.04 last 87.84) total: 4500 open/close in 52.72 seconds: 85.36 ops/second - /unlink 1620 (time 1786183447.45 total 10.08 last 160.73) - /unlink 3001 (time 1786183457.45 total 20.08 last 138.08) total: 4500 /unlink in 29.51 seconds: 152.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 797 (time 1786183480.91 total 10.00 last 79.67) - open/close 1670 (time 1786183490.94 total 20.03 last 87.05) - open/close 2630 (time 1786183500.95 total 30.04 last 95.91) - open/close 3527 (time 1786183510.99 total 40.08 last 89.39) - open/close 4418 (time 1786183520.99 total 50.08 last 89.06) total: 4500 open/close in 50.87 seconds: 88.46 ops/second - /unlink 1710 (time 1786183536.76 total 10.00 last 170.94) - /unlink 3352 (time 1786183546.76 total 20.00 last 164.18) total: 4500 /unlink in 27.17 seconds: 165.62 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95157 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local oleg360-server: modules unloaded.