== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:14:17 (1788156857) Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg202-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server oleg202-server: oleg202-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing load_modules_local oleg202-server: Loading modules from /home/green/git/lustre-release/lustre oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg202-server: detected 4 online CPUs by sysfs oleg202-server: MODOPTS_LIBCFS= oleg202-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg202-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg202-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg202-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 895 (time 1788157076.05 total 10.01 last 89.41) - open/close 1666 (time 1788157086.06 total 20.02 last 77.01) - open/close 2310 (time 1788157096.09 total 30.06 last 64.19) - open/close 3152 (time 1788157106.12 total 40.08 last 84.02) - open/close 3985 (time 1788157116.12 total 50.08 last 83.25) total: 4500 open/close in 56.41 seconds: 79.77 ops/second - /unlink 1477 (time 1788157136.96 total 10.00 last 147.65) - /unlink 2701 (time 1788157146.97 total 20.01 last 122.36) - /unlink 3858 (time 1788157156.97 total 30.01 last 115.67) total: 4500 /unlink in 33.49 seconds: 134.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 581 (time 1788157176.57 total 10.01 last 58.06) - open/close 1453 (time 1788157186.58 total 20.01 last 87.16) - open/close 2252 (time 1788157196.58 total 30.02 last 79.83) - open/close 3177 (time 1788157206.59 total 40.03 last 92.45) - open/close 4066 (time 1788157216.60 total 50.04 last 88.80) total: 4500 open/close in 55.69 seconds: 80.80 ops/second - /unlink 1351 (time 1788157237.31 total 10.01 last 135.00) - /unlink 2789 (time 1788157247.31 total 20.01 last 143.73) - /unlink 4157 (time 1788157257.32 total 30.02 last 136.69) total: 4500 /unlink in 32.88 seconds: 136.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 637 (time 1788157275.35 total 10.00 last 63.67) - open/close 1400 (time 1788157285.35 total 20.00 last 76.30) - open/close 2108 (time 1788157295.36 total 30.02 last 70.73) - open/close 2847 (time 1788157305.36 total 40.02 last 73.88) - open/close 3660 (time 1788157315.37 total 50.03 last 81.20) - open/close 4453 (time 1788157325.38 total 60.03 last 79.28) total: 4500 open/close in 60.44 seconds: 74.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1386 (time 1788157340.62 total 10.00 last 138.55) - /unlink 2607 (time 1788157350.63 total 20.01 last 122.06) - /unlink 3698 (time 1788157360.63 total 30.01 last 109.05) total: 4500 /unlink in 36.29 seconds: 124.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 730 (time 1788157382.38 total 10.01 last 72.90) - open/close 1625 (time 1788157392.39 total 20.02 last 89.44) - open/close 2433 (time 1788157402.39 total 30.03 last 80.74) - open/close 3310 (time 1788157412.39 total 40.03 last 87.69) - open/close 4158 (time 1788157422.42 total 50.05 last 84.60) total: 4500 open/close in 53.56 seconds: 84.02 ops/second - /unlink 1524 (time 1788157440.08 total 10.00 last 152.39) - /unlink 3010 (time 1788157450.08 total 20.00 last 148.56) - /unlink 4279 (time 1788157460.08 total 30.01 last 126.81) total: 4500 /unlink in 31.81 seconds: 141.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 490 (time 1788157477.88 total 10.00 last 48.98) - open/close 1007 (time 1788157487.91 total 20.04 last 51.53) - open/close 1704 (time 1788157497.91 total 30.04 last 69.66) - open/close 2467 (time 1788157507.92 total 40.04 last 76.30) - open/close 3053 (time 1788157517.92 total 50.05 last 58.58) - open/close 3927 (time 1788157527.92 total 60.05 last 87.34) total: 4500 open/close in 66.80 seconds: 67.37 ops/second - /unlink 1291 (time 1788157549.92 total 10.01 last 129.03) - /unlink 2718 (time 1788157559.92 total 20.01 last 142.66) - /unlink 4209 (time 1788157569.92 total 30.01 last 149.08) total: 4500 /unlink in 31.90 seconds: 141.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 891 (time 1788157586.27 total 10.01 last 89.00) - open/close 1704 (time 1788157596.28 total 20.03 last 81.17) - open/close 2377 (time 1788157606.30 total 30.04 last 67.20) - open/close 3004 (time 1788157616.30 total 40.04 last 62.70) - open/close 3703 (time 1788157626.30 total 50.05 last 69.86) total: 4500 open/close in 59.70 seconds: 75.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1418 (time 1788157651.04 total 10.00 last 141.76) - /unlink 2996 (time 1788157661.05 total 20.02 last 157.56) - /unlink 4442 (time 1788157671.06 total 30.02 last 144.51) total: 4500 /unlink in 30.42 seconds: 147.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 817 (time 1788157686.21 total 10.00 last 81.69) - open/close 1678 (time 1788157696.23 total 20.02 last 85.94) - open/close 2438 (time 1788157706.24 total 30.03 last 75.94) - open/close 3356 (time 1788157716.25 total 40.04 last 91.70) - open/close 3917 (time 1788157726.25 total 50.04 last 56.08) total: 4500 open/close in 57.59 seconds: 78.14 ops/second - /unlink 1457 (time 1788157748.80 total 10.00 last 145.66) - /unlink 3101 (time 1788157758.81 total 20.01 last 164.35) total: 4500 /unlink in 28.38 seconds: 158.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 864 (time 1788157781.42 total 10.00 last 86.39) - open/close 1622 (time 1788157791.43 total 20.01 last 75.71) - open/close 2474 (time 1788157801.44 total 30.02 last 85.18) - open/close 3251 (time 1788157811.44 total 40.02 last 77.69) - open/close 4048 (time 1788157821.45 total 50.03 last 79.58) total: 4500 open/close in 58.94 seconds: 76.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1596 (time 1788157845.05 total 10.00 last 159.53) - /unlink 2955 (time 1788157855.05 total 20.01 last 135.85) - /unlink 4481 (time 1788157865.05 total 30.01 last 152.59) total: 4500 /unlink in 30.11 seconds: 149.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 747 (time 1788157880.28 total 10.01 last 74.62) - open/close 1519 (time 1788157890.29 total 20.02 last 77.12) - open/close 2301 (time 1788157900.30 total 30.03 last 78.13) - open/close 2910 (time 1788157910.33 total 40.06 last 60.71) - open/close 3641 (time 1788157920.34 total 50.07 last 73.02) - open/close 4456 (time 1788157930.35 total 60.08 last 81.44) total: 4500 open/close in 60.46 seconds: 74.42 ops/second - /unlink 1404 (time 1788157945.87 total 10.01 last 140.31) - /unlink 2991 (time 1788157955.88 total 20.01 last 158.57) total: 4500 /unlink in 29.71 seconds: 151.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 826 (time 1788157980.11 total 10.01 last 82.51) - open/close 1613 (time 1788157990.11 total 20.01 last 78.68) - open/close 2336 (time 1788158000.12 total 30.03 last 72.21) - open/close 3111 (time 1788158010.15 total 40.05 last 77.31) - open/close 3627 (time 1788158020.15 total 50.05 last 51.60) - open/close 4339 (time 1788158030.15 total 60.05 last 71.18) total: 4500 open/close in 61.91 seconds: 72.68 ops/second - /unlink 1402 (time 1788158046.08 total 10.01 last 140.08) - /unlink 2910 (time 1788158056.09 total 20.02 last 150.64) total: 4500 /unlink in 29.65 seconds: 151.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 854 (time 1788158080.16 total 10.02 last 85.25) - open/close 1665 (time 1788158090.16 total 20.02 last 81.05) - open/close 2421 (time 1788158100.17 total 30.03 last 75.54) - open/close 3313 (time 1788158110.19 total 40.05 last 89.04) - open/close 4164 (time 1788158120.19 total 50.05 last 85.07) total: 4500 open/close in 53.63 seconds: 83.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1823 (time 1788158138.00 total 10.00 last 182.29) - /unlink 3312 (time 1788158148.03 total 20.02 last 148.55) - /unlink 4424 (time 1788158158.03 total 30.03 last 111.18) total: 4500 /unlink in 30.56 seconds: 147.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 735 (time 1788158172.96 total 10.00 last 73.50) - open/close 1614 (time 1788158182.96 total 20.00 last 87.89) - open/close 2553 (time 1788158192.96 total 30.00 last 93.88) - open/close 3447 (time 1788158202.96 total 40.00 last 89.40) - open/close 4277 (time 1788158212.97 total 50.01 last 82.95) total: 4500 open/close in 52.82 seconds: 85.19 ops/second - /unlink 1733 (time 1788158230.22 total 10.00 last 173.28) - /unlink 3311 (time 1788158240.23 total 20.00 last 157.75) total: 4500 /unlink in 26.98 seconds: 166.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 760 (time 1788158261.28 total 10.00 last 75.98) - open/close 1614 (time 1788158271.29 total 20.01 last 85.35) - open/close 2495 (time 1788158281.30 total 30.02 last 88.03) - open/close 3318 (time 1788158291.30 total 40.02 last 82.29) - open/close 4320 (time 1788158301.30 total 50.02 last 100.14) total: 4500 open/close in 51.66 seconds: 87.11 ops/second - /unlink 1850 (time 1788158316.29 total 10.00 last 184.95) - /unlink 3286 (time 1788158326.29 total 20.00 last 143.60) total: 4500 /unlink in 26.97 seconds: 166.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 932 (time 1788158347.80 total 10.01 last 93.06) - open/close 1925 (time 1788158357.81 total 20.03 last 99.19) - open/close 2850 (time 1788158367.82 total 30.04 last 92.39) - open/close 3861 (time 1788158377.82 total 40.04 last 101.07) total: 4500 open/close in 46.55 seconds: 96.68 ops/second - /unlink 1935 (time 1788158397.58 total 10.01 last 193.36) - /unlink 3680 (time 1788158407.59 total 20.02 last 174.35) total: 4500 /unlink in 24.68 seconds: 182.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 685 (time 1788158427.22 total 10.03 last 68.32) - open/close 1674 (time 1788158437.23 total 20.04 last 98.72) - open/close 2576 (time 1788158447.23 total 30.04 last 90.20) - open/close 3686 (time 1788158457.24 total 40.05 last 110.97) total: 4500 open/close in 47.96 seconds: 93.83 ops/second - /unlink 1668 (time 1788158479.28 total 10.00 last 166.76) - /unlink 3285 (time 1788158489.28 total 20.00 last 161.69) total: 4500 /unlink in 28.11 seconds: 160.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1007 (time 1788158511.31 total 10.00 last 100.67) - open/close 1871 (time 1788158521.32 total 20.01 last 86.33) - open/close 2681 (time 1788158531.34 total 30.03 last 80.87) - open/close 3593 (time 1788158541.35 total 40.03 last 91.13) total: 4500 open/close in 49.86 seconds: 90.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1771 (time 1788158564.96 total 10.00 last 177.05) - /unlink 3465 (time 1788158574.96 total 20.00 last 169.37) total: 4500 /unlink in 28.09 seconds: 160.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 860 (time 1788158597.50 total 10.02 last 85.83) - open/close 1722 (time 1788158607.50 total 20.02 last 86.19) - open/close 2683 (time 1788158617.50 total 30.02 last 96.08) - open/close 3578 (time 1788158627.51 total 40.03 last 89.45) - open/close 4443 (time 1788158637.51 total 50.03 last 86.50) total: 4500 open/close in 50.78 seconds: 88.61 ops/second - /unlink 1519 (time 1788158652.48 total 10.00 last 151.89) - /unlink 3116 (time 1788158662.48 total 20.00 last 159.66) total: 4500 /unlink in 29.61 seconds: 152.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 985 (time 1788158686.48 total 10.01 last 98.42) - open/close 1742 (time 1788158696.49 total 20.01 last 75.65) - open/close 2573 (time 1788158706.49 total 30.02 last 83.08) - open/close 3494 (time 1788158716.50 total 40.02 last 92.03) - open/close 4297 (time 1788158726.51 total 50.03 last 80.22) total: 4500 open/close in 52.14 seconds: 86.31 ops/second - /unlink 1604 (time 1788158743.08 total 10.01 last 160.29) - /unlink 2920 (time 1788158753.08 total 20.01 last 131.57) - /unlink 4463 (time 1788158763.08 total 30.01 last 154.30) total: 4500 /unlink in 30.26 seconds: 148.72 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95220 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg202-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server unloading modules via unload_modules_local on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing unload_modules_local oleg202-server: modules unloaded.