== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:09:29 (1788444569) Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg140-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg140-server oleg140-server: oleg140-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg140-server: detected 4 online CPUs by sysfs oleg140-server: MODOPTS_LIBCFS= oleg140-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg140-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg140-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg140-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 832 (time 1788444796.67 total 10.00 last 83.20) - open/close 1635 (time 1788444806.67 total 20.01 last 80.25) - open/close 2578 (time 1788444816.68 total 30.01 last 94.28) - open/close 3522 (time 1788444826.68 total 40.01 last 94.37) - open/close 4357 (time 1788444836.68 total 50.01 last 83.48) total: 4500 open/close in 52.05 seconds: 86.46 ops/second - /unlink 1465 (time 1788444852.93 total 10.00 last 146.49) - /unlink 2648 (time 1788444862.94 total 20.01 last 118.20) - /unlink 3801 (time 1788444872.95 total 30.02 last 115.15) total: 4500 /unlink in 34.52 seconds: 130.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 853 (time 1788444892.55 total 10.01 last 85.19) - open/close 1599 (time 1788444902.56 total 20.03 last 74.51) - open/close 2295 (time 1788444912.58 total 30.04 last 69.47) - open/close 3108 (time 1788444922.58 total 40.05 last 81.27) - open/close 3948 (time 1788444932.59 total 50.05 last 83.94) total: 4500 open/close in 56.77 seconds: 79.27 ops/second - /unlink 1274 (time 1788444954.04 total 10.02 last 127.17) - /unlink 2506 (time 1788444964.04 total 20.02 last 123.20) - /unlink 4063 (time 1788444974.04 total 30.02 last 155.60) total: 4500 /unlink in 33.95 seconds: 132.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 917 (time 1788444992.90 total 10.00 last 91.70) - open/close 1824 (time 1788445002.90 total 20.00 last 90.67) - open/close 2671 (time 1788445012.92 total 30.02 last 84.58) - open/close 3443 (time 1788445022.92 total 40.02 last 77.14) - open/close 4285 (time 1788445032.93 total 50.03 last 84.14) total: 4500 open/close in 52.83 seconds: 85.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1169 (time 1788445050.83 total 10.00 last 116.87) - /unlink 2384 (time 1788445060.84 total 20.01 last 121.45) - /unlink 3892 (time 1788445070.84 total 30.01 last 150.78) total: 4500 /unlink in 33.96 seconds: 132.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 794 (time 1788445089.38 total 10.01 last 79.34) - open/close 1550 (time 1788445099.40 total 20.02 last 75.51) - open/close 2458 (time 1788445109.40 total 30.02 last 90.78) - open/close 3265 (time 1788445119.40 total 40.03 last 80.65) - open/close 4091 (time 1788445129.41 total 50.03 last 82.56) total: 4500 open/close in 54.94 seconds: 81.90 ops/second - /unlink 1482 (time 1788445148.66 total 10.00 last 148.16) - /unlink 3028 (time 1788445158.67 total 20.01 last 154.49) total: 4500 /unlink in 29.99 seconds: 150.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 700 (time 1788445182.87 total 10.01 last 69.92) - open/close 1383 (time 1788445192.88 total 20.02 last 68.21) - open/close 2204 (time 1788445202.89 total 30.04 last 81.99) - open/close 3058 (time 1788445212.90 total 40.04 last 85.37) - open/close 3894 (time 1788445222.90 total 50.05 last 83.55) total: 4500 open/close in 58.43 seconds: 77.01 ops/second - /unlink 1432 (time 1788445245.97 total 10.00 last 143.20) - /unlink 2798 (time 1788445255.97 total 20.00 last 136.59) - /unlink 4294 (time 1788445265.97 total 30.01 last 149.51) total: 4500 /unlink in 31.34 seconds: 143.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 822 (time 1788445281.11 total 10.02 last 82.03) - open/close 1606 (time 1788445291.12 total 20.03 last 78.31) - open/close 2075 (time 1788445301.16 total 30.07 last 46.73) - open/close 2713 (time 1788445311.16 total 40.07 last 63.77) - open/close 3555 (time 1788445321.17 total 50.08 last 84.11) - open/close 4494 (time 1788445331.17 total 60.09 last 93.88) total: 4500 open/close in 60.14 seconds: 74.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1528 (time 1788445346.02 total 10.00 last 152.76) - /unlink 2988 (time 1788445356.02 total 20.00 last 145.97) - /unlink 4351 (time 1788445366.03 total 30.02 last 136.10) total: 4500 /unlink in 31.98 seconds: 140.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 740 (time 1788445382.78 total 10.01 last 73.94) - open/close 1316 (time 1788445392.80 total 20.02 last 57.51) - open/close 1959 (time 1788445402.80 total 30.03 last 64.28) - open/close 2749 (time 1788445412.80 total 40.03 last 78.95) - open/close 3581 (time 1788445422.81 total 50.03 last 83.18) - open/close 4419 (time 1788445432.81 total 60.04 last 83.79) total: 4500 open/close in 61.02 seconds: 73.74 ops/second - /unlink 1504 (time 1788445448.61 total 10.00 last 150.37) - /unlink 2755 (time 1788445458.61 total 20.00 last 125.08) - /unlink 3935 (time 1788445468.61 total 30.01 last 117.99) total: 4500 /unlink in 33.83 seconds: 133.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 781 (time 1788445487.57 total 10.01 last 78.02) - open/close 1618 (time 1788445497.58 total 20.02 last 83.63) - open/close 2383 (time 1788445507.60 total 30.04 last 76.34) - open/close 3156 (time 1788445517.61 total 40.05 last 77.22) - open/close 3976 (time 1788445527.61 total 50.05 last 81.99) total: 4500 open/close in 56.46 seconds: 79.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1346 (time 1788445548.19 total 10.00 last 134.59) - /unlink 2930 (time 1788445558.19 total 20.00 last 158.39) - /unlink 4361 (time 1788445568.20 total 30.01 last 143.03) total: 4500 /unlink in 31.48 seconds: 142.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 668 (time 1788445587.85 total 10.02 last 66.65) - open/close 1178 (time 1788445597.85 total 20.03 last 50.97) - open/close 1751 (time 1788445607.86 total 30.04 last 57.26) - open/close 2555 (time 1788445617.86 total 40.04 last 80.38) - open/close 3467 (time 1788445627.87 total 50.04 last 91.16) - open/close 4255 (time 1788445637.87 total 60.05 last 78.78) total: 4500 open/close in 64.37 seconds: 69.91 ops/second - /unlink 1455 (time 1788445657.00 total 10.02 last 145.15) - /unlink 2956 (time 1788445667.01 total 20.03 last 149.96) total: 4500 /unlink in 29.29 seconds: 153.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 920 (time 1788445691.19 total 10.00 last 91.96) - open/close 1773 (time 1788445701.20 total 20.02 last 85.16) - open/close 2482 (time 1788445711.21 total 30.03 last 70.85) - open/close 3356 (time 1788445721.22 total 40.04 last 87.32) - open/close 4068 (time 1788445731.23 total 50.04 last 71.16) total: 4500 open/close in 54.97 seconds: 81.86 ops/second - /unlink 1355 (time 1788445750.77 total 10.00 last 135.46) - /unlink 2735 (time 1788445760.78 total 20.01 last 137.89) - /unlink 4156 (time 1788445770.79 total 30.02 last 141.95) total: 4500 /unlink in 32.19 seconds: 139.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 828 (time 1788445788.14 total 10.01 last 82.73) - open/close 1671 (time 1788445798.15 total 20.02 last 84.19) - open/close 2489 (time 1788445808.16 total 30.02 last 81.78) - open/close 3350 (time 1788445818.16 total 40.03 last 86.04) - open/close 4108 (time 1788445828.17 total 50.04 last 75.75) total: 4500 open/close in 53.95 seconds: 83.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1526 (time 1788445846.74 total 10.00 last 152.59) - /unlink 2874 (time 1788445856.76 total 20.02 last 134.51) - /unlink 4248 (time 1788445866.77 total 30.02 last 137.39) total: 4500 /unlink in 31.90 seconds: 141.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 850 (time 1788445883.91 total 10.01 last 84.89) - open/close 1742 (time 1788445893.92 total 20.02 last 89.16) - open/close 2706 (time 1788445903.92 total 30.02 last 96.36) - open/close 3643 (time 1788445913.92 total 40.02 last 93.67) total: 4500 open/close in 49.81 seconds: 90.34 ops/second - /unlink 1583 (time 1788445937.73 total 10.00 last 158.30) - /unlink 3078 (time 1788445947.74 total 20.00 last 149.46) total: 4500 /unlink in 28.97 seconds: 155.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 872 (time 1788445971.65 total 10.00 last 87.19) - open/close 1527 (time 1788445981.66 total 20.02 last 65.41) - open/close 2117 (time 1788445991.70 total 30.05 last 58.79) - open/close 2786 (time 1788446001.71 total 40.06 last 66.83) - open/close 3410 (time 1788446011.72 total 50.07 last 62.34) - open/close 4047 (time 1788446021.73 total 60.09 last 63.59) total: 4500 open/close in 66.85 seconds: 67.31 ops/second - /unlink 1556 (time 1788446043.73 total 10.00 last 155.53) - /unlink 2865 (time 1788446053.73 total 20.01 last 130.89) - /unlink 4250 (time 1788446063.73 total 30.01 last 138.45) total: 4500 /unlink in 32.14 seconds: 139.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 930 (time 1788446082.35 total 10.00 last 92.97) - open/close 1729 (time 1788446092.40 total 20.05 last 79.56) - open/close 2493 (time 1788446102.40 total 30.05 last 76.35) - open/close 3331 (time 1788446112.41 total 40.06 last 83.74) - open/close 4258 (time 1788446122.41 total 50.06 last 92.67) total: 4500 open/close in 54.27 seconds: 82.92 ops/second - /unlink 1798 (time 1788446141.08 total 10.01 last 179.57) - /unlink 3195 (time 1788446151.09 total 20.03 last 139.53) total: 4500 /unlink in 28.78 seconds: 156.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 869 (time 1788446174.76 total 10.00 last 86.90) - open/close 1788 (time 1788446184.77 total 20.00 last 91.87) - open/close 2574 (time 1788446194.77 total 30.01 last 78.58) - open/close 3361 (time 1788446204.77 total 40.01 last 78.68) - open/close 4242 (time 1788446214.78 total 50.02 last 88.00) total: 4500 open/close in 53.13 seconds: 84.69 ops/second - /unlink 1584 (time 1788446231.53 total 10.01 last 158.30) - /unlink 3032 (time 1788446241.53 total 20.01 last 144.67) total: 4500 /unlink in 29.66 seconds: 151.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 912 (time 1788446265.34 total 10.01 last 91.12) - open/close 1807 (time 1788446275.35 total 20.02 last 89.39) - open/close 2701 (time 1788446285.35 total 30.03 last 89.37) - open/close 3581 (time 1788446295.37 total 40.05 last 87.81) - open/close 4499 (time 1788446305.38 total 50.06 last 91.73) total: 4500 open/close in 50.06 seconds: 89.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1547 (time 1788446319.83 total 10.01 last 154.59) - /unlink 3127 (time 1788446329.84 total 20.02 last 157.86) total: 4500 /unlink in 28.71 seconds: 156.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 909 (time 1788446353.60 total 10.01 last 90.81) - open/close 1695 (time 1788446363.61 total 20.02 last 78.53) - open/close 2665 (time 1788446373.61 total 30.03 last 96.93) - open/close 3533 (time 1788446383.62 total 40.03 last 86.76) - open/close 4434 (time 1788446393.64 total 50.05 last 89.93) total: 4500 open/close in 50.62 seconds: 88.90 ops/second - /unlink 1631 (time 1788446408.09 total 10.00 last 163.02) - /unlink 3248 (time 1788446418.09 total 20.01 last 161.65) total: 4500 /unlink in 28.34 seconds: 158.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 824 (time 1788446441.94 total 10.00 last 82.37) - open/close 1506 (time 1788446451.95 total 20.01 last 68.13) - open/close 2263 (time 1788446461.97 total 30.04 last 75.51) - open/close 3258 (time 1788446471.98 total 40.05 last 99.44) - open/close 4146 (time 1788446481.99 total 50.06 last 88.71) total: 4500 open/close in 53.92 seconds: 83.45 ops/second - /unlink 1554 (time 1788446499.86 total 10.00 last 155.33) - /unlink 3055 (time 1788446509.90 total 20.04 last 149.52) total: 4500 /unlink in 29.58 seconds: 152.14 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95396 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules via unload_modules_local on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local oleg140-server: modules unloaded.