== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:25:29 (1789086329) Stopping clients: oleg621-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg621-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg621-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg621-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg621-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg621-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg621-server oleg621-server: oleg621-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing load_modules_local oleg621-server: Loading modules from /home/green/git/lustre-release/lustre oleg621-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg621-server: detected 4 online CPUs by sysfs oleg621-server: MODOPTS_LIBCFS= oleg621-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg621-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg621-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg621-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 836 (time 1789086518.97 total 10.01 last 83.54) - open/close 1538 (time 1789086528.98 total 20.01 last 70.15) - open/close 2350 (time 1789086538.99 total 30.02 last 81.13) - open/close 3252 (time 1789086549.00 total 40.04 last 90.09) - open/close 4163 (time 1789086559.01 total 50.04 last 91.05) total: 4500 open/close in 54.08 seconds: 83.21 ops/second - /unlink 1433 (time 1789086577.10 total 10.00 last 143.24) - /unlink 2774 (time 1789086587.11 total 20.02 last 133.93) - /unlink 4300 (time 1789086597.11 total 30.02 last 152.56) total: 4500 /unlink in 31.49 seconds: 142.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 752 (time 1789086613.02 total 10.01 last 75.09) - open/close 1563 (time 1789086623.03 total 20.02 last 81.04) - open/close 2275 (time 1789086633.03 total 30.03 last 71.14) - open/close 2959 (time 1789086643.07 total 40.06 last 68.18) - open/close 3708 (time 1789086653.08 total 50.08 last 74.80) - open/close 4392 (time 1789086663.09 total 60.08 last 68.35) total: 4500 open/close in 61.66 seconds: 72.98 ops/second - /unlink 1219 (time 1789086679.15 total 10.00 last 121.85) - /unlink 2270 (time 1789086689.16 total 20.01 last 105.09) - /unlink 3645 (time 1789086699.16 total 30.01 last 137.41) total: 4500 /unlink in 37.34 seconds: 120.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 666 (time 1789086722.03 total 10.00 last 66.58) - open/close 1311 (time 1789086732.05 total 20.02 last 64.40) - open/close 2123 (time 1789086742.08 total 30.05 last 80.93) - open/close 2790 (time 1789086752.09 total 40.06 last 66.67) - open/close 3488 (time 1789086762.09 total 50.06 last 69.80) - open/close 4156 (time 1789086772.10 total 60.07 last 66.73) total: 4500 open/close in 65.77 seconds: 68.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1210 (time 1789086792.89 total 10.01 last 120.86) - /unlink 2365 (time 1789086802.89 total 20.02 last 115.42) - /unlink 3592 (time 1789086812.90 total 30.02 last 122.64) total: 4500 /unlink in 37.05 seconds: 121.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 569 (time 1789086835.60 total 10.00 last 56.87) - open/close 1197 (time 1789086845.60 total 20.01 last 62.77) - open/close 1767 (time 1789086855.61 total 30.01 last 56.97) - open/close 2428 (time 1789086865.61 total 40.02 last 66.09) - open/close 3153 (time 1789086875.61 total 50.02 last 72.48) - open/close 3830 (time 1789086885.61 total 60.02 last 67.68) total: 4500 open/close in 69.25 seconds: 64.99 ops/second - /unlink 1413 (time 1789086909.45 total 10.01 last 141.16) - /unlink 2583 (time 1789086919.46 total 20.02 last 116.92) - /unlink 3824 (time 1789086929.46 total 30.02 last 124.05) total: 4500 /unlink in 35.04 seconds: 128.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 721 (time 1789086949.48 total 10.00 last 72.07) - open/close 1393 (time 1789086959.48 total 20.01 last 67.16) - open/close 2064 (time 1789086969.50 total 30.03 last 66.97) - open/close 2848 (time 1789086979.51 total 40.03 last 78.38) - open/close 3616 (time 1789086989.51 total 50.04 last 76.76) - open/close 4435 (time 1789086999.52 total 60.04 last 81.86) total: 4500 open/close in 61.00 seconds: 73.77 ops/second - /unlink 1296 (time 1789087014.95 total 10.00 last 129.59) - /unlink 2521 (time 1789087024.96 total 20.02 last 122.29) - /unlink 3786 (time 1789087034.97 total 30.02 last 126.45) total: 4500 /unlink in 35.21 seconds: 127.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 766 (time 1789087055.06 total 10.00 last 76.58) - open/close 1429 (time 1789087065.06 total 20.00 last 66.29) - open/close 1997 (time 1789087075.06 total 30.01 last 56.77) - open/close 2576 (time 1789087085.07 total 40.01 last 57.88) - open/close 3198 (time 1789087095.07 total 50.01 last 62.19) - open/close 3916 (time 1789087105.08 total 60.03 last 71.71) total: 4500 open/close in 67.33 seconds: 66.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1268 (time 1789087127.52 total 10.01 last 126.74) - /unlink 2595 (time 1789087137.53 total 20.01 last 132.59) - /unlink 3873 (time 1789087147.54 total 30.02 last 127.67) total: 4500 /unlink in 35.04 seconds: 128.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 753 (time 1789087168.24 total 10.01 last 75.23) - open/close 1477 (time 1789087178.24 total 20.01 last 72.39) - open/close 2178 (time 1789087188.25 total 30.02 last 70.05) - open/close 2892 (time 1789087198.26 total 40.02 last 71.37) - open/close 3588 (time 1789087208.27 total 50.03 last 69.52) - open/close 4237 (time 1789087218.27 total 60.03 last 64.89) total: 4500 open/close in 63.46 seconds: 70.91 ops/second - /unlink 1222 (time 1789087237.00 total 10.00 last 122.16) - /unlink 2425 (time 1789087247.01 total 20.01 last 120.21) - /unlink 3659 (time 1789087257.02 total 30.02 last 123.30) total: 4500 /unlink in 37.00 seconds: 121.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 766 (time 1789087278.76 total 10.01 last 76.55) - open/close 1505 (time 1789087288.77 total 20.01 last 73.84) - open/close 2289 (time 1789087298.77 total 30.02 last 78.35) - open/close 3139 (time 1789087308.78 total 40.02 last 84.99) - open/close 3952 (time 1789087318.78 total 50.03 last 81.28) total: 4500 open/close in 57.41 seconds: 78.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1459 (time 1789087340.34 total 10.00 last 145.84) - /unlink 2991 (time 1789087350.35 total 20.01 last 153.07) - /unlink 4498 (time 1789087360.35 total 30.01 last 150.67) total: 4500 /unlink in 30.03 seconds: 149.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 858 (time 1789087375.03 total 10.00 last 85.80) - open/close 1637 (time 1789087385.05 total 20.02 last 77.73) - open/close 2485 (time 1789087395.05 total 30.03 last 84.77) - open/close 3318 (time 1789087405.05 total 40.03 last 83.29) - open/close 4032 (time 1789087415.06 total 50.03 last 71.34) total: 4500 open/close in 56.12 seconds: 80.18 ops/second - /unlink 1369 (time 1789087436.45 total 10.01 last 136.80) - /unlink 2926 (time 1789087446.45 total 20.01 last 155.59) - /unlink 4412 (time 1789087456.46 total 30.03 last 148.43) total: 4500 /unlink in 30.78 seconds: 146.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 844 (time 1789087473.05 total 10.00 last 84.36) - open/close 1621 (time 1789087483.09 total 20.04 last 77.43) - open/close 2455 (time 1789087493.09 total 30.04 last 83.36) - open/close 3302 (time 1789087503.09 total 40.05 last 84.69) - open/close 4157 (time 1789087513.11 total 50.06 last 85.40) total: 4500 open/close in 53.86 seconds: 83.54 ops/second - /unlink 1555 (time 1789087531.47 total 10.00 last 155.45) - /unlink 3078 (time 1789087541.47 total 20.01 last 152.25) - /unlink 4165 (time 1789087551.48 total 30.01 last 108.66) total: 4500 /unlink in 32.26 seconds: 139.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 699 (time 1789087570.16 total 10.00 last 69.90) - open/close 1440 (time 1789087580.17 total 20.01 last 74.04) - open/close 2190 (time 1789087590.17 total 30.01 last 74.98) - open/close 2979 (time 1789087600.18 total 40.02 last 78.83) - open/close 3863 (time 1789087610.20 total 50.04 last 88.24) total: 4500 open/close in 57.43 seconds: 78.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1660 (time 1789087631.82 total 10.00 last 165.94) - /unlink 3237 (time 1789087641.82 total 20.01 last 157.65) total: 4500 /unlink in 28.60 seconds: 157.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 853 (time 1789087664.90 total 10.00 last 85.28) - open/close 1684 (time 1789087674.91 total 20.01 last 83.03) - open/close 2568 (time 1789087684.91 total 30.01 last 88.39) - open/close 3475 (time 1789087694.92 total 40.02 last 90.64) - open/close 4364 (time 1789087704.93 total 50.03 last 88.78) total: 4500 open/close in 51.57 seconds: 87.27 ops/second - /unlink 1570 (time 1789087720.95 total 10.01 last 156.90) - /unlink 3324 (time 1789087730.95 total 20.01 last 175.38) total: 4500 /unlink in 28.08 seconds: 160.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 827 (time 1789087754.79 total 10.00 last 82.69) - open/close 1648 (time 1789087764.80 total 20.01 last 82.04) - open/close 2550 (time 1789087774.81 total 30.02 last 90.12) - open/close 3347 (time 1789087784.81 total 40.02 last 79.67) - open/close 4161 (time 1789087794.82 total 50.04 last 81.29) total: 4500 open/close in 54.05 seconds: 83.26 ops/second - /unlink 1597 (time 1789087812.97 total 10.00 last 159.64) - /unlink 3065 (time 1789087822.98 total 20.01 last 146.75) total: 4500 /unlink in 28.92 seconds: 155.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 820 (time 1789087845.78 total 10.00 last 81.97) - open/close 1704 (time 1789087855.78 total 20.01 last 88.39) - open/close 2520 (time 1789087865.78 total 30.01 last 81.59) - open/close 3373 (time 1789087875.78 total 40.01 last 85.29) - open/close 4258 (time 1789087885.80 total 50.03 last 88.35) total: 4500 open/close in 53.24 seconds: 84.52 ops/second - /unlink 1635 (time 1789087903.30 total 10.00 last 163.45) - /unlink 3197 (time 1789087913.30 total 20.01 last 156.17) total: 4500 /unlink in 28.27 seconds: 159.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 848 (time 1789087936.31 total 10.00 last 84.77) - open/close 1700 (time 1789087946.32 total 20.02 last 85.10) - open/close 2539 (time 1789087956.33 total 30.03 last 83.82) - open/close 3373 (time 1789087966.33 total 40.03 last 83.37) - open/close 4320 (time 1789087976.34 total 50.04 last 94.63) total: 4500 open/close in 51.65 seconds: 87.12 ops/second - /unlink 1637 (time 1789087991.85 total 10.00 last 163.68) - /unlink 3261 (time 1789088001.86 total 20.01 last 162.33) total: 4500 /unlink in 28.24 seconds: 159.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 850 (time 1789088025.51 total 10.00 last 84.99) - open/close 1680 (time 1789088035.52 total 20.01 last 82.90) - open/close 2527 (time 1789088045.54 total 30.02 last 84.61) - open/close 3450 (time 1789088055.55 total 40.04 last 92.15) - open/close 4347 (time 1789088065.56 total 50.05 last 89.62) total: 4500 open/close in 52.00 seconds: 86.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1662 (time 1789088081.52 total 10.01 last 166.11) - /unlink 3203 (time 1789088091.52 total 20.01 last 154.08) total: 4500 /unlink in 28.35 seconds: 158.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 899 (time 1789088114.32 total 10.00 last 89.87) - open/close 1716 (time 1789088124.33 total 20.01 last 81.62) - open/close 2604 (time 1789088134.34 total 30.02 last 88.72) - open/close 3488 (time 1789088144.34 total 40.02 last 88.39) - open/close 4380 (time 1789088154.34 total 50.02 last 89.20) total: 4500 open/close in 51.30 seconds: 87.71 ops/second - /unlink 1580 (time 1789088170.39 total 10.01 last 157.89) - /unlink 3016 (time 1789088180.40 total 20.01 last 143.59) total: 4500 /unlink in 29.60 seconds: 152.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 776 (time 1789088204.42 total 10.00 last 77.58) - open/close 1581 (time 1789088214.42 total 20.01 last 80.48) - open/close 2445 (time 1789088224.43 total 30.02 last 86.31) - open/close 3264 (time 1789088234.44 total 40.03 last 81.82) - open/close 4002 (time 1789088244.44 total 50.03 last 73.77) total: 4500 open/close in 55.78 seconds: 80.68 ops/second - /unlink 1536 (time 1789088264.19 total 10.00 last 153.57) - /unlink 3149 (time 1789088274.20 total 20.00 last 161.29) total: 4500 /unlink in 28.80 seconds: 156.23 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95242 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg621-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg621-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg621-server stop mds service on oleg621-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg621-server stop mds service on oleg621-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg621-server unloading modules via unload_modules_local on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing unload_modules_local oleg621-server: modules unloaded.