== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:21:08 (1790738468) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 739 (time 1790738678.17 total 10.01 last 73.86) - open/close 1709 (time 1790738688.18 total 20.01 last 96.97) - open/close 2430 (time 1790738698.18 total 30.01 last 72.08) - open/close 3364 (time 1790738708.18 total 40.01 last 93.37) - open/close 4219 (time 1790738718.19 total 50.02 last 85.47) total: 4500 open/close in 53.78 seconds: 83.68 ops/second - /unlink 1602 (time 1790738736.76 total 10.00 last 160.17) - /unlink 3114 (time 1790738746.76 total 20.00 last 151.19) total: 4500 /unlink in 29.97 seconds: 150.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 608 (time 1790738770.57 total 10.02 last 60.67) - open/close 1400 (time 1790738780.57 total 20.02 last 79.17) - open/close 2047 (time 1790738790.58 total 30.03 last 64.64) - open/close 2602 (time 1790738800.59 total 40.04 last 55.47) - open/close 3206 (time 1790738810.60 total 50.05 last 60.34) - open/close 3990 (time 1790738820.60 total 60.05 last 78.37) total: 4500 open/close in 66.81 seconds: 67.35 ops/second - /unlink 1532 (time 1790738842.46 total 10.01 last 153.10) - /unlink 2948 (time 1790738852.46 total 20.01 last 141.57) - /unlink 4459 (time 1790738862.46 total 30.01 last 151.06) total: 4500 /unlink in 30.49 seconds: 147.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 745 (time 1790738877.96 total 10.02 last 74.34) - open/close 1489 (time 1790738887.97 total 20.03 last 74.31) - open/close 2236 (time 1790738897.98 total 30.04 last 74.62) - open/close 2897 (time 1790738907.98 total 40.05 last 66.08) - open/close 3630 (time 1790738917.99 total 50.05 last 73.27) total: 4500 open/close in 58.69 seconds: 76.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1533 (time 1790738940.67 total 10.00 last 153.26) - /unlink 3072 (time 1790738950.67 total 20.00 last 153.87) total: 4500 /unlink in 29.82 seconds: 150.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 865 (time 1790738975.70 total 10.01 last 86.46) - open/close 1646 (time 1790738985.71 total 20.01 last 78.04) - open/close 2452 (time 1790738995.75 total 30.06 last 80.24) - open/close 3320 (time 1790739005.75 total 40.06 last 86.77) - open/close 4013 (time 1790739015.75 total 50.06 last 69.30) total: 4500 open/close in 55.10 seconds: 81.68 ops/second - /unlink 1208 (time 1790739035.41 total 10.01 last 120.73) - /unlink 2825 (time 1790739045.41 total 20.01 last 161.63) - /unlink 4125 (time 1790739055.42 total 30.01 last 129.96) total: 4500 /unlink in 32.16 seconds: 139.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 786 (time 1790739072.63 total 10.02 last 78.46) - open/close 1579 (time 1790739082.64 total 20.03 last 79.23) - open/close 2437 (time 1790739092.64 total 30.03 last 85.80) - open/close 3305 (time 1790739102.65 total 40.03 last 86.74) - open/close 4131 (time 1790739112.65 total 50.04 last 82.59) total: 4500 open/close in 54.53 seconds: 82.53 ops/second - /unlink 1709 (time 1790739131.18 total 10.00 last 170.84) - /unlink 3371 (time 1790739141.19 total 20.01 last 166.10) total: 4500 /unlink in 26.89 seconds: 167.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 684 (time 1790739163.79 total 10.01 last 68.32) - open/close 1349 (time 1790739173.80 total 20.02 last 66.46) - open/close 2066 (time 1790739183.80 total 30.02 last 71.65) - open/close 2861 (time 1790739193.82 total 40.04 last 79.40) - open/close 3516 (time 1790739203.82 total 50.04 last 65.48) - open/close 4165 (time 1790739213.83 total 60.05 last 64.80) total: 4500 open/close in 65.16 seconds: 69.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1321 (time 1790739234.32 total 10.00 last 132.09) - /unlink 2837 (time 1790739244.33 total 20.01 last 151.45) - /unlink 4443 (time 1790739254.33 total 30.01 last 160.55) total: 4500 /unlink in 30.63 seconds: 146.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 947 (time 1790739269.67 total 10.01 last 94.65) - open/close 1717 (time 1790739279.68 total 20.02 last 76.92) - open/close 2424 (time 1790739289.69 total 30.02 last 70.67) - open/close 3191 (time 1790739299.69 total 40.02 last 76.67) - open/close 3961 (time 1790739309.69 total 50.02 last 77.00) total: 4500 open/close in 57.20 seconds: 78.67 ops/second - /unlink 1418 (time 1790739332.36 total 10.00 last 141.75) - /unlink 2931 (time 1790739342.37 total 20.01 last 151.16) total: 4500 /unlink in 29.26 seconds: 153.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 797 (time 1790739366.04 total 10.01 last 79.60) - open/close 1635 (time 1790739376.07 total 20.03 last 83.63) - open/close 2582 (time 1790739386.07 total 30.04 last 94.64) - open/close 3443 (time 1790739396.08 total 40.05 last 85.99) - open/close 4293 (time 1790739406.09 total 50.06 last 84.92) total: 4500 open/close in 52.77 seconds: 85.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1600 (time 1790739423.40 total 10.00 last 159.96) - /unlink 3178 (time 1790739433.40 total 20.00 last 157.77) total: 4500 /unlink in 28.79 seconds: 156.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 801 (time 1790739457.58 total 10.00 last 80.09) - open/close 1631 (time 1790739467.58 total 20.01 last 82.96) - open/close 2468 (time 1790739477.59 total 30.02 last 83.63) - open/close 3302 (time 1790739487.59 total 40.02 last 83.38) - open/close 4060 (time 1790739497.62 total 50.04 last 75.62) total: 4500 open/close in 55.73 seconds: 80.74 ops/second - /unlink 1553 (time 1790739518.58 total 10.01 last 155.15) - /unlink 2872 (time 1790739528.58 total 20.01 last 131.85) - /unlink 4490 (time 1790739538.58 total 30.01 last 161.80) total: 4500 /unlink in 30.07 seconds: 149.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1009 (time 1790739552.96 total 10.00 last 100.88) - open/close 1896 (time 1790739562.96 total 20.01 last 88.66) - open/close 2692 (time 1790739572.96 total 30.01 last 79.59) - open/close 3663 (time 1790739582.97 total 40.01 last 97.08) - open/close 4488 (time 1790739592.97 total 50.01 last 82.47) total: 4500 open/close in 50.16 seconds: 89.71 ops/second - /unlink 1560 (time 1790739607.68 total 10.00 last 155.97) - /unlink 2952 (time 1790739617.68 total 20.00 last 139.20) - /unlink 4208 (time 1790739627.69 total 30.01 last 125.53) total: 4500 /unlink in 32.70 seconds: 137.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 654 (time 1790739647.31 total 10.01 last 65.31) - open/close 1430 (time 1790739657.31 total 20.01 last 77.60) - open/close 2204 (time 1790739667.33 total 30.03 last 77.25) - open/close 3101 (time 1790739677.35 total 40.05 last 89.54) - open/close 3783 (time 1790739687.35 total 50.05 last 68.18) total: 4500 open/close in 58.27 seconds: 77.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1625 (time 1790739709.65 total 10.00 last 162.48) - /unlink 3167 (time 1790739719.67 total 20.02 last 153.96) total: 4500 /unlink in 27.92 seconds: 161.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 924 (time 1790739741.94 total 10.02 last 92.22) - open/close 1742 (time 1790739751.95 total 20.03 last 81.75) - open/close 2751 (time 1790739761.97 total 30.04 last 100.72) - open/close 3392 (time 1790739771.97 total 40.05 last 64.05) - open/close 4171 (time 1790739781.98 total 50.06 last 77.82) total: 4500 open/close in 53.60 seconds: 83.95 ops/second - /unlink 1737 (time 1790739800.13 total 10.00 last 173.69) - /unlink 3353 (time 1790739810.13 total 20.00 last 161.57) total: 4500 /unlink in 27.33 seconds: 164.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 825 (time 1790739831.56 total 10.01 last 82.44) - open/close 1675 (time 1790739841.57 total 20.01 last 84.98) - open/close 2449 (time 1790739851.57 total 30.02 last 77.34) - open/close 3202 (time 1790739861.57 total 40.02 last 75.30) - open/close 3933 (time 1790739871.59 total 50.03 last 72.98) total: 4500 open/close in 56.16 seconds: 80.13 ops/second - /unlink 1528 (time 1790739891.68 total 10.00 last 152.75) - /unlink 3212 (time 1790739901.68 total 20.01 last 168.36) total: 4500 /unlink in 28.21 seconds: 159.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 813 (time 1790739925.87 total 10.01 last 81.24) - open/close 1606 (time 1790739935.88 total 20.02 last 79.21) - open/close 2363 (time 1790739945.89 total 30.03 last 75.63) - open/close 3193 (time 1790739955.89 total 40.03 last 82.99) - open/close 4018 (time 1790739965.89 total 50.03 last 82.48) total: 4500 open/close in 56.53 seconds: 79.60 ops/second - /unlink 1730 (time 1790739986.78 total 10.00 last 172.94) - /unlink 3331 (time 1790739996.78 total 20.00 last 160.10) total: 4500 /unlink in 28.93 seconds: 155.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 800 (time 1790740021.55 total 10.01 last 79.91) - open/close 1688 (time 1790740031.56 total 20.03 last 88.64) - open/close 2556 (time 1790740041.57 total 30.03 last 86.76) - open/close 3292 (time 1790740051.57 total 40.04 last 73.56) - open/close 4193 (time 1790740061.57 total 50.04 last 90.10) total: 4500 open/close in 54.31 seconds: 82.86 ops/second - /unlink 1581 (time 1790740080.74 total 10.00 last 158.03) - /unlink 3064 (time 1790740090.75 total 20.01 last 148.25) - /unlink 4309 (time 1790740100.75 total 30.01 last 124.46) total: 4500 /unlink in 31.65 seconds: 142.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 544 (time 1790740118.11 total 10.01 last 54.34) - open/close 1391 (time 1790740128.13 total 20.03 last 84.58) - open/close 2309 (time 1790740138.14 total 30.04 last 91.68) - open/close 3116 (time 1790740148.16 total 40.05 last 80.57) - open/close 4011 (time 1790740158.16 total 50.05 last 89.50) total: 4500 open/close in 55.42 seconds: 81.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1495 (time 1790740177.47 total 10.01 last 149.40) - /unlink 2948 (time 1790740187.48 total 20.01 last 145.28) total: 4500 /unlink in 29.97 seconds: 150.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1022 (time 1790740212.42 total 10.00 last 102.18) - open/close 1805 (time 1790740222.43 total 20.01 last 78.24) - open/close 2642 (time 1790740232.44 total 30.02 last 83.63) - open/close 3534 (time 1790740242.44 total 40.02 last 89.20) - open/close 4303 (time 1790740252.44 total 50.02 last 76.89) total: 4500 open/close in 53.09 seconds: 84.76 ops/second - /unlink 1752 (time 1790740270.09 total 10.00 last 175.12) - /unlink 3192 (time 1790740280.09 total 20.00 last 144.00) total: 4500 /unlink in 28.67 seconds: 156.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 688 (time 1790740304.07 total 10.01 last 68.72) - open/close 1205 (time 1790740314.07 total 20.02 last 51.67) - open/close 1982 (time 1790740324.07 total 30.02 last 77.69) - open/close 2851 (time 1790740334.08 total 40.02 last 86.84) - open/close 3711 (time 1790740344.08 total 50.03 last 85.99) total: 4500 open/close in 57.88 seconds: 77.75 ops/second - /unlink 1875 (time 1790740366.05 total 10.00 last 187.43) - /unlink 3428 (time 1790740376.06 total 20.01 last 155.23) total: 4500 /unlink in 27.78 seconds: 161.99 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95386 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded.