== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:29:15 (1788668955) Stopping clients: oleg645-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg645-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg645-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg645-server oleg645-server: oleg645-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: kmemleak disabled oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg645-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg645-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 796 (time 1788669157.63 total 10.00 last 79.57) - open/close 1530 (time 1788669167.64 total 20.01 last 73.33) - open/close 2294 (time 1788669177.65 total 30.02 last 76.36) - open/close 3083 (time 1788669187.65 total 40.02 last 78.89) - open/close 3851 (time 1788669197.65 total 50.02 last 76.80) total: 4500 open/close in 58.75 seconds: 76.60 ops/second - /unlink 1638 (time 1788669220.94 total 10.00 last 163.77) - /unlink 3225 (time 1788669230.94 total 20.01 last 158.59) total: 4500 /unlink in 29.18 seconds: 154.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 785 (time 1788669255.57 total 10.01 last 78.44) - open/close 1698 (time 1788669265.58 total 20.01 last 91.25) - open/close 2474 (time 1788669275.58 total 30.01 last 77.58) - open/close 3316 (time 1788669285.59 total 40.03 last 84.08) - open/close 4112 (time 1788669295.60 total 50.03 last 79.56) total: 4500 open/close in 54.76 seconds: 82.18 ops/second - /unlink 1442 (time 1788669314.00 total 10.00 last 144.18) - /unlink 2956 (time 1788669324.00 total 20.00 last 151.37) total: 4500 /unlink in 29.98 seconds: 150.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 824 (time 1788669348.53 total 10.01 last 82.35) - open/close 1649 (time 1788669358.54 total 20.02 last 82.39) - open/close 2467 (time 1788669368.55 total 30.03 last 81.74) - open/close 3313 (time 1788669378.55 total 40.03 last 84.59) - open/close 4089 (time 1788669388.55 total 50.03 last 77.58) total: 4500 open/close in 54.78 seconds: 82.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1474 (time 1788669407.56 total 10.00 last 147.37) - /unlink 2939 (time 1788669417.57 total 20.01 last 146.37) - /unlink 4447 (time 1788669427.57 total 30.01 last 150.75) total: 4500 /unlink in 30.49 seconds: 147.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 804 (time 1788669441.50 total 10.02 last 80.27) - open/close 1489 (time 1788669451.51 total 20.03 last 68.42) - open/close 2262 (time 1788669461.51 total 30.03 last 77.29) - open/close 3017 (time 1788669471.51 total 40.03 last 75.50) - open/close 3882 (time 1788669481.51 total 50.03 last 86.48) total: 4500 open/close in 57.89 seconds: 77.73 ops/second - /unlink 1507 (time 1788669503.31 total 10.00 last 150.67) - /unlink 3056 (time 1788669513.31 total 20.00 last 154.87) total: 4500 /unlink in 29.37 seconds: 153.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 916 (time 1788669536.91 total 10.01 last 91.47) - open/close 1685 (time 1788669546.92 total 20.03 last 76.80) - open/close 2564 (time 1788669556.92 total 30.03 last 87.90) - open/close 3373 (time 1788669566.92 total 40.03 last 80.89) - open/close 4185 (time 1788669576.94 total 50.04 last 81.09) total: 4500 open/close in 54.06 seconds: 83.25 ops/second - /unlink 1457 (time 1788669595.53 total 10.01 last 145.60) - /unlink 3078 (time 1788669605.53 total 20.01 last 162.08) total: 4500 /unlink in 29.73 seconds: 151.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 760 (time 1788669628.88 total 10.01 last 75.89) - open/close 1638 (time 1788669638.89 total 20.02 last 87.75) - open/close 2532 (time 1788669648.90 total 30.03 last 89.29) - open/close 3421 (time 1788669658.92 total 40.05 last 88.79) - open/close 4338 (time 1788669668.92 total 50.05 last 91.67) total: 4500 open/close in 51.76 seconds: 86.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1674 (time 1788669684.68 total 10.01 last 167.19) - /unlink 3132 (time 1788669694.68 total 20.02 last 145.71) total: 4500 /unlink in 28.19 seconds: 159.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 784 (time 1788669717.43 total 10.01 last 78.32) - open/close 1718 (time 1788669727.43 total 20.01 last 93.38) - open/close 2604 (time 1788669737.44 total 30.02 last 88.52) - open/close 3516 (time 1788669747.44 total 40.03 last 91.13) - open/close 4409 (time 1788669757.46 total 50.04 last 89.20) total: 4500 open/close in 50.95 seconds: 88.32 ops/second - /unlink 1645 (time 1788669771.52 total 10.00 last 164.45) - /unlink 3337 (time 1788669781.52 total 20.00 last 169.18) total: 4500 /unlink in 26.12 seconds: 172.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 911 (time 1788669801.55 total 10.01 last 91.04) - open/close 1744 (time 1788669811.56 total 20.01 last 83.26) - open/close 2699 (time 1788669821.56 total 30.02 last 95.42) - open/close 3634 (time 1788669831.57 total 40.03 last 93.41) total: 4500 open/close in 49.46 seconds: 90.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1762 (time 1788669854.40 total 10.01 last 176.09) - /unlink 3273 (time 1788669864.40 total 20.01 last 151.08) total: 4500 /unlink in 27.04 seconds: 166.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 908 (time 1788669885.49 total 10.01 last 90.74) - open/close 1808 (time 1788669895.50 total 20.01 last 89.98) - open/close 2752 (time 1788669905.50 total 30.01 last 94.40) - open/close 3603 (time 1788669915.50 total 40.01 last 85.08) - open/close 4324 (time 1788669925.50 total 50.01 last 72.09) total: 4500 open/close in 52.29 seconds: 86.06 ops/second - /unlink 1304 (time 1788669943.07 total 10.00 last 130.37) - /unlink 2817 (time 1788669953.08 total 20.02 last 151.10) total: 4500 /unlink in 30.00 seconds: 149.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 735 (time 1788669978.47 total 10.01 last 73.46) - open/close 1643 (time 1788669988.48 total 20.01 last 90.77) - open/close 2423 (time 1788669998.49 total 30.02 last 77.91) - open/close 3224 (time 1788670008.49 total 40.02 last 80.09) - open/close 4144 (time 1788670018.49 total 50.03 last 91.96) total: 4500 open/close in 53.69 seconds: 83.82 ops/second - /unlink 1751 (time 1788670035.62 total 10.00 last 175.03) - /unlink 3422 (time 1788670045.63 total 20.01 last 166.96) total: 4500 /unlink in 25.97 seconds: 173.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 922 (time 1788670065.13 total 10.01 last 92.11) - open/close 1828 (time 1788670075.13 total 20.01 last 90.60) - open/close 2735 (time 1788670085.13 total 30.02 last 90.65) - open/close 3606 (time 1788670095.14 total 40.02 last 87.10) - open/close 4412 (time 1788670105.14 total 50.02 last 80.54) total: 4500 open/close in 51.29 seconds: 87.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1786 (time 1788670120.35 total 10.00 last 178.53) - /unlink 3525 (time 1788670130.35 total 20.01 last 173.84) total: 4500 /unlink in 25.71 seconds: 175.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 875 (time 1788670149.68 total 10.00 last 87.49) - open/close 1840 (time 1788670159.69 total 20.00 last 96.48) - open/close 2736 (time 1788670169.69 total 30.00 last 89.59) - open/close 3592 (time 1788670179.70 total 40.01 last 85.52) total: 4500 open/close in 49.88 seconds: 90.22 ops/second - /unlink 1715 (time 1788670203.23 total 10.00 last 171.44) - /unlink 3447 (time 1788670213.23 total 20.01 last 173.10) total: 4500 /unlink in 26.40 seconds: 170.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 949 (time 1788670234.02 total 10.00 last 94.86) - open/close 1782 (time 1788670244.03 total 20.01 last 83.21) - open/close 2579 (time 1788670254.03 total 30.02 last 79.69) - open/close 3487 (time 1788670264.04 total 40.02 last 90.77) - open/close 4402 (time 1788670274.04 total 50.02 last 91.46) total: 4500 open/close in 50.90 seconds: 88.41 ops/second - /unlink 1705 (time 1788670288.28 total 10.00 last 170.46) - /unlink 3430 (time 1788670298.29 total 20.02 last 172.25) total: 4500 /unlink in 26.89 seconds: 167.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 834 (time 1788670318.58 total 10.00 last 83.36) - open/close 1716 (time 1788670328.59 total 20.01 last 88.15) - open/close 2599 (time 1788670338.60 total 30.02 last 88.19) - open/close 3481 (time 1788670348.61 total 40.03 last 88.10) - open/close 4373 (time 1788670358.62 total 50.04 last 89.11) total: 4500 open/close in 51.63 seconds: 87.16 ops/second - /unlink 1659 (time 1788670374.21 total 10.01 last 165.74) - /unlink 3386 (time 1788670384.21 total 20.01 last 172.66) total: 4500 /unlink in 26.66 seconds: 168.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 930 (time 1788670404.11 total 10.01 last 92.93) - open/close 1834 (time 1788670414.11 total 20.01 last 90.37) - open/close 2785 (time 1788670424.12 total 30.02 last 95.02) - open/close 3761 (time 1788670434.12 total 40.02 last 97.58) total: 4500 open/close in 48.10 seconds: 93.56 ops/second - /unlink 1838 (time 1788670455.44 total 10.01 last 183.67) - /unlink 3683 (time 1788670465.45 total 20.01 last 184.44) total: 4500 /unlink in 24.27 seconds: 185.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1003 (time 1788670482.61 total 10.00 last 100.28) - open/close 1921 (time 1788670492.61 total 20.01 last 91.77) - open/close 2852 (time 1788670502.62 total 30.02 last 93.01) - open/close 3828 (time 1788670512.63 total 40.02 last 97.53) total: 4500 open/close in 47.54 seconds: 94.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1742 (time 1788670533.86 total 10.00 last 174.20) - /unlink 3465 (time 1788670543.86 total 20.00 last 172.27) total: 4500 /unlink in 26.57 seconds: 169.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 890 (time 1788670563.77 total 10.00 last 88.98) - open/close 1765 (time 1788670573.78 total 20.02 last 87.38) - open/close 2745 (time 1788670583.79 total 30.02 last 97.95) - open/close 3692 (time 1788670593.79 total 40.03 last 94.65) total: 4500 open/close in 48.12 seconds: 93.52 ops/second - /unlink 1797 (time 1788670615.19 total 10.00 last 179.67) - /unlink 3584 (time 1788670625.19 total 20.01 last 178.64) total: 4500 /unlink in 25.06 seconds: 179.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 872 (time 1788670643.28 total 10.01 last 87.14) - open/close 1732 (time 1788670653.29 total 20.01 last 85.93) - open/close 2606 (time 1788670663.29 total 30.02 last 87.37) - open/close 3508 (time 1788670673.29 total 40.02 last 90.18) - open/close 4174 (time 1788670683.31 total 50.04 last 66.50) total: 4500 open/close in 54.97 seconds: 81.87 ops/second - /unlink 1401 (time 1788670703.79 total 10.00 last 140.05) - /unlink 2895 (time 1788670713.79 total 20.01 last 149.33) total: 4500 /unlink in 29.77 seconds: 151.17 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95233 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: kmemleak disabled modules unloaded.