== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:27:42 (1788467262) Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg434-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg434-server oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: kmemleak disabled oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: MODOPTS_LIBCFS= oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg434-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg434-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg434-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 856 (time 1788467434.25 total 10.01 last 85.48) - open/close 1631 (time 1788467444.26 total 20.02 last 77.44) - open/close 2436 (time 1788467454.27 total 30.03 last 80.44) - open/close 3284 (time 1788467464.28 total 40.04 last 84.70) - open/close 4117 (time 1788467474.30 total 50.06 last 83.15) total: 4500 open/close in 54.68 seconds: 82.30 ops/second - /unlink 1548 (time 1788467492.83 total 10.00 last 154.77) - /unlink 3062 (time 1788467502.83 total 20.01 last 151.34) total: 4500 /unlink in 29.31 seconds: 153.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 809 (time 1788467526.11 total 10.02 last 80.73) - open/close 1580 (time 1788467536.12 total 20.03 last 77.02) - open/close 2367 (time 1788467546.13 total 30.04 last 78.65) - open/close 3121 (time 1788467556.14 total 40.05 last 75.32) - open/close 3948 (time 1788467566.15 total 50.05 last 82.65) total: 4500 open/close in 57.71 seconds: 77.97 ops/second - /unlink 1507 (time 1788467587.97 total 10.01 last 150.61) - /unlink 3071 (time 1788467597.97 total 20.01 last 156.31) total: 4500 /unlink in 29.02 seconds: 155.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 821 (time 1788467620.61 total 10.00 last 82.07) - open/close 1616 (time 1788467630.62 total 20.01 last 79.43) - open/close 2450 (time 1788467640.62 total 30.02 last 83.34) - open/close 3294 (time 1788467650.63 total 40.02 last 84.37) - open/close 4163 (time 1788467660.63 total 50.03 last 86.87) total: 4500 open/close in 53.75 seconds: 83.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1709 (time 1788467677.65 total 10.00 last 170.83) - /unlink 3187 (time 1788467687.66 total 20.01 last 147.77) total: 4500 /unlink in 28.37 seconds: 158.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 844 (time 1788467709.48 total 10.01 last 84.33) - open/close 1720 (time 1788467719.48 total 20.01 last 87.60) - open/close 2571 (time 1788467729.49 total 30.02 last 85.02) - open/close 3452 (time 1788467739.50 total 40.03 last 88.03) - open/close 4309 (time 1788467749.50 total 50.03 last 85.69) total: 4500 open/close in 52.28 seconds: 86.07 ops/second - /unlink 1652 (time 1788467765.01 total 10.00 last 165.18) - /unlink 3185 (time 1788467775.02 total 20.01 last 153.23) total: 4500 /unlink in 28.51 seconds: 157.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 831 (time 1788467798.14 total 10.01 last 83.03) - open/close 1516 (time 1788467808.15 total 20.01 last 68.45) - open/close 2274 (time 1788467818.16 total 30.02 last 75.73) - open/close 3033 (time 1788467828.20 total 40.07 last 75.58) - open/close 3814 (time 1788467838.21 total 50.07 last 78.05) total: 4500 open/close in 58.63 seconds: 76.75 ops/second - /unlink 1369 (time 1788467860.85 total 10.00 last 136.85) - /unlink 2891 (time 1788467870.85 total 20.01 last 152.15) - /unlink 4253 (time 1788467880.85 total 30.01 last 136.17) total: 4500 /unlink in 32.02 seconds: 140.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 655 (time 1788467897.63 total 10.00 last 65.48) - open/close 1489 (time 1788467907.64 total 20.01 last 83.38) - open/close 2323 (time 1788467917.64 total 30.01 last 83.38) - open/close 3175 (time 1788467927.65 total 40.02 last 85.08) - open/close 3931 (time 1788467937.66 total 50.03 last 75.55) total: 4500 open/close in 56.88 seconds: 79.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1426 (time 1788467959.36 total 10.00 last 142.55) - /unlink 2919 (time 1788467969.37 total 20.01 last 149.25) - /unlink 4375 (time 1788467979.38 total 30.02 last 145.47) total: 4500 /unlink in 31.22 seconds: 144.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 741 (time 1788467995.09 total 10.01 last 74.06) - open/close 1537 (time 1788468005.09 total 20.01 last 79.55) - open/close 2324 (time 1788468015.10 total 30.02 last 78.67) - open/close 3096 (time 1788468025.10 total 40.02 last 77.17) - open/close 3935 (time 1788468035.10 total 50.02 last 83.88) total: 4500 open/close in 56.24 seconds: 80.02 ops/second - /unlink 1529 (time 1788468054.93 total 10.00 last 152.88) - /unlink 2882 (time 1788468064.93 total 20.00 last 135.26) - /unlink 4345 (time 1788468074.94 total 30.01 last 146.20) total: 4500 /unlink in 31.13 seconds: 144.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 791 (time 1788468090.05 total 10.01 last 79.04) - open/close 1600 (time 1788468100.06 total 20.02 last 80.80) - open/close 2409 (time 1788468110.06 total 30.02 last 80.89) - open/close 3223 (time 1788468120.07 total 40.03 last 81.36) - open/close 3996 (time 1788468130.08 total 50.04 last 77.20) total: 4500 open/close in 58.35 seconds: 77.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1430 (time 1788468153.01 total 10.00 last 142.99) - /unlink 2714 (time 1788468163.01 total 20.01 last 128.31) - /unlink 4106 (time 1788468173.02 total 30.01 last 139.14) total: 4500 /unlink in 32.96 seconds: 136.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 689 (time 1788468190.77 total 10.03 last 68.71) - open/close 1338 (time 1788468200.78 total 20.03 last 64.86) - open/close 2055 (time 1788468210.78 total 30.04 last 71.64) - open/close 2806 (time 1788468220.79 total 40.04 last 75.09) - open/close 3606 (time 1788468230.79 total 50.05 last 79.97) - open/close 4333 (time 1788468240.80 total 60.06 last 72.60) total: 4500 open/close in 62.52 seconds: 71.97 ops/second - /unlink 1425 (time 1788468257.35 total 10.00 last 142.47) - /unlink 2834 (time 1788468267.37 total 20.01 last 140.75) - /unlink 4299 (time 1788468277.37 total 30.01 last 146.50) total: 4500 /unlink in 31.55 seconds: 142.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 735 (time 1788468293.77 total 10.00 last 73.47) - open/close 1407 (time 1788468303.79 total 20.02 last 67.11) - open/close 2061 (time 1788468313.79 total 30.02 last 65.38) - open/close 2802 (time 1788468323.79 total 40.02 last 74.10) - open/close 3431 (time 1788468333.80 total 50.03 last 62.85) - open/close 4006 (time 1788468343.80 total 60.03 last 57.47) total: 4500 open/close in 66.80 seconds: 67.36 ops/second - /unlink 1244 (time 1788468365.53 total 10.00 last 124.35) - /unlink 2568 (time 1788468375.54 total 20.01 last 132.28) - /unlink 3820 (time 1788468385.54 total 30.02 last 125.16) total: 4500 /unlink in 36.14 seconds: 124.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 672 (time 1788468406.42 total 10.01 last 67.15) - open/close 1394 (time 1788468416.43 total 20.01 last 72.15) - open/close 2154 (time 1788468426.44 total 30.02 last 75.95) - open/close 3000 (time 1788468436.44 total 40.02 last 84.57) - open/close 3742 (time 1788468446.44 total 50.03 last 74.19) total: 4500 open/close in 59.43 seconds: 75.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1502 (time 1788468469.99 total 10.01 last 150.12) - /unlink 3052 (time 1788468480.00 total 20.01 last 154.95) - /unlink 4463 (time 1788468490.00 total 30.01 last 141.07) total: 4500 /unlink in 30.25 seconds: 148.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 604 (time 1788468506.42 total 10.02 last 60.28) - open/close 1158 (time 1788468516.43 total 20.04 last 55.31) - open/close 1629 (time 1788468526.45 total 30.05 last 47.01) - open/close 2322 (time 1788468536.47 total 40.08 last 69.16) - open/close 3064 (time 1788468546.48 total 50.08 last 74.17) - open/close 3848 (time 1788468556.48 total 60.08 last 78.37) total: 4500 open/close in 69.76 seconds: 64.51 ops/second - /unlink 1262 (time 1788468580.27 total 10.00 last 126.16) - /unlink 2281 (time 1788468590.28 total 20.01 last 101.84) - /unlink 3431 (time 1788468600.28 total 30.01 last 114.97) total: 4500 /unlink in 39.15 seconds: 114.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 704 (time 1788468625.17 total 10.02 last 70.25) - open/close 1409 (time 1788468635.18 total 20.03 last 70.44) - open/close 2119 (time 1788468645.19 total 30.04 last 70.94) - open/close 2868 (time 1788468655.19 total 40.05 last 74.84) - open/close 3528 (time 1788468665.22 total 50.07 last 65.83) - open/close 4216 (time 1788468675.23 total 60.08 last 68.75) total: 4500 open/close in 63.88 seconds: 70.45 ops/second - /unlink 1139 (time 1788468694.19 total 10.01 last 113.80) - /unlink 2360 (time 1788468704.20 total 20.01 last 122.08) - /unlink 3760 (time 1788468714.20 total 30.01 last 139.99) total: 4500 /unlink in 34.81 seconds: 129.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 657 (time 1788468733.46 total 10.00 last 65.70) - open/close 1488 (time 1788468743.46 total 20.00 last 83.06) - open/close 2259 (time 1788468753.47 total 30.01 last 77.08) - open/close 3027 (time 1788468763.47 total 40.01 last 76.77) - open/close 3773 (time 1788468773.48 total 50.02 last 74.51) total: 4500 open/close in 59.62 seconds: 75.48 ops/second - /unlink 1349 (time 1788468797.87 total 10.00 last 134.86) - /unlink 2431 (time 1788468807.87 total 20.01 last 108.17) - /unlink 3749 (time 1788468817.87 total 30.01 last 131.75) total: 4500 /unlink in 34.86 seconds: 129.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 711 (time 1788468837.78 total 10.01 last 71.01) - open/close 1350 (time 1788468847.79 total 20.02 last 63.86) - open/close 2069 (time 1788468857.80 total 30.03 last 71.83) - open/close 2757 (time 1788468867.81 total 40.04 last 68.72) - open/close 3506 (time 1788468877.81 total 50.04 last 74.90) - open/close 4162 (time 1788468887.82 total 60.05 last 65.54) total: 4500 open/close in 65.10 seconds: 69.12 ops/second - /unlink 1489 (time 1788468907.16 total 10.00 last 148.87) - /unlink 2821 (time 1788468917.16 total 20.01 last 133.10) - /unlink 4303 (time 1788468927.18 total 30.02 last 147.97) total: 4500 /unlink in 31.30 seconds: 143.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 714 (time 1788468942.82 total 10.01 last 71.30) - open/close 1504 (time 1788468952.83 total 20.02 last 78.96) - open/close 2204 (time 1788468962.83 total 30.02 last 69.99) - open/close 2768 (time 1788468972.85 total 40.05 last 56.24) - open/close 3413 (time 1788468982.87 total 50.07 last 64.38) - open/close 4138 (time 1788468992.90 total 60.09 last 72.33) total: 4500 open/close in 65.66 seconds: 68.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1192 (time 1788469013.62 total 10.00 last 119.16) - /unlink 2672 (time 1788469023.62 total 20.01 last 147.97) - /unlink 4223 (time 1788469033.63 total 30.01 last 155.00) total: 4500 /unlink in 31.93 seconds: 140.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 765 (time 1788469050.53 total 10.00 last 76.48) - open/close 1305 (time 1788469060.55 total 20.02 last 53.89) - open/close 1822 (time 1788469070.56 total 30.03 last 51.66) - open/close 2469 (time 1788469080.56 total 40.03 last 64.70) - open/close 3118 (time 1788469090.58 total 50.05 last 64.79) - open/close 3692 (time 1788469100.60 total 60.07 last 57.27) - open/close 4269 (time 1788469110.61 total 70.07 last 57.67) total: 4500 open/close in 73.33 seconds: 61.36 ops/second - /unlink 1105 (time 1788469129.23 total 10.01 last 110.42) - /unlink 2423 (time 1788469139.23 total 20.01 last 131.79) - /unlink 3617 (time 1788469149.23 total 30.01 last 119.32) total: 4500 /unlink in 39.33 seconds: 114.42 ops/second conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7524:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11725:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7901:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7964:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7763:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11731:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1788469173.log kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11351: 95177 Terminated __test_135_file_thread "$service" rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg434-server.1788469173.log.95Zhva" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg434-server.1788469173.log.5o0yBk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95173 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules via unload_modules_local on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local oleg434-server: kmemleak disabled modules unloaded.