== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:05:11 (1781222711) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg335-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg335-server oleg335-server: oleg335-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg335-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 606 (time 1781222929.00 total 10.02 last 60.47) - open/close 1071 (time 1781222939.01 total 20.03 last 46.47) - open/close 1554 (time 1781222949.01 total 30.03 last 48.30) - open/close 2173 (time 1781222959.03 total 40.04 last 61.81) - open/close 2843 (time 1781222969.03 total 50.05 last 66.99) - open/close 3539 (time 1781222979.03 total 60.05 last 69.56) - open/close 4236 (time 1781222989.04 total 70.06 last 69.66) total: 4500 open/close in 74.11 seconds: 60.72 ops/second - /unlink 1297 (time 1781223007.25 total 10.01 last 129.51) - /unlink 2475 (time 1781223017.25 total 20.02 last 117.77) - /unlink 3752 (time 1781223027.26 total 30.02 last 127.61) total: 4500 /unlink in 36.13 seconds: 124.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 649 (time 1781223048.96 total 10.01 last 64.81) - open/close 1331 (time 1781223058.97 total 20.02 last 68.13) - open/close 2006 (time 1781223068.97 total 30.03 last 67.47) - open/close 2669 (time 1781223078.99 total 40.05 last 66.17) - open/close 3350 (time 1781223089.00 total 50.05 last 68.08) - open/close 4031 (time 1781223099.00 total 60.05 last 68.10) total: 4500 open/close in 66.32 seconds: 67.85 ops/second - /unlink 1250 (time 1781223120.09 total 10.00 last 124.97) - /unlink 2340 (time 1781223130.09 total 20.00 last 109.00) - /unlink 3624 (time 1781223140.09 total 30.00 last 128.37) total: 4500 /unlink in 38.25 seconds: 117.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 609 (time 1781223163.20 total 10.01 last 60.82) - open/close 1325 (time 1781223173.23 total 20.04 last 71.42) - open/close 2052 (time 1781223183.23 total 30.04 last 72.69) - open/close 2667 (time 1781223193.23 total 40.04 last 61.49) - open/close 3330 (time 1781223203.24 total 50.05 last 66.27) - open/close 4037 (time 1781223213.24 total 60.05 last 70.66) total: 4500 open/close in 66.52 seconds: 67.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1187 (time 1781223234.46 total 10.00 last 118.65) - /unlink 2244 (time 1781223244.47 total 20.01 last 105.63) - /unlink 3391 (time 1781223254.47 total 30.01 last 114.69) total: 4500 /unlink in 39.44 seconds: 114.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 656 (time 1781223279.37 total 10.01 last 65.55) - open/close 1376 (time 1781223289.37 total 20.01 last 71.97) - open/close 2052 (time 1781223299.38 total 30.02 last 67.56) - open/close 2765 (time 1781223309.39 total 40.02 last 71.26) - open/close 3463 (time 1781223319.39 total 50.03 last 69.76) - open/close 4140 (time 1781223329.41 total 60.05 last 67.60) total: 4500 open/close in 65.26 seconds: 68.95 ops/second - /unlink 1214 (time 1781223349.43 total 10.00 last 121.39) - /unlink 2510 (time 1781223359.43 total 20.00 last 129.58) - /unlink 3787 (time 1781223369.44 total 30.01 last 127.61) total: 4500 /unlink in 35.48 seconds: 126.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 765 (time 1781223389.32 total 10.01 last 76.41) - open/close 1458 (time 1781223399.33 total 20.03 last 69.19) - open/close 2139 (time 1781223409.35 total 30.05 last 67.97) - open/close 2874 (time 1781223419.37 total 40.06 last 73.39) - open/close 3523 (time 1781223429.38 total 50.08 last 64.81) - open/close 4170 (time 1781223439.39 total 60.08 last 64.66) total: 4500 open/close in 65.25 seconds: 68.97 ops/second - /unlink 1074 (time 1781223459.39 total 10.00 last 107.36) - /unlink 2151 (time 1781223469.39 total 20.01 last 107.68) - /unlink 3261 (time 1781223479.39 total 30.01 last 110.96) - /unlink 4404 (time 1781223489.39 total 40.01 last 114.28) total: 4500 /unlink in 40.93 seconds: 109.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 676 (time 1781223505.77 total 10.02 last 67.46) - open/close 1316 (time 1781223515.78 total 20.02 last 63.99) - open/close 2011 (time 1781223525.78 total 30.03 last 69.48) - open/close 2630 (time 1781223535.78 total 40.03 last 61.88) - open/close 3309 (time 1781223545.81 total 50.06 last 67.71) - open/close 4007 (time 1781223555.83 total 60.07 last 69.68) total: 4500 open/close in 67.30 seconds: 66.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1146 (time 1781223578.12 total 10.01 last 114.53) - /unlink 2383 (time 1781223588.12 total 20.01 last 123.68) - /unlink 3638 (time 1781223598.13 total 30.01 last 125.41) total: 4500 /unlink in 37.37 seconds: 120.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 655 (time 1781223620.65 total 10.01 last 65.47) - open/close 1348 (time 1781223630.65 total 20.01 last 69.29) - open/close 1914 (time 1781223640.65 total 30.01 last 56.58) - open/close 2483 (time 1781223650.68 total 40.03 last 56.77) - open/close 2999 (time 1781223660.69 total 50.05 last 51.51) - open/close 3600 (time 1781223670.70 total 60.06 last 60.03) - open/close 4284 (time 1781223680.71 total 70.06 last 68.37) total: 4500 open/close in 73.54 seconds: 61.19 ops/second - /unlink 1276 (time 1781223699.07 total 10.00 last 127.60) - /unlink 2387 (time 1781223709.08 total 20.00 last 111.09) - /unlink 3618 (time 1781223719.08 total 30.00 last 123.09) total: 4500 /unlink in 37.31 seconds: 120.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 665 (time 1781223741.50 total 10.01 last 66.42) - open/close 1345 (time 1781223751.52 total 20.03 last 67.88) - open/close 2024 (time 1781223761.53 total 30.04 last 67.82) - open/close 2633 (time 1781223771.54 total 40.06 last 60.81) - open/close 3237 (time 1781223781.54 total 50.06 last 60.40) - open/close 3826 (time 1781223791.55 total 60.06 last 58.89) - open/close 4357 (time 1781223801.55 total 70.06 last 53.07) total: 4500 open/close in 71.88 seconds: 62.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1350 (time 1781223817.80 total 10.00 last 134.93) - /unlink 2508 (time 1781223827.80 total 20.01 last 115.78) - /unlink 3697 (time 1781223837.81 total 30.01 last 118.81) total: 4500 /unlink in 36.59 seconds: 123.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 691 (time 1781223858.99 total 10.01 last 69.04) - open/close 1317 (time 1781223869.00 total 20.02 last 62.52) - open/close 1946 (time 1781223879.01 total 30.02 last 62.90) - open/close 2591 (time 1781223889.01 total 40.02 last 64.49) - open/close 3300 (time 1781223899.02 total 50.03 last 70.84) - open/close 3947 (time 1781223909.03 total 60.04 last 64.64) total: 4500 open/close in 68.56 seconds: 65.64 ops/second - /unlink 1268 (time 1781223932.23 total 10.00 last 126.80) - /unlink 2631 (time 1781223942.23 total 20.01 last 136.20) - /unlink 3884 (time 1781223952.24 total 30.01 last 125.26) total: 4500 /unlink in 36.13 seconds: 124.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 651 (time 1781223974.04 total 10.01 last 65.01) - open/close 1404 (time 1781223984.07 total 20.05 last 75.03) - open/close 2165 (time 1781223994.07 total 30.05 last 76.10) - open/close 2782 (time 1781224004.08 total 40.06 last 61.66) - open/close 3421 (time 1781224014.08 total 50.06 last 63.89) - open/close 4032 (time 1781224024.10 total 60.08 last 61.00) total: 4500 open/close in 67.07 seconds: 67.09 ops/second - /unlink 1147 (time 1781224046.56 total 10.00 last 114.67) - /unlink 2318 (time 1781224056.56 total 20.00 last 117.08) - /unlink 3599 (time 1781224066.57 total 30.01 last 127.97) total: 4500 /unlink in 36.85 seconds: 122.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 657 (time 1781224088.53 total 10.02 last 65.57) - open/close 1310 (time 1781224098.55 total 20.03 last 65.22) - open/close 2054 (time 1781224108.55 total 30.04 last 74.38) - open/close 2614 (time 1781224118.56 total 40.04 last 55.96) - open/close 3224 (time 1781224128.56 total 50.05 last 60.95) - open/close 3915 (time 1781224138.57 total 60.06 last 69.05) total: 4500 open/close in 68.53 seconds: 65.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1419 (time 1781224160.94 total 10.00 last 141.84) - /unlink 2414 (time 1781224170.94 total 20.01 last 99.49) - /unlink 3567 (time 1781224180.95 total 30.01 last 115.25) total: 4500 /unlink in 37.69 seconds: 119.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 646 (time 1781224203.99 total 10.02 last 64.45) - open/close 1121 (time 1781224214.00 total 20.04 last 47.43) - open/close 1518 (time 1781224224.02 total 30.06 last 39.63) - open/close 2049 (time 1781224234.03 total 40.07 last 53.03) - open/close 2749 (time 1781224244.04 total 50.07 last 69.96) - open/close 3395 (time 1781224254.05 total 60.08 last 64.55) - open/close 4130 (time 1781224264.06 total 70.10 last 73.38) total: 4500 open/close in 75.02 seconds: 59.98 ops/second - /unlink 1287 (time 1781224283.69 total 10.00 last 128.69) - /unlink 2473 (time 1781224293.69 total 20.01 last 118.54) - /unlink 3754 (time 1781224303.70 total 30.01 last 128.02) total: 4500 /unlink in 35.82 seconds: 125.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 696 (time 1781224324.83 total 10.00 last 69.58) - open/close 1273 (time 1781224334.85 total 20.02 last 57.62) - open/close 1882 (time 1781224344.87 total 30.04 last 60.76) - open/close 2492 (time 1781224354.87 total 40.04 last 61.00) - open/close 3155 (time 1781224364.89 total 50.06 last 66.20) - open/close 3798 (time 1781224374.90 total 60.07 last 64.23) - open/close 4418 (time 1781224384.90 total 70.07 last 61.98) total: 4500 open/close in 71.44 seconds: 62.99 ops/second - /unlink 1229 (time 1781224402.37 total 10.00 last 122.89) - /unlink 2439 (time 1781224412.37 total 20.01 last 120.94) - /unlink 3536 (time 1781224422.39 total 30.02 last 109.52) total: 4500 /unlink in 39.55 seconds: 113.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 622 (time 1781224449.27 total 10.02 last 62.06) - open/close 1203 (time 1781224459.30 total 20.05 last 57.93) - open/close 1695 (time 1781224469.30 total 30.05 last 49.19) - open/close 2356 (time 1781224479.31 total 40.06 last 66.06) - open/close 2954 (time 1781224489.32 total 50.08 last 59.69) - open/close 3613 (time 1781224499.33 total 60.09 last 65.83) - open/close 4269 (time 1781224509.38 total 70.13 last 65.33) total: 4500 open/close in 73.35 seconds: 61.35 ops/second - /unlink 1305 (time 1781224527.40 total 10.01 last 130.43) - /unlink 2567 (time 1781224537.40 total 20.01 last 126.12) - /unlink 3842 (time 1781224547.40 total 30.01 last 127.49) total: 4500 /unlink in 35.06 seconds: 128.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 650 (time 1781224568.46 total 10.02 last 64.89) - open/close 1389 (time 1781224578.47 total 20.03 last 73.83) - open/close 2080 (time 1781224588.49 total 30.05 last 68.96) - open/close 2844 (time 1781224598.50 total 40.06 last 76.29) - open/close 3470 (time 1781224608.51 total 50.07 last 62.54) - open/close 4140 (time 1781224618.52 total 60.08 last 66.94) total: 4500 open/close in 65.12 seconds: 69.11 ops/second - /unlink 1232 (time 1781224638.85 total 10.00 last 123.15) - /unlink 2468 (time 1781224648.85 total 20.01 last 123.58) - /unlink 3692 (time 1781224658.85 total 30.01 last 122.40) total: 4500 /unlink in 36.70 seconds: 122.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 617 (time 1781224681.79 total 10.00 last 61.69) - open/close 1256 (time 1781224691.81 total 20.02 last 63.80) - open/close 1891 (time 1781224701.81 total 30.02 last 63.46) - open/close 2490 (time 1781224711.82 total 40.03 last 59.87) - open/close 3173 (time 1781224721.84 total 50.05 last 68.18) - open/close 3798 (time 1781224731.85 total 60.06 last 62.44) - open/close 4441 (time 1781224741.86 total 70.07 last 64.18) total: 4500 open/close in 70.79 seconds: 63.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1249 (time 1781224757.34 total 10.00 last 124.89) - /unlink 2489 (time 1781224767.34 total 20.00 last 123.96) - /unlink 3874 (time 1781224777.34 total 30.00 last 138.50) total: 4500 /unlink in 35.67 seconds: 126.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 631 (time 1781224797.58 total 10.01 last 63.06) - open/close 1287 (time 1781224807.59 total 20.01 last 65.56) - open/close 1865 (time 1781224817.61 total 30.03 last 57.71) - open/close 2407 (time 1781224827.62 total 40.04 last 54.12) - open/close 3005 (time 1781224837.62 total 50.04 last 59.80) - open/close 3721 (time 1781224847.65 total 60.07 last 71.41) - open/close 4347 (time 1781224857.69 total 70.11 last 62.34) total: 4500 open/close in 72.66 seconds: 61.93 ops/second - /unlink 1020 (time 1781224876.16 total 10.00 last 101.99) - /unlink 1973 (time 1781224886.17 total 20.01 last 95.24) - /unlink 3052 (time 1781224896.17 total 30.01 last 107.84) - /unlink 4246 (time 1781224906.18 total 40.01 last 119.39) total: 4500 /unlink in 42.14 seconds: 106.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 637 (time 1781224923.59 total 10.02 last 63.59) - open/close 1237 (time 1781224933.59 total 20.02 last 59.99) - open/close 1785 (time 1781224943.59 total 30.02 last 54.78) - open/close 2439 (time 1781224953.62 total 40.04 last 65.26) - open/close 3137 (time 1781224963.62 total 50.05 last 69.76) - open/close 3801 (time 1781224973.63 total 60.05 last 66.37) - open/close 4462 (time 1781224983.63 total 70.06 last 66.04) total: 4500 open/close in 70.58 seconds: 63.76 ops/second - /unlink 1238 (time 1781224998.95 total 10.00 last 123.79) - /unlink 2437 (time 1781225008.95 total 20.01 last 119.85) - /unlink 3628 (time 1781225018.95 total 30.01 last 119.07) total: 4500 /unlink in 37.35 seconds: 120.49 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11195: 94222 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: modules unloaded.