== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:07:56 (1786723676) Stopping clients: oleg607-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg607-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg607-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg607-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg607-server oleg607-server: oleg607-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing load_modules_local oleg607-server: Loading modules from /home/green/git/lustre-release/lustre oleg607-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg607-server: detected 4 online CPUs by sysfs oleg607-server: MODOPTS_LIBCFS= oleg607-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg607-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg607-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg607-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg607-client.virtnet: -o user_xattr,flock 192.168.206.107@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 604 (time 1786723881.04 total 10.01 last 60.35) - open/close 1285 (time 1786723891.04 total 20.01 last 68.10) - open/close 2005 (time 1786723901.04 total 30.01 last 71.97) - open/close 2737 (time 1786723911.07 total 40.04 last 72.98) - open/close 3394 (time 1786723921.08 total 50.05 last 65.65) - open/close 4008 (time 1786723931.09 total 60.06 last 61.37) total: 4500 open/close in 68.33 seconds: 65.86 ops/second - /unlink 1349 (time 1786723953.90 total 10.01 last 134.77) - /unlink 2630 (time 1786723963.91 total 20.02 last 128.01) - /unlink 3841 (time 1786723973.91 total 30.02 last 121.08) total: 4500 /unlink in 34.71 seconds: 129.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 747 (time 1786723993.04 total 10.00 last 74.68) - open/close 1541 (time 1786724003.05 total 20.01 last 79.35) - open/close 2323 (time 1786724013.05 total 30.01 last 78.18) - open/close 2944 (time 1786724023.06 total 40.02 last 62.06) - open/close 3469 (time 1786724033.08 total 50.04 last 52.41) - open/close 4011 (time 1786724043.08 total 60.04 last 54.19) total: 4500 open/close in 67.16 seconds: 67.01 ops/second - /unlink 1262 (time 1786724065.16 total 10.01 last 126.13) - /unlink 2393 (time 1786724075.17 total 20.01 last 113.06) - /unlink 3590 (time 1786724085.18 total 30.02 last 119.53) total: 4500 /unlink in 39.97 seconds: 112.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 570 (time 1786724111.86 total 10.03 last 56.84) - open/close 1116 (time 1786724121.89 total 20.06 last 54.45) - open/close 1721 (time 1786724131.89 total 30.06 last 60.47) - open/close 2252 (time 1786724141.90 total 40.07 last 53.06) - open/close 2834 (time 1786724151.90 total 50.07 last 58.19) - open/close 3453 (time 1786724161.91 total 60.08 last 61.83) - open/close 4028 (time 1786724171.95 total 70.12 last 57.29) total: 4500 open/close in 78.70 seconds: 57.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1203 (time 1786724195.91 total 10.00 last 120.27) - /unlink 2371 (time 1786724205.91 total 20.00 last 116.78) - /unlink 3482 (time 1786724215.92 total 30.01 last 111.07) total: 4500 /unlink in 38.36 seconds: 117.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 599 (time 1786724239.73 total 10.02 last 59.77) - open/close 1278 (time 1786724249.73 total 20.03 last 67.86) - open/close 1937 (time 1786724259.75 total 30.04 last 65.81) - open/close 2650 (time 1786724269.76 total 40.06 last 71.20) - open/close 3348 (time 1786724279.77 total 50.06 last 69.75) - open/close 4066 (time 1786724289.77 total 60.07 last 71.77) total: 4500 open/close in 66.85 seconds: 67.32 ops/second - /unlink 1409 (time 1786724311.06 total 10.00 last 140.84) - /unlink 2838 (time 1786724321.06 total 20.00 last 142.90) - /unlink 4198 (time 1786724331.06 total 30.01 last 135.93) total: 4500 /unlink in 32.18 seconds: 139.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 745 (time 1786724348.06 total 10.02 last 74.37) - open/close 1538 (time 1786724358.07 total 20.03 last 79.23) - open/close 2276 (time 1786724368.07 total 30.03 last 73.75) - open/close 3022 (time 1786724378.08 total 40.04 last 74.56) - open/close 3784 (time 1786724388.08 total 50.04 last 76.16) total: 4500 open/close in 59.87 seconds: 75.17 ops/second - /unlink 1281 (time 1786724412.61 total 10.00 last 128.07) - /unlink 2495 (time 1786724422.61 total 20.01 last 121.36) - /unlink 3900 (time 1786724432.62 total 30.01 last 140.38) total: 4500 /unlink in 34.32 seconds: 131.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 804 (time 1786724451.25 total 10.02 last 80.23) - open/close 1541 (time 1786724461.26 total 20.04 last 73.58) - open/close 2379 (time 1786724471.26 total 30.04 last 83.78) - open/close 3155 (time 1786724481.27 total 40.05 last 77.53) - open/close 3943 (time 1786724491.30 total 50.07 last 78.62) total: 4500 open/close in 57.81 seconds: 77.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1354 (time 1786724513.88 total 10.00 last 135.35) - /unlink 2713 (time 1786724523.88 total 20.01 last 135.81) - /unlink 4084 (time 1786724533.89 total 30.02 last 136.99) total: 4500 /unlink in 32.97 seconds: 136.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 669 (time 1786724551.25 total 10.01 last 66.86) - open/close 1396 (time 1786724561.26 total 20.01 last 72.65) - open/close 2175 (time 1786724571.26 total 30.02 last 77.87) - open/close 2884 (time 1786724581.27 total 40.02 last 70.84) - open/close 3431 (time 1786724591.27 total 50.03 last 54.69) - open/close 3912 (time 1786724601.29 total 60.05 last 47.97) total: 4500 open/close in 69.82 seconds: 64.45 ops/second - /unlink 1392 (time 1786724625.39 total 10.01 last 139.05) - /unlink 2717 (time 1786724635.40 total 20.01 last 132.45) - /unlink 3870 (time 1786724645.41 total 30.03 last 115.10) total: 4500 /unlink in 36.10 seconds: 124.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 532 (time 1786724666.38 total 10.01 last 53.13) - open/close 1098 (time 1786724676.39 total 20.03 last 56.52) - open/close 1835 (time 1786724686.40 total 30.04 last 73.62) - open/close 2589 (time 1786724696.41 total 40.05 last 75.29) - open/close 3287 (time 1786724706.43 total 50.07 last 69.71) - open/close 3943 (time 1786724716.44 total 60.08 last 65.50) total: 4500 open/close in 69.57 seconds: 64.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1263 (time 1786724741.19 total 10.01 last 126.21) - /unlink 2675 (time 1786724751.19 total 20.01 last 141.15) - /unlink 3937 (time 1786724761.20 total 30.02 last 126.14) total: 4500 /unlink in 34.34 seconds: 131.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 691 (time 1786724780.15 total 10.01 last 69.01) - open/close 1222 (time 1786724790.16 total 20.02 last 53.06) - open/close 1828 (time 1786724800.16 total 30.03 last 60.57) - open/close 2546 (time 1786724810.19 total 40.06 last 71.58) - open/close 3298 (time 1786724820.20 total 50.06 last 75.16) - open/close 4021 (time 1786724830.21 total 60.07 last 72.21) total: 4500 open/close in 66.41 seconds: 67.76 ops/second - /unlink 1372 (time 1786724851.16 total 10.01 last 137.12) - /unlink 2691 (time 1786724861.17 total 20.01 last 131.84) - /unlink 4092 (time 1786724871.17 total 30.02 last 140.03) total: 4500 /unlink in 32.87 seconds: 136.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 718 (time 1786724887.93 total 10.02 last 71.64) - open/close 1484 (time 1786724897.94 total 20.03 last 76.54) - open/close 2123 (time 1786724907.96 total 30.05 last 63.75) - open/close 2754 (time 1786724917.97 total 40.06 last 63.06) - open/close 3449 (time 1786724927.97 total 50.06 last 69.48) - open/close 4093 (time 1786724937.97 total 60.07 last 64.36) total: 4500 open/close in 66.21 seconds: 67.96 ops/second - /unlink 1214 (time 1786724958.70 total 10.01 last 121.28) - /unlink 2461 (time 1786724968.70 total 20.01 last 124.64) - /unlink 3688 (time 1786724978.70 total 30.02 last 122.68) total: 4500 /unlink in 36.06 seconds: 124.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 705 (time 1786724998.70 total 10.01 last 70.46) - open/close 1271 (time 1786725008.73 total 20.04 last 56.42) - open/close 1822 (time 1786725018.75 total 30.06 last 55.00) - open/close 2483 (time 1786725028.75 total 40.06 last 66.07) - open/close 3116 (time 1786725038.75 total 50.06 last 63.29) - open/close 3839 (time 1786725048.76 total 60.07 last 72.27) total: 4500 open/close in 69.26 seconds: 64.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1366 (time 1786725072.41 total 10.01 last 136.53) - /unlink 2677 (time 1786725082.42 total 20.01 last 131.03) - /unlink 3838 (time 1786725092.43 total 30.02 last 116.02) total: 4500 /unlink in 36.65 seconds: 122.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 617 (time 1786725114.68 total 10.01 last 61.66) - open/close 1142 (time 1786725124.71 total 20.03 last 52.39) - open/close 1774 (time 1786725134.74 total 30.06 last 62.99) - open/close 2500 (time 1786725144.74 total 40.06 last 72.59) - open/close 3220 (time 1786725154.75 total 50.08 last 71.91) - open/close 3940 (time 1786725164.76 total 60.08 last 71.96) total: 4500 open/close in 67.69 seconds: 66.48 ops/second - /unlink 1438 (time 1786725186.41 total 10.01 last 143.68) - /unlink 2737 (time 1786725196.44 total 20.04 last 129.52) - /unlink 4060 (time 1786725206.44 total 30.04 last 132.25) total: 4500 /unlink in 33.88 seconds: 132.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 667 (time 1786725225.37 total 10.01 last 66.67) - open/close 1405 (time 1786725235.38 total 20.02 last 73.69) - open/close 2134 (time 1786725245.39 total 30.03 last 72.83) - open/close 2827 (time 1786725255.40 total 40.04 last 69.21) - open/close 3552 (time 1786725265.42 total 50.06 last 72.41) - open/close 4222 (time 1786725275.43 total 60.07 last 66.91) total: 4500 open/close in 64.80 seconds: 69.45 ops/second - /unlink 1393 (time 1786725295.10 total 10.00 last 139.24) - /unlink 2676 (time 1786725305.10 total 20.01 last 128.22) - /unlink 4009 (time 1786725315.12 total 30.03 last 133.02) total: 4500 /unlink in 33.58 seconds: 134.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 686 (time 1786725333.39 total 10.01 last 68.56) - open/close 1391 (time 1786725343.40 total 20.02 last 70.42) - open/close 2114 (time 1786725353.40 total 30.02 last 72.28) - open/close 2838 (time 1786725363.40 total 40.02 last 72.38) - open/close 3564 (time 1786725373.40 total 50.02 last 72.60) - open/close 4191 (time 1786725383.41 total 60.03 last 62.66) total: 4500 open/close in 64.04 seconds: 70.26 ops/second - /unlink 1307 (time 1786725401.60 total 10.01 last 130.60) - /unlink 2555 (time 1786725411.61 total 20.02 last 124.68) - /unlink 3943 (time 1786725421.62 total 30.02 last 138.73) total: 4500 /unlink in 34.72 seconds: 129.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 671 (time 1786725441.85 total 10.00 last 67.08) - open/close 1378 (time 1786725451.86 total 20.02 last 70.61) - open/close 2104 (time 1786725461.87 total 30.02 last 72.55) - open/close 2846 (time 1786725471.89 total 40.04 last 74.10) - open/close 3573 (time 1786725481.90 total 50.05 last 72.58) - open/close 4316 (time 1786725491.91 total 60.06 last 74.24) total: 4500 open/close in 62.39 seconds: 72.13 ops/second - /unlink 1351 (time 1786725508.46 total 10.01 last 135.03) - /unlink 2834 (time 1786725518.46 total 20.01 last 148.30) - /unlink 4178 (time 1786725528.47 total 30.01 last 134.34) total: 4500 /unlink in 32.66 seconds: 137.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 718 (time 1786725544.50 total 10.00 last 71.80) - open/close 1436 (time 1786725554.50 total 20.00 last 71.80) - open/close 2215 (time 1786725564.51 total 30.02 last 77.76) - open/close 2965 (time 1786725574.52 total 40.02 last 74.96) - open/close 3733 (time 1786725584.52 total 50.03 last 76.78) - open/close 4479 (time 1786725594.53 total 60.03 last 74.55) total: 4500 open/close in 60.27 seconds: 74.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1425 (time 1786725608.43 total 10.01 last 142.42) - /unlink 2759 (time 1786725618.44 total 20.01 last 133.31) - /unlink 4174 (time 1786725628.44 total 30.01 last 141.48) total: 4500 /unlink in 32.38 seconds: 138.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 759 (time 1786725644.62 total 10.02 last 75.74) - open/close 1510 (time 1786725654.63 total 20.03 last 75.03) - open/close 2285 (time 1786725664.64 total 30.05 last 77.37) - open/close 3032 (time 1786725674.65 total 40.06 last 74.61) - open/close 3719 (time 1786725684.66 total 50.07 last 68.66) - open/close 4481 (time 1786725694.67 total 60.07 last 76.15) total: 4500 open/close in 60.32 seconds: 74.60 ops/second - /unlink 1373 (time 1786725708.97 total 10.00 last 137.25) - /unlink 2676 (time 1786725718.97 total 20.01 last 130.27) - /unlink 4065 (time 1786725728.97 total 30.01 last 138.87) total: 4500 /unlink in 33.28 seconds: 135.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 744 (time 1786725746.38 total 10.01 last 74.30) - open/close 1571 (time 1786725756.39 total 20.02 last 82.63) - open/close 2273 (time 1786725766.41 total 30.04 last 70.10) - open/close 3046 (time 1786725776.42 total 40.05 last 77.20) - open/close 3847 (time 1786725786.43 total 50.05 last 80.05) total: 4500 open/close in 59.08 seconds: 76.17 ops/second - /unlink 1366 (time 1786725809.09 total 10.01 last 136.50) - /unlink 2592 (time 1786725819.09 total 20.01 last 122.58) - /unlink 3896 (time 1786725829.09 total 30.01 last 130.35) total: 4500 /unlink in 34.86 seconds: 129.09 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95339 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg607-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg607-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg607-server unloading modules via unload_modules_local on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing unload_modules_local oleg607-server: modules unloaded.