== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:16:32 (1787710592) Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg442-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg442-server oleg442-server: oleg442-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg442-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg442-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 687 (time 1787710814.03 total 10.01 last 68.65) - open/close 1337 (time 1787710824.04 total 20.02 last 64.93) - open/close 1991 (time 1787710834.05 total 30.03 last 65.33) - open/close 2589 (time 1787710844.06 total 40.05 last 59.70) - open/close 3273 (time 1787710854.08 total 50.06 last 68.33) - open/close 3989 (time 1787710864.08 total 60.06 last 71.56) total: 4500 open/close in 68.02 seconds: 66.16 ops/second - /unlink 1316 (time 1787710886.51 total 10.00 last 131.54) - /unlink 2543 (time 1787710896.51 total 20.00 last 122.69) - /unlink 3725 (time 1787710906.52 total 30.01 last 118.17) total: 4500 /unlink in 36.55 seconds: 123.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 523 (time 1787710927.19 total 10.00 last 52.29) - open/close 1105 (time 1787710937.22 total 20.03 last 58.02) - open/close 1786 (time 1787710947.23 total 30.04 last 68.10) - open/close 2556 (time 1787710957.23 total 40.04 last 76.94) - open/close 3209 (time 1787710967.24 total 50.05 last 65.24) - open/close 3851 (time 1787710977.25 total 60.06 last 64.16) - open/close 4300 (time 1787710987.26 total 70.07 last 44.86) total: 4500 open/close in 75.13 seconds: 59.89 ops/second - /unlink 1234 (time 1787711008.21 total 10.01 last 123.34) - /unlink 2448 (time 1787711018.21 total 20.01 last 121.39) - /unlink 3621 (time 1787711028.21 total 30.01 last 117.28) total: 4500 /unlink in 37.49 seconds: 120.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 704 (time 1787711050.33 total 10.01 last 70.36) - open/close 1327 (time 1787711060.34 total 20.02 last 62.23) - open/close 1958 (time 1787711070.35 total 30.03 last 63.01) - open/close 2649 (time 1787711080.36 total 40.03 last 69.08) - open/close 3337 (time 1787711090.36 total 50.04 last 68.78) - open/close 3998 (time 1787711100.36 total 60.04 last 66.08) total: 4500 open/close in 68.18 seconds: 66.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1025 (time 1787711124.13 total 10.01 last 102.45) - /unlink 2145 (time 1787711134.14 total 20.01 last 111.92) - /unlink 3302 (time 1787711144.14 total 30.02 last 115.65) total: 4500 /unlink in 39.59 seconds: 113.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 631 (time 1787711168.44 total 10.00 last 63.07) - open/close 1263 (time 1787711178.45 total 20.02 last 63.13) - open/close 1836 (time 1787711188.46 total 30.02 last 57.25) - open/close 2456 (time 1787711198.47 total 40.03 last 61.95) - open/close 3154 (time 1787711208.47 total 50.03 last 69.79) - open/close 3854 (time 1787711218.48 total 60.04 last 69.94) total: 4500 open/close in 68.98 seconds: 65.24 ops/second - /unlink 1277 (time 1787711241.44 total 10.01 last 127.63) - /unlink 2576 (time 1787711251.45 total 20.01 last 129.84) - /unlink 3868 (time 1787711261.45 total 30.01 last 129.19) total: 4500 /unlink in 35.03 seconds: 128.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 627 (time 1787711281.68 total 10.01 last 62.61) - open/close 1253 (time 1787711291.68 total 20.02 last 62.58) - open/close 1897 (time 1787711301.69 total 30.02 last 64.38) - open/close 2588 (time 1787711311.69 total 40.02 last 69.07) - open/close 3264 (time 1787711321.70 total 50.03 last 67.55) - open/close 3950 (time 1787711331.71 total 60.05 last 68.50) total: 4500 open/close in 68.48 seconds: 65.71 ops/second - /unlink 1222 (time 1787711354.58 total 10.01 last 122.11) - /unlink 2470 (time 1787711364.58 total 20.01 last 124.79) - /unlink 3639 (time 1787711374.58 total 30.01 last 116.84) total: 4500 /unlink in 37.33 seconds: 120.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 623 (time 1787711397.53 total 10.00 last 62.28) - open/close 1258 (time 1787711407.53 total 20.01 last 63.48) - open/close 1877 (time 1787711417.54 total 30.01 last 61.88) - open/close 2545 (time 1787711427.54 total 40.01 last 66.79) - open/close 3192 (time 1787711437.54 total 50.02 last 64.66) - open/close 3836 (time 1787711447.56 total 60.03 last 64.31) total: 4500 open/close in 69.83 seconds: 64.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1304 (time 1787711472.27 total 10.00 last 130.35) - /unlink 2558 (time 1787711482.27 total 20.01 last 125.37) - /unlink 3789 (time 1787711492.28 total 30.01 last 123.04) total: 4500 /unlink in 37.38 seconds: 120.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 628 (time 1787711515.17 total 10.00 last 62.79) - open/close 1217 (time 1787711525.17 total 20.00 last 58.89) - open/close 1763 (time 1787711535.21 total 30.04 last 54.40) - open/close 2195 (time 1787711545.22 total 40.05 last 43.15) - open/close 2779 (time 1787711555.24 total 50.07 last 58.31) - open/close 3317 (time 1787711565.24 total 60.07 last 53.77) - open/close 3905 (time 1787711575.26 total 70.09 last 58.72) total: 4500 open/close in 79.75 seconds: 56.42 ops/second - /unlink 1204 (time 1787711600.46 total 10.01 last 120.27) - /unlink 2290 (time 1787711610.47 total 20.01 last 108.58) - /unlink 3441 (time 1787711620.47 total 30.02 last 115.03) total: 4500 /unlink in 38.79 seconds: 116.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 612 (time 1787711644.44 total 10.01 last 61.15) - open/close 1305 (time 1787711654.44 total 20.01 last 69.29) - open/close 2029 (time 1787711664.44 total 30.01 last 72.36) - open/close 2740 (time 1787711674.46 total 40.03 last 71.00) - open/close 3321 (time 1787711684.46 total 50.03 last 58.10) - open/close 3993 (time 1787711694.46 total 60.03 last 67.19) total: 4500 open/close in 67.20 seconds: 66.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1315 (time 1787711716.49 total 10.00 last 131.48) - /unlink 2481 (time 1787711726.49 total 20.00 last 116.56) - /unlink 3633 (time 1787711736.49 total 30.01 last 115.20) total: 4500 /unlink in 37.74 seconds: 119.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 687 (time 1787711759.28 total 10.01 last 68.60) - open/close 1313 (time 1787711769.29 total 20.02 last 62.54) - open/close 1941 (time 1787711779.29 total 30.03 last 62.79) - open/close 2540 (time 1787711789.29 total 40.03 last 59.90) - open/close 3240 (time 1787711799.31 total 50.05 last 69.87) - open/close 3996 (time 1787711809.31 total 60.05 last 75.58) total: 4500 open/close in 67.72 seconds: 66.45 ops/second - /unlink 1231 (time 1787711831.81 total 10.01 last 123.00) - /unlink 2290 (time 1787711841.81 total 20.01 last 105.86) - /unlink 3499 (time 1787711851.81 total 30.01 last 120.88) total: 4500 /unlink in 37.96 seconds: 118.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 673 (time 1787711875.91 total 10.00 last 67.30) - open/close 1358 (time 1787711885.91 total 20.00 last 68.49) - open/close 2039 (time 1787711895.92 total 30.01 last 68.03) - open/close 2649 (time 1787711905.94 total 40.03 last 60.92) - open/close 3270 (time 1787711915.95 total 50.04 last 62.03) - open/close 3970 (time 1787711925.96 total 60.05 last 69.93) total: 4500 open/close in 67.59 seconds: 66.58 ops/second - /unlink 1322 (time 1787711948.17 total 10.00 last 132.18) - /unlink 2642 (time 1787711958.18 total 20.01 last 131.86) - /unlink 3692 (time 1787711968.18 total 30.01 last 104.97) total: 4500 /unlink in 37.54 seconds: 119.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 675 (time 1787711990.55 total 10.01 last 67.40) - open/close 1348 (time 1787712000.57 total 20.04 last 67.15) - open/close 2036 (time 1787712010.58 total 30.05 last 68.72) - open/close 2759 (time 1787712020.58 total 40.05 last 72.27) - open/close 3422 (time 1787712030.59 total 50.06 last 66.23) - open/close 4053 (time 1787712040.61 total 60.07 last 63.03) total: 4500 open/close in 66.96 seconds: 67.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1219 (time 1787712062.29 total 10.00 last 121.85) - /unlink 2454 (time 1787712072.29 total 20.01 last 123.40) - /unlink 3631 (time 1787712082.30 total 30.02 last 117.66) total: 4500 /unlink in 37.83 seconds: 118.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 608 (time 1787712106.52 total 10.01 last 60.75) - open/close 1290 (time 1787712116.53 total 20.02 last 68.13) - open/close 1893 (time 1787712126.54 total 30.03 last 60.22) - open/close 2519 (time 1787712136.55 total 40.04 last 62.53) - open/close 3210 (time 1787712146.57 total 50.06 last 68.97) - open/close 3834 (time 1787712156.57 total 60.06 last 62.40) - open/close 4449 (time 1787712166.58 total 70.07 last 61.46) total: 4500 open/close in 70.96 seconds: 63.41 ops/second - /unlink 1146 (time 1787712183.52 total 10.00 last 114.59) - /unlink 2381 (time 1787712193.53 total 20.01 last 123.36) - /unlink 3456 (time 1787712203.55 total 30.03 last 107.29) total: 4500 /unlink in 37.93 seconds: 118.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 661 (time 1787712228.20 total 10.02 last 66.00) - open/close 1374 (time 1787712238.21 total 20.02 last 71.24) - open/close 2023 (time 1787712248.21 total 30.02 last 64.90) - open/close 2687 (time 1787712258.21 total 40.03 last 66.36) - open/close 3326 (time 1787712268.21 total 50.03 last 63.89) - open/close 3964 (time 1787712278.21 total 60.03 last 63.80) total: 4500 open/close in 68.03 seconds: 66.14 ops/second - /unlink 922 (time 1787712303.94 total 10.01 last 92.12) - /unlink 1832 (time 1787712313.95 total 20.02 last 90.91) - /unlink 2631 (time 1787712323.95 total 30.02 last 79.89) - /unlink 3437 (time 1787712333.96 total 40.03 last 80.54) - /unlink 4485 (time 1787712343.96 total 50.03 last 104.75) total: 4500 /unlink in 50.11 seconds: 89.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 647 (time 1787712358.38 total 10.01 last 64.66) - open/close 1342 (time 1787712368.41 total 20.04 last 69.27) - open/close 1978 (time 1787712378.42 total 30.05 last 63.54) - open/close 2569 (time 1787712388.42 total 40.05 last 59.09) - open/close 3204 (time 1787712398.42 total 50.05 last 63.50) - open/close 3871 (time 1787712408.42 total 60.05 last 66.68) total: 4500 open/close in 69.84 seconds: 64.43 ops/second - /unlink 1339 (time 1787712432.66 total 10.00 last 133.83) - /unlink 2659 (time 1787712442.67 total 20.01 last 131.89) - /unlink 3743 (time 1787712452.69 total 30.03 last 108.16) total: 4500 /unlink in 36.30 seconds: 123.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 601 (time 1787712474.95 total 10.01 last 60.04) - open/close 1206 (time 1787712484.96 total 20.01 last 60.47) - open/close 1846 (time 1787712494.96 total 30.02 last 63.95) - open/close 2295 (time 1787712504.97 total 40.03 last 44.87) - open/close 2902 (time 1787712514.98 total 50.03 last 60.67) - open/close 3552 (time 1787712524.99 total 60.05 last 64.92) - open/close 4205 (time 1787712534.99 total 70.05 last 65.27) total: 4500 open/close in 74.34 seconds: 60.53 ops/second - /unlink 1272 (time 1787712553.63 total 10.00 last 127.16) - /unlink 2607 (time 1787712563.64 total 20.00 last 133.49) - /unlink 3915 (time 1787712573.64 total 30.01 last 130.69) total: 4500 /unlink in 34.66 seconds: 129.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 635 (time 1787712593.13 total 10.01 last 63.42) - open/close 1235 (time 1787712603.16 total 20.04 last 59.84) - open/close 1842 (time 1787712613.16 total 30.04 last 60.67) - open/close 2448 (time 1787712623.16 total 40.05 last 60.59) - open/close 3127 (time 1787712633.17 total 50.05 last 67.85) - open/close 3804 (time 1787712643.17 total 60.06 last 67.68) - open/close 4452 (time 1787712653.18 total 70.06 last 64.78) total: 4500 open/close in 70.60 seconds: 63.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1252 (time 1787712667.94 total 10.01 last 125.12) - /unlink 2444 (time 1787712677.94 total 20.01 last 119.20) - /unlink 3701 (time 1787712687.97 total 30.04 last 125.34) total: 4500 /unlink in 36.64 seconds: 122.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 566 (time 1787712710.29 total 10.03 last 56.41) - open/close 1059 (time 1787712720.29 total 20.04 last 49.27) - open/close 1610 (time 1787712730.30 total 30.05 last 55.07) - open/close 2182 (time 1787712740.31 total 40.05 last 57.15) - open/close 2738 (time 1787712750.32 total 50.07 last 55.51) - open/close 3306 (time 1787712760.34 total 60.09 last 56.68) - open/close 3864 (time 1787712770.35 total 70.09 last 55.79) total: 4500 open/close in 79.55 seconds: 56.57 ops/second - /unlink 1156 (time 1787712795.15 total 10.00 last 115.58) - /unlink 2445 (time 1787712805.15 total 20.00 last 128.88) - /unlink 3582 (time 1787712815.15 total 30.01 last 113.68) total: 4500 /unlink in 37.29 seconds: 120.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 591 (time 1787712837.69 total 10.00 last 59.08) - open/close 1243 (time 1787712847.70 total 20.01 last 65.16) - open/close 1888 (time 1787712857.71 total 30.02 last 64.41) - open/close 2550 (time 1787712867.72 total 40.04 last 66.12) - open/close 3165 (time 1787712877.73 total 50.05 last 61.44) - open/close 3774 (time 1787712887.75 total 60.06 last 60.81) - open/close 4418 (time 1787712897.76 total 70.07 last 64.35) total: 4500 open/close in 71.68 seconds: 62.78 ops/second - /unlink 1001 (time 1787712914.56 total 10.01 last 100.00) - /unlink 2091 (time 1787712924.56 total 20.01 last 108.95) - /unlink 3213 (time 1787712934.57 total 30.02 last 112.15) - /unlink 4347 (time 1787712944.57 total 40.02 last 113.34) total: 4500 /unlink in 41.50 seconds: 108.43 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95425 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: modules unloaded.