== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:12:19 (1787364739) Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg232-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server oleg232-server: oleg232-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg232-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg232-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 629 (time 1787364957.42 total 10.01 last 62.85) - open/close 1352 (time 1787364967.43 total 20.02 last 72.21) - open/close 1986 (time 1787364977.45 total 30.03 last 63.32) - open/close 2713 (time 1787364987.45 total 40.04 last 72.69) - open/close 3390 (time 1787364997.45 total 50.04 last 67.68) - open/close 4057 (time 1787365007.48 total 60.07 last 66.48) total: 4500 open/close in 66.93 seconds: 67.24 ops/second - /unlink 1296 (time 1787365029.35 total 10.00 last 129.54) - /unlink 2689 (time 1787365039.36 total 20.01 last 139.22) - /unlink 3951 (time 1787365049.36 total 30.01 last 126.18) total: 4500 /unlink in 34.47 seconds: 130.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 650 (time 1787365068.15 total 10.00 last 64.99) - open/close 1415 (time 1787365078.17 total 20.02 last 76.38) - open/close 2063 (time 1787365088.17 total 30.02 last 64.79) - open/close 2716 (time 1787365098.18 total 40.03 last 65.22) - open/close 3491 (time 1787365108.19 total 50.04 last 77.45) - open/close 4213 (time 1787365118.19 total 60.04 last 72.17) total: 4500 open/close in 64.03 seconds: 70.28 ops/second - /unlink 1369 (time 1787365136.94 total 10.00 last 136.85) - /unlink 2649 (time 1787365146.96 total 20.02 last 127.83) - /unlink 3987 (time 1787365156.96 total 30.02 last 133.76) total: 4500 /unlink in 34.52 seconds: 130.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 609 (time 1787365176.51 total 10.01 last 60.86) - open/close 1374 (time 1787365186.51 total 20.01 last 76.50) - open/close 2072 (time 1787365196.52 total 30.02 last 69.74) - open/close 2825 (time 1787365206.52 total 40.02 last 75.30) - open/close 3639 (time 1787365216.52 total 50.02 last 81.36) - open/close 4375 (time 1787365226.53 total 60.03 last 73.53) total: 4500 open/close in 61.53 seconds: 73.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1154 (time 1787365242.86 total 10.00 last 115.36) - /unlink 2439 (time 1787365252.86 total 20.00 last 128.49) - /unlink 3846 (time 1787365262.86 total 30.01 last 140.65) total: 4500 /unlink in 34.54 seconds: 130.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 647 (time 1787365281.77 total 10.01 last 64.65) - open/close 1236 (time 1787365291.78 total 20.02 last 58.85) - open/close 1931 (time 1787365301.78 total 30.02 last 69.48) - open/close 2673 (time 1787365311.79 total 40.03 last 74.14) - open/close 3431 (time 1787365321.80 total 50.03 last 75.75) - open/close 4279 (time 1787365331.80 total 60.03 last 84.80) total: 4500 open/close in 63.46 seconds: 70.91 ops/second - /unlink 1554 (time 1787365349.66 total 10.00 last 155.38) - /unlink 3005 (time 1787365359.68 total 20.01 last 144.93) - /unlink 4205 (time 1787365369.68 total 30.01 last 120.00) total: 4500 /unlink in 32.67 seconds: 137.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 725 (time 1787365387.15 total 10.00 last 72.48) - open/close 1382 (time 1787365397.15 total 20.01 last 65.67) - open/close 2059 (time 1787365407.17 total 30.03 last 67.54) - open/close 2901 (time 1787365417.19 total 40.05 last 84.06) - open/close 3607 (time 1787365427.20 total 50.06 last 70.51) - open/close 4366 (time 1787365437.21 total 60.07 last 75.82) total: 4500 open/close in 61.90 seconds: 72.70 ops/second - /unlink 1449 (time 1787365453.84 total 10.01 last 144.82) - /unlink 2521 (time 1787365463.84 total 20.01 last 107.17) - /unlink 3831 (time 1787365473.85 total 30.02 last 130.91) total: 4500 /unlink in 36.40 seconds: 123.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 753 (time 1787365496.22 total 10.02 last 75.12) - open/close 1555 (time 1787365506.22 total 20.03 last 80.17) - open/close 2299 (time 1787365516.23 total 30.03 last 74.35) - open/close 3031 (time 1787365526.25 total 40.05 last 73.05) - open/close 3670 (time 1787365536.25 total 50.06 last 63.86) - open/close 4343 (time 1787365546.28 total 60.08 last 67.15) total: 4500 open/close in 61.87 seconds: 72.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1396 (time 1787365562.84 total 10.00 last 139.57) - /unlink 2902 (time 1787365572.84 total 20.01 last 150.52) - /unlink 4314 (time 1787365582.84 total 30.01 last 141.20) total: 4500 /unlink in 31.36 seconds: 143.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 680 (time 1787365598.56 total 10.02 last 67.88) - open/close 1483 (time 1787365608.56 total 20.02 last 80.30) - open/close 2219 (time 1787365618.58 total 30.04 last 73.43) - open/close 2923 (time 1787365628.59 total 40.05 last 70.31) - open/close 3545 (time 1787365638.60 total 50.07 last 62.12) - open/close 4308 (time 1787365648.61 total 60.07 last 76.27) total: 4500 open/close in 62.84 seconds: 71.61 ops/second - /unlink 1393 (time 1787365665.40 total 10.00 last 139.28) - /unlink 2949 (time 1787365675.40 total 20.01 last 155.53) - /unlink 4307 (time 1787365685.41 total 30.01 last 135.72) total: 4500 /unlink in 31.51 seconds: 142.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 862 (time 1787365700.97 total 10.01 last 86.09) - open/close 1700 (time 1787365710.97 total 20.01 last 83.79) - open/close 2524 (time 1787365721.01 total 30.05 last 82.12) - open/close 3297 (time 1787365731.02 total 40.06 last 77.20) - open/close 4080 (time 1787365741.02 total 50.06 last 78.29) total: 4500 open/close in 55.61 seconds: 80.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1492 (time 1787365760.64 total 10.01 last 149.06) - /unlink 2808 (time 1787365770.64 total 20.01 last 131.54) - /unlink 4378 (time 1787365780.65 total 30.02 last 156.96) total: 4500 /unlink in 30.98 seconds: 145.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 737 (time 1787365796.34 total 10.00 last 73.68) - open/close 1450 (time 1787365806.34 total 20.01 last 71.27) - open/close 2241 (time 1787365816.35 total 30.01 last 79.08) - open/close 2974 (time 1787365826.35 total 40.01 last 73.28) - open/close 3674 (time 1787365836.36 total 50.02 last 69.93) - open/close 4207 (time 1787365846.37 total 60.04 last 53.23) total: 4500 open/close in 65.16 seconds: 69.06 ops/second - /unlink 1315 (time 1787365866.63 total 10.00 last 131.45) - /unlink 2483 (time 1787365876.65 total 20.02 last 116.65) - /unlink 4014 (time 1787365886.65 total 30.02 last 153.09) total: 4500 /unlink in 33.53 seconds: 134.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 763 (time 1787365905.02 total 10.00 last 76.30) - open/close 1489 (time 1787365915.03 total 20.01 last 72.54) - open/close 2207 (time 1787365925.04 total 30.02 last 71.75) - open/close 2849 (time 1787365935.04 total 40.02 last 64.17) - open/close 3325 (time 1787365945.06 total 50.04 last 47.50) - open/close 4135 (time 1787365955.09 total 60.07 last 80.76) total: 4500 open/close in 64.90 seconds: 69.34 ops/second - /unlink 1504 (time 1787365974.45 total 10.01 last 150.31) - /unlink 2880 (time 1787365984.46 total 20.01 last 137.55) - /unlink 4320 (time 1787365994.46 total 30.01 last 143.97) total: 4500 /unlink in 31.48 seconds: 142.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 771 (time 1787366010.58 total 10.01 last 77.05) - open/close 1625 (time 1787366020.59 total 20.02 last 85.32) - open/close 2414 (time 1787366030.59 total 30.02 last 78.84) - open/close 3184 (time 1787366040.60 total 40.03 last 76.98) - open/close 3895 (time 1787366050.60 total 50.03 last 71.08) total: 4500 open/close in 57.45 seconds: 78.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1405 (time 1787366072.78 total 10.00 last 140.43) - /unlink 2910 (time 1787366082.79 total 20.01 last 150.38) - /unlink 4351 (time 1787366092.79 total 30.02 last 144.01) total: 4500 /unlink in 31.05 seconds: 144.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 759 (time 1787366108.35 total 10.01 last 75.84) - open/close 1645 (time 1787366118.35 total 20.01 last 88.55) - open/close 2419 (time 1787366128.36 total 30.02 last 77.34) - open/close 3051 (time 1787366138.38 total 40.03 last 63.11) - open/close 3670 (time 1787366148.38 total 50.04 last 61.89) - open/close 4464 (time 1787366158.38 total 60.04 last 79.35) total: 4500 open/close in 60.51 seconds: 74.37 ops/second - /unlink 1331 (time 1787366173.41 total 10.00 last 133.10) - /unlink 2691 (time 1787366183.41 total 20.00 last 135.96) - /unlink 4167 (time 1787366193.41 total 30.01 last 147.56) total: 4500 /unlink in 32.29 seconds: 139.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 831 (time 1787366210.20 total 10.00 last 83.07) - open/close 1688 (time 1787366220.21 total 20.01 last 85.61) - open/close 2404 (time 1787366230.22 total 30.02 last 71.53) - open/close 3220 (time 1787366240.22 total 40.02 last 81.59) - open/close 4127 (time 1787366250.23 total 50.03 last 90.66) total: 4500 open/close in 54.17 seconds: 83.07 ops/second - /unlink 1342 (time 1787366268.79 total 10.00 last 134.18) - /unlink 2806 (time 1787366278.79 total 20.00 last 146.35) - /unlink 4347 (time 1787366288.79 total 30.01 last 154.10) total: 4500 /unlink in 31.22 seconds: 144.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 762 (time 1787366304.10 total 10.01 last 76.15) - open/close 1538 (time 1787366314.10 total 20.01 last 77.55) - open/close 2413 (time 1787366324.12 total 30.03 last 87.36) - open/close 3225 (time 1787366334.12 total 40.03 last 81.17) - open/close 4037 (time 1787366344.12 total 50.03 last 81.20) total: 4500 open/close in 56.56 seconds: 79.56 ops/second - /unlink 1504 (time 1787366365.93 total 10.00 last 150.39) - /unlink 2673 (time 1787366375.94 total 20.02 last 116.69) - /unlink 4008 (time 1787366385.95 total 30.03 last 133.40) total: 4500 /unlink in 34.02 seconds: 132.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 778 (time 1787366404.91 total 10.01 last 77.75) - open/close 1576 (time 1787366414.93 total 20.02 last 79.67) - open/close 2351 (time 1787366424.93 total 30.02 last 77.49) - open/close 3197 (time 1787366434.93 total 40.03 last 84.58) - open/close 3856 (time 1787366444.95 total 50.04 last 65.79) total: 4500 open/close in 58.28 seconds: 77.21 ops/second - /unlink 1404 (time 1787366467.70 total 10.01 last 140.28) - /unlink 2665 (time 1787366477.71 total 20.01 last 126.04) - /unlink 3977 (time 1787366487.72 total 30.02 last 131.10) total: 4500 /unlink in 33.76 seconds: 133.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 694 (time 1787366506.99 total 10.01 last 69.33) - open/close 1315 (time 1787366517.00 total 20.02 last 62.04) - open/close 2092 (time 1787366527.00 total 30.02 last 77.70) - open/close 2899 (time 1787366537.00 total 40.02 last 80.68) - open/close 3646 (time 1787366547.00 total 50.02 last 74.70) - open/close 4406 (time 1787366557.01 total 60.03 last 75.97) total: 4500 open/close in 61.40 seconds: 73.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1512 (time 1787366573.14 total 10.00 last 151.15) - /unlink 2919 (time 1787366583.15 total 20.01 last 140.63) - /unlink 4141 (time 1787366593.16 total 30.02 last 122.07) total: 4500 /unlink in 32.46 seconds: 138.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 624 (time 1787366611.48 total 10.00 last 62.37) - open/close 1276 (time 1787366621.48 total 20.01 last 65.18) - open/close 2109 (time 1787366631.51 total 30.03 last 83.11) - open/close 2799 (time 1787366641.52 total 40.04 last 68.94) - open/close 3496 (time 1787366651.52 total 50.04 last 69.68) - open/close 4198 (time 1787366661.52 total 60.04 last 70.19) total: 4500 open/close in 63.83 seconds: 70.50 ops/second - /unlink 1486 (time 1787366679.32 total 10.00 last 148.55) - /unlink 2793 (time 1787366689.33 total 20.01 last 130.57) - /unlink 4257 (time 1787366699.34 total 30.02 last 146.35) total: 4500 /unlink in 32.20 seconds: 139.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 767 (time 1787366716.26 total 10.00 last 76.69) - open/close 1635 (time 1787366726.27 total 20.01 last 86.77) - open/close 2395 (time 1787366736.27 total 30.01 last 75.98) - open/close 3219 (time 1787366746.27 total 40.01 last 82.36) - open/close 3969 (time 1787366756.28 total 50.02 last 74.98) total: 4500 open/close in 57.04 seconds: 78.89 ops/second - /unlink 1611 (time 1787366777.17 total 10.00 last 161.09) - /unlink 3151 (time 1787366787.18 total 20.01 last 153.85) - /unlink 4483 (time 1787366797.18 total 30.01 last 133.20) total: 4500 /unlink in 30.23 seconds: 148.83 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95435 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local oleg232-server: modules unloaded.