== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:21:41 (1780334501) Stopping clients: oleg355-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg355-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg355-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg355-server oleg355-server: oleg355-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg355-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg355-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg355-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 856 (time 1780334677.41 total 10.02 last 85.43) - open/close 1810 (time 1780334687.41 total 20.02 last 95.37) - open/close 2659 (time 1780334697.41 total 30.02 last 84.88) - open/close 3579 (time 1780334707.41 total 40.02 last 92.00) - open/close 4438 (time 1780334717.41 total 50.03 last 85.88) total: 4500 open/close in 50.94 seconds: 88.34 ops/second - /unlink 1501 (time 1780334732.20 total 10.00 last 150.09) - /unlink 3062 (time 1780334742.21 total 20.00 last 156.04) total: 4500 /unlink in 28.98 seconds: 155.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 711 (time 1780334765.95 total 10.01 last 71.02) - open/close 1455 (time 1780334775.96 total 20.02 last 74.36) - open/close 2237 (time 1780334785.96 total 30.02 last 78.15) - open/close 2999 (time 1780334795.98 total 40.03 last 76.10) - open/close 3823 (time 1780334805.98 total 50.04 last 82.32) total: 4500 open/close in 59.05 seconds: 76.21 ops/second - /unlink 1527 (time 1780334829.87 total 10.00 last 152.63) - /unlink 3068 (time 1780334839.87 total 20.01 last 154.08) total: 4500 /unlink in 29.27 seconds: 153.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 902 (time 1780334862.44 total 10.01 last 90.15) - open/close 1775 (time 1780334872.44 total 20.01 last 87.27) - open/close 2579 (time 1780334882.44 total 30.01 last 80.40) - open/close 3461 (time 1780334892.45 total 40.02 last 88.13) - open/close 4207 (time 1780334902.45 total 50.02 last 74.58) total: 4500 open/close in 53.74 seconds: 83.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1550 (time 1780334919.67 total 10.00 last 155.00) - /unlink 3176 (time 1780334929.68 total 20.01 last 162.52) total: 4500 /unlink in 28.48 seconds: 157.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 901 (time 1780334952.12 total 10.00 last 90.09) - open/close 1748 (time 1780334962.13 total 20.01 last 84.61) - open/close 2526 (time 1780334972.15 total 30.03 last 77.67) - open/close 3353 (time 1780334982.16 total 40.04 last 82.57) - open/close 4185 (time 1780334992.17 total 50.05 last 83.17) total: 4500 open/close in 53.76 seconds: 83.70 ops/second - /unlink 1491 (time 1780335010.22 total 10.00 last 149.06) - /unlink 2911 (time 1780335020.22 total 20.01 last 141.94) - /unlink 4476 (time 1780335030.22 total 30.01 last 156.48) total: 4500 /unlink in 30.22 seconds: 148.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 857 (time 1780335044.05 total 10.01 last 85.64) - open/close 1734 (time 1780335054.06 total 20.01 last 87.64) - open/close 2575 (time 1780335064.08 total 30.03 last 83.96) - open/close 3433 (time 1780335074.08 total 40.03 last 85.79) - open/close 4279 (time 1780335084.08 total 50.03 last 84.59) total: 4500 open/close in 52.69 seconds: 85.41 ops/second - /unlink 1522 (time 1780335100.27 total 10.00 last 152.15) - /unlink 3086 (time 1780335110.27 total 20.00 last 156.39) total: 4500 /unlink in 29.19 seconds: 154.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 713 (time 1780335133.65 total 10.00 last 71.26) - open/close 1579 (time 1780335143.66 total 20.01 last 86.52) - open/close 2362 (time 1780335153.66 total 30.02 last 78.26) - open/close 3208 (time 1780335163.67 total 40.02 last 84.55) - open/close 4044 (time 1780335173.68 total 50.03 last 83.52) total: 4500 open/close in 54.56 seconds: 82.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1513 (time 1780335192.77 total 10.00 last 151.27) - /unlink 2866 (time 1780335202.78 total 20.01 last 135.21) - /unlink 4227 (time 1780335212.78 total 30.01 last 136.06) total: 4500 /unlink in 32.05 seconds: 140.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 834 (time 1780335229.13 total 10.01 last 83.32) - open/close 1685 (time 1780335239.14 total 20.02 last 85.00) - open/close 2427 (time 1780335249.15 total 30.03 last 74.13) - open/close 3282 (time 1780335259.16 total 40.03 last 85.46) - open/close 4078 (time 1780335269.17 total 50.04 last 79.54) total: 4500 open/close in 55.30 seconds: 81.38 ops/second - /unlink 1592 (time 1780335287.74 total 10.00 last 159.15) - /unlink 3050 (time 1780335297.74 total 20.01 last 145.75) - /unlink 4448 (time 1780335307.75 total 30.01 last 139.70) total: 4500 /unlink in 30.46 seconds: 147.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 813 (time 1780335323.44 total 10.00 last 81.28) - open/close 1683 (time 1780335333.45 total 20.01 last 86.93) - open/close 2503 (time 1780335343.45 total 30.01 last 81.99) - open/close 3374 (time 1780335353.45 total 40.01 last 87.08) - open/close 4198 (time 1780335363.46 total 50.02 last 82.36) total: 4500 open/close in 53.36 seconds: 84.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1415 (time 1780335380.47 total 10.00 last 141.43) - /unlink 2882 (time 1780335390.48 total 20.01 last 146.64) - /unlink 4369 (time 1780335400.48 total 30.01 last 148.62) total: 4500 /unlink in 31.27 seconds: 143.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 748 (time 1780335417.36 total 10.01 last 74.73) - open/close 1416 (time 1780335427.38 total 20.02 last 66.71) - open/close 2186 (time 1780335437.39 total 30.03 last 76.94) - open/close 2969 (time 1780335447.40 total 40.04 last 78.22) - open/close 3780 (time 1780335457.41 total 50.06 last 80.96) total: 4500 open/close in 58.78 seconds: 76.55 ops/second - /unlink 1397 (time 1780335479.85 total 10.00 last 139.67) - /unlink 2748 (time 1780335489.85 total 20.01 last 135.04) - /unlink 4288 (time 1780335499.87 total 30.03 last 153.72) total: 4500 /unlink in 32.10 seconds: 140.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 765 (time 1780335516.41 total 10.01 last 76.46) - open/close 1569 (time 1780335526.43 total 20.02 last 80.25) - open/close 2264 (time 1780335536.43 total 30.03 last 69.46) - open/close 3055 (time 1780335546.44 total 40.04 last 79.05) - open/close 3883 (time 1780335556.44 total 50.04 last 82.79) total: 4500 open/close in 57.09 seconds: 78.83 ops/second - /unlink 1384 (time 1780335577.41 total 10.01 last 138.28) - /unlink 2840 (time 1780335587.41 total 20.01 last 145.60) - /unlink 4365 (time 1780335597.42 total 30.02 last 152.28) total: 4500 /unlink in 30.99 seconds: 145.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 838 (time 1780335612.57 total 10.00 last 83.76) - open/close 1634 (time 1780335622.58 total 20.01 last 79.53) - open/close 2497 (time 1780335632.59 total 30.02 last 86.24) - open/close 3303 (time 1780335642.59 total 40.03 last 80.56) - open/close 4082 (time 1780335652.60 total 50.03 last 77.88) total: 4500 open/close in 55.36 seconds: 81.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1379 (time 1780335672.33 total 10.00 last 137.86) - /unlink 2810 (time 1780335682.34 total 20.02 last 142.86) - /unlink 4327 (time 1780335692.34 total 30.02 last 151.66) total: 4500 /unlink in 31.25 seconds: 144.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 812 (time 1780335707.47 total 10.00 last 81.16) - open/close 1633 (time 1780335717.47 total 20.01 last 82.09) - open/close 2382 (time 1780335727.48 total 30.01 last 74.87) - open/close 3191 (time 1780335737.49 total 40.02 last 80.78) - open/close 3934 (time 1780335747.50 total 50.03 last 74.26) total: 4500 open/close in 58.68 seconds: 76.69 ops/second - /unlink 1593 (time 1780335770.43 total 10.01 last 159.18) - /unlink 2989 (time 1780335780.43 total 20.01 last 139.55) - /unlink 4420 (time 1780335790.44 total 30.02 last 143.04) total: 4500 /unlink in 30.83 seconds: 145.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 808 (time 1780335805.43 total 10.00 last 80.78) - open/close 1465 (time 1780335815.44 total 20.00 last 65.69) - open/close 2225 (time 1780335825.44 total 30.01 last 75.97) - open/close 3063 (time 1780335835.44 total 40.01 last 83.77) - open/close 3862 (time 1780335845.45 total 50.01 last 79.88) total: 4500 open/close in 57.43 seconds: 78.36 ops/second - /unlink 1487 (time 1780335867.52 total 10.01 last 148.56) - /unlink 2897 (time 1780335877.53 total 20.02 last 140.82) - /unlink 4401 (time 1780335887.54 total 30.03 last 150.33) total: 4500 /unlink in 30.87 seconds: 145.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 735 (time 1780335903.38 total 10.01 last 73.44) - open/close 1466 (time 1780335913.40 total 20.02 last 73.01) - open/close 2325 (time 1780335923.40 total 30.03 last 85.83) - open/close 3059 (time 1780335933.42 total 40.04 last 73.31) - open/close 3894 (time 1780335943.43 total 50.06 last 83.38) total: 4500 open/close in 58.20 seconds: 77.32 ops/second - /unlink 1454 (time 1780335965.42 total 10.01 last 145.28) - /unlink 2921 (time 1780335975.43 total 20.01 last 146.67) - /unlink 4432 (time 1780335985.43 total 30.01 last 151.06) total: 4500 /unlink in 30.63 seconds: 146.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 823 (time 1780336000.49 total 10.01 last 82.26) - open/close 1625 (time 1780336010.50 total 20.01 last 80.18) - open/close 2418 (time 1780336020.51 total 30.02 last 79.23) - open/close 3275 (time 1780336030.53 total 40.04 last 85.52) - open/close 4127 (time 1780336040.53 total 50.04 last 85.18) total: 4500 open/close in 54.45 seconds: 82.64 ops/second - /unlink 1597 (time 1780336058.83 total 10.00 last 159.68) - /unlink 3112 (time 1780336068.84 total 20.00 last 151.45) total: 4500 /unlink in 28.60 seconds: 157.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 807 (time 1780336092.38 total 10.00 last 80.68) - open/close 1613 (time 1780336102.38 total 20.00 last 80.59) - open/close 2388 (time 1780336112.38 total 30.00 last 77.50) - open/close 3199 (time 1780336122.39 total 40.01 last 81.07) - open/close 4041 (time 1780336132.39 total 50.01 last 84.18) total: 4500 open/close in 56.01 seconds: 80.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1293 (time 1780336152.65 total 10.00 last 129.24) - /unlink 2884 (time 1780336162.66 total 20.02 last 158.92) - /unlink 4379 (time 1780336172.68 total 30.03 last 149.24) total: 4500 /unlink in 30.92 seconds: 145.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 801 (time 1780336187.31 total 10.01 last 80.03) - open/close 1617 (time 1780336197.32 total 20.02 last 81.50) - open/close 2478 (time 1780336207.32 total 30.02 last 86.06) - open/close 3342 (time 1780336217.32 total 40.03 last 86.39) - open/close 4200 (time 1780336227.32 total 50.03 last 85.80) total: 4500 open/close in 53.47 seconds: 84.16 ops/second - /unlink 1636 (time 1780336243.99 total 10.00 last 163.57) - /unlink 3133 (time 1780336253.99 total 20.01 last 149.63) total: 4500 /unlink in 28.57 seconds: 157.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 832 (time 1780336276.62 total 10.01 last 83.15) - open/close 1659 (time 1780336286.63 total 20.02 last 82.62) - open/close 2494 (time 1780336296.63 total 30.02 last 83.44) - open/close 3296 (time 1780336306.64 total 40.03 last 80.15) - open/close 4153 (time 1780336316.65 total 50.04 last 85.64) total: 4500 open/close in 54.23 seconds: 82.98 ops/second - /unlink 1658 (time 1780336334.92 total 10.00 last 165.76) - /unlink 2754 (time 1780336344.92 total 20.00 last 109.58) - /unlink 4235 (time 1780336354.92 total 30.01 last 148.08) total: 4500 /unlink in 31.91 seconds: 141.03 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93982 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg355-server unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local oleg355-server: modules unloaded.