== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:50:19 (1787838619) Stopping clients: oleg646-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg646-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg646-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg646-server oleg646-server: oleg646-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing load_modules_local oleg646-server: Loading modules from /home/green/git/lustre-release/lustre oleg646-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg646-server: detected 4 online CPUs by sysfs oleg646-server: MODOPTS_LIBCFS= oleg646-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg646-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg646-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg646-client.virtnet: -o user_xattr,flock 192.168.206.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 764 (time 1787838771.63 total 10.01 last 76.31) - open/close 1554 (time 1787838781.64 total 20.02 last 78.94) - open/close 2363 (time 1787838791.67 total 30.05 last 80.65) - open/close 3216 (time 1787838801.68 total 40.06 last 85.22) - open/close 4074 (time 1787838811.69 total 50.07 last 85.74) total: 4500 open/close in 56.03 seconds: 80.32 ops/second - /unlink 1551 (time 1787838831.51 total 10.00 last 155.10) - /unlink 2946 (time 1787838841.51 total 20.00 last 139.46) - /unlink 4418 (time 1787838851.51 total 30.01 last 147.15) total: 4500 /unlink in 30.67 seconds: 146.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 819 (time 1787838865.81 total 10.01 last 81.80) - open/close 1556 (time 1787838875.83 total 20.03 last 73.59) - open/close 2386 (time 1787838885.83 total 30.03 last 82.99) - open/close 3210 (time 1787838895.84 total 40.04 last 82.35) - open/close 4027 (time 1787838905.85 total 50.04 last 81.63) total: 4500 open/close in 55.50 seconds: 81.09 ops/second - /unlink 1435 (time 1787838925.66 total 10.00 last 143.45) - /unlink 2830 (time 1787838935.66 total 20.01 last 139.48) - /unlink 4168 (time 1787838945.66 total 30.01 last 133.78) total: 4500 /unlink in 32.26 seconds: 139.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 839 (time 1787838962.73 total 10.00 last 83.86) - open/close 1604 (time 1787838972.75 total 20.03 last 76.33) - open/close 2340 (time 1787838982.75 total 30.03 last 73.60) - open/close 3044 (time 1787838992.80 total 40.08 last 70.06) - open/close 3876 (time 1787839002.80 total 50.08 last 83.16) total: 4500 open/close in 58.45 seconds: 76.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1526 (time 1787839024.90 total 10.00 last 152.53) - /unlink 3023 (time 1787839034.90 total 20.01 last 149.66) total: 4500 /unlink in 29.93 seconds: 150.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 815 (time 1787839058.67 total 10.02 last 81.38) - open/close 1589 (time 1787839068.68 total 20.02 last 77.33) - open/close 2506 (time 1787839078.69 total 30.03 last 91.62) - open/close 3349 (time 1787839088.69 total 40.03 last 84.29) - open/close 4163 (time 1787839098.70 total 50.04 last 81.34) total: 4500 open/close in 54.17 seconds: 83.08 ops/second - /unlink 1532 (time 1787839116.74 total 10.01 last 153.06) - /unlink 3082 (time 1787839126.74 total 20.01 last 154.99) total: 4500 /unlink in 29.74 seconds: 151.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 833 (time 1787839150.50 total 10.02 last 83.16) - open/close 1662 (time 1787839160.51 total 20.03 last 82.81) - open/close 2535 (time 1787839170.51 total 30.03 last 87.28) - open/close 3432 (time 1787839180.52 total 40.03 last 89.66) - open/close 4262 (time 1787839190.52 total 50.04 last 82.94) total: 4500 open/close in 52.69 seconds: 85.41 ops/second - /unlink 1622 (time 1787839206.67 total 10.00 last 162.20) - /unlink 3220 (time 1787839216.67 total 20.00 last 159.79) total: 4500 /unlink in 28.06 seconds: 160.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 836 (time 1787839239.80 total 10.01 last 83.55) - open/close 1638 (time 1787839249.81 total 20.01 last 80.17) - open/close 2491 (time 1787839259.81 total 30.02 last 85.23) - open/close 3323 (time 1787839269.82 total 40.02 last 83.17) - open/close 4179 (time 1787839279.83 total 50.03 last 85.49) total: 4500 open/close in 53.89 seconds: 83.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1693 (time 1787839297.63 total 10.00 last 169.27) - /unlink 3325 (time 1787839307.64 total 20.00 last 163.18) total: 4500 /unlink in 29.07 seconds: 154.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 798 (time 1787839331.34 total 10.00 last 79.80) - open/close 1544 (time 1787839341.35 total 20.01 last 74.55) - open/close 2289 (time 1787839351.35 total 30.01 last 74.49) - open/close 3030 (time 1787839361.36 total 40.02 last 74.03) - open/close 3758 (time 1787839371.37 total 50.03 last 72.71) total: 4500 open/close in 59.72 seconds: 75.35 ops/second - /unlink 1407 (time 1787839395.43 total 10.00 last 140.66) - /unlink 2966 (time 1787839405.44 total 20.01 last 155.86) total: 4500 /unlink in 30.00 seconds: 149.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 849 (time 1787839429.27 total 10.01 last 84.82) - open/close 1696 (time 1787839439.28 total 20.02 last 84.65) - open/close 2460 (time 1787839449.28 total 30.02 last 76.37) - open/close 3324 (time 1787839459.29 total 40.03 last 86.35) - open/close 4107 (time 1787839469.29 total 50.03 last 78.25) total: 4500 open/close in 57.07 seconds: 78.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1231 (time 1787839491.68 total 10.01 last 123.02) - /unlink 2534 (time 1787839501.68 total 20.01 last 130.25) - /unlink 3894 (time 1787839511.68 total 30.01 last 135.97) total: 4500 /unlink in 34.34 seconds: 131.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 805 (time 1787839530.25 total 10.01 last 80.44) - open/close 1638 (time 1787839540.25 total 20.01 last 83.29) - open/close 2495 (time 1787839550.26 total 30.02 last 85.62) - open/close 3265 (time 1787839560.27 total 40.02 last 76.97) - open/close 4096 (time 1787839570.27 total 50.02 last 83.08) total: 4500 open/close in 55.77 seconds: 80.69 ops/second - /unlink 1420 (time 1787839590.33 total 10.00 last 141.98) - /unlink 2672 (time 1787839600.33 total 20.01 last 125.10) - /unlink 4187 (time 1787839610.33 total 30.01 last 151.49) total: 4500 /unlink in 32.42 seconds: 138.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 858 (time 1787839626.75 total 10.00 last 85.78) - open/close 1641 (time 1787839636.78 total 20.03 last 78.08) - open/close 2468 (time 1787839646.78 total 30.04 last 82.65) - open/close 3313 (time 1787839656.79 total 40.04 last 84.47) - open/close 4143 (time 1787839666.79 total 50.04 last 82.98) total: 4500 open/close in 54.08 seconds: 83.21 ops/second - /unlink 1583 (time 1787839684.28 total 10.01 last 158.14) - /unlink 3043 (time 1787839694.29 total 20.02 last 145.87) - /unlink 4444 (time 1787839704.29 total 30.03 last 140.01) total: 4500 /unlink in 30.34 seconds: 148.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 929 (time 1787839718.83 total 10.01 last 92.84) - open/close 1745 (time 1787839728.84 total 20.01 last 81.53) - open/close 2598 (time 1787839738.84 total 30.02 last 85.28) - open/close 3509 (time 1787839748.84 total 40.02 last 91.09) - open/close 4356 (time 1787839758.85 total 50.02 last 84.68) total: 4500 open/close in 51.52 seconds: 87.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1629 (time 1787839773.48 total 10.00 last 162.83) - /unlink 3079 (time 1787839783.48 total 20.01 last 144.95) total: 4500 /unlink in 28.96 seconds: 155.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 787 (time 1787839806.84 total 10.00 last 78.68) - open/close 1643 (time 1787839816.86 total 20.02 last 85.49) - open/close 2481 (time 1787839826.87 total 30.03 last 83.70) - open/close 3280 (time 1787839836.88 total 40.03 last 79.85) - open/close 4163 (time 1787839846.88 total 50.04 last 88.22) total: 4500 open/close in 53.41 seconds: 84.25 ops/second - /unlink 1595 (time 1787839864.06 total 10.00 last 159.45) - /unlink 3116 (time 1787839874.06 total 20.01 last 152.02) total: 4500 /unlink in 28.81 seconds: 156.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 823 (time 1787839896.75 total 10.00 last 82.29) - open/close 1645 (time 1787839906.76 total 20.01 last 82.14) - open/close 2433 (time 1787839916.78 total 30.03 last 78.63) - open/close 3281 (time 1787839926.78 total 40.03 last 84.77) - open/close 4130 (time 1787839936.79 total 50.04 last 84.81) total: 4500 open/close in 54.23 seconds: 82.99 ops/second - /unlink 1532 (time 1787839953.90 total 10.00 last 153.19) - /unlink 3096 (time 1787839963.90 total 20.00 last 156.37) total: 4500 /unlink in 29.21 seconds: 154.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 994 (time 1787839986.35 total 10.01 last 99.29) - open/close 2050 (time 1787839996.35 total 20.01 last 105.59) - open/close 3020 (time 1787840006.37 total 30.02 last 96.90) - open/close 3919 (time 1787840016.37 total 40.03 last 89.87) total: 4500 open/close in 46.65 seconds: 96.47 ops/second - /unlink 1601 (time 1787840036.29 total 10.00 last 160.04) - /unlink 3183 (time 1787840046.30 total 20.02 last 158.01) total: 4500 /unlink in 28.54 seconds: 157.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 913 (time 1787840068.96 total 10.01 last 91.21) - open/close 1753 (time 1787840078.96 total 20.01 last 83.96) - open/close 2612 (time 1787840088.97 total 30.02 last 85.85) - open/close 3480 (time 1787840098.97 total 40.03 last 86.75) - open/close 4384 (time 1787840108.97 total 50.03 last 90.38) total: 4500 open/close in 51.37 seconds: 87.61 ops/second - /unlink 1530 (time 1787840123.67 total 10.00 last 152.93) - /unlink 3123 (time 1787840133.67 total 20.01 last 159.26) total: 4500 /unlink in 29.68 seconds: 151.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 797 (time 1787840157.20 total 10.01 last 79.64) - open/close 1595 (time 1787840167.20 total 20.01 last 79.79) - open/close 2417 (time 1787840177.21 total 30.01 last 82.19) - open/close 3168 (time 1787840187.21 total 40.01 last 75.10) - open/close 4018 (time 1787840197.22 total 50.02 last 84.91) total: 4500 open/close in 57.23 seconds: 78.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1390 (time 1787840218.95 total 10.01 last 138.92) - /unlink 2777 (time 1787840228.95 total 20.01 last 138.69) - /unlink 4186 (time 1787840238.95 total 30.01 last 140.88) total: 4500 /unlink in 32.11 seconds: 140.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 802 (time 1787840255.00 total 10.01 last 80.13) - open/close 1619 (time 1787840265.01 total 20.02 last 81.63) - open/close 2477 (time 1787840275.03 total 30.03 last 85.66) - open/close 3293 (time 1787840285.03 total 40.03 last 81.59) - open/close 4090 (time 1787840295.05 total 50.05 last 79.55) total: 4500 open/close in 55.72 seconds: 80.77 ops/second - /unlink 1314 (time 1787840315.61 total 10.01 last 131.30) - /unlink 2729 (time 1787840325.62 total 20.02 last 141.34) - /unlink 4244 (time 1787840335.62 total 30.02 last 151.48) total: 4500 /unlink in 32.01 seconds: 140.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 732 (time 1787840351.51 total 10.00 last 73.20) - open/close 1495 (time 1787840361.52 total 20.01 last 76.20) - open/close 2286 (time 1787840371.52 total 30.02 last 79.09) - open/close 3095 (time 1787840381.53 total 40.03 last 80.82) - open/close 3791 (time 1787840391.54 total 50.03 last 69.57) total: 4500 open/close in 59.82 seconds: 75.23 ops/second - /unlink 1439 (time 1787840415.66 total 10.00 last 143.86) - /unlink 2847 (time 1787840425.66 total 20.01 last 140.75) - /unlink 4371 (time 1787840435.67 total 30.01 last 152.29) total: 4500 /unlink in 30.88 seconds: 145.74 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11330: 75444 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg646-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server unloading modules via unload_modules_local on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing unload_modules_local oleg646-server: modules unloaded.