== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:50:44 (1787399444) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg156-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg156-server oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg156-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 702 (time 1787399654.17 total 10.02 last 70.07) - open/close 1402 (time 1787399664.18 total 20.03 last 69.92) - open/close 2150 (time 1787399674.20 total 30.05 last 74.66) - open/close 2820 (time 1787399684.20 total 40.05 last 66.99) - open/close 3597 (time 1787399694.21 total 50.06 last 77.65) - open/close 4317 (time 1787399704.21 total 60.06 last 71.98) total: 4500 open/close in 63.57 seconds: 70.79 ops/second - /unlink 1535 (time 1787399722.16 total 10.00 last 153.46) - /unlink 3233 (time 1787399732.17 total 20.00 last 169.78) total: 4500 /unlink in 27.84 seconds: 161.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 802 (time 1787399755.24 total 10.01 last 80.15) - open/close 1577 (time 1787399765.25 total 20.02 last 77.43) - open/close 2453 (time 1787399775.25 total 30.02 last 87.58) - open/close 3284 (time 1787399785.26 total 40.03 last 83.02) - open/close 4265 (time 1787399795.26 total 50.03 last 98.09) total: 4500 open/close in 52.80 seconds: 85.23 ops/second - /unlink 1605 (time 1787399811.84 total 10.00 last 160.50) - /unlink 3373 (time 1787399821.85 total 20.00 last 176.79) total: 4500 /unlink in 26.92 seconds: 167.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 940 (time 1787399842.82 total 10.02 last 93.85) - open/close 1916 (time 1787399852.83 total 20.03 last 97.51) - open/close 2731 (time 1787399862.84 total 30.03 last 81.45) - open/close 3478 (time 1787399872.85 total 40.04 last 74.63) - open/close 4288 (time 1787399882.85 total 50.05 last 80.96) total: 4500 open/close in 52.12 seconds: 86.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1354 (time 1787399899.98 total 10.01 last 135.33) - /unlink 2629 (time 1787399909.98 total 20.01 last 127.44) - /unlink 4037 (time 1787399919.98 total 30.01 last 140.79) total: 4500 /unlink in 33.52 seconds: 134.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 644 (time 1787399938.32 total 10.03 last 64.23) - open/close 1220 (time 1787399948.33 total 20.04 last 57.52) - open/close 1935 (time 1787399958.34 total 30.05 last 71.46) - open/close 2804 (time 1787399968.34 total 40.05 last 86.83) - open/close 3528 (time 1787399978.37 total 50.08 last 72.24) - open/close 4276 (time 1787399988.37 total 60.08 last 74.79) total: 4500 open/close in 62.20 seconds: 72.35 ops/second - /unlink 1642 (time 1787400005.52 total 10.01 last 164.09) - /unlink 3233 (time 1787400015.53 total 20.01 last 159.05) total: 4500 /unlink in 27.62 seconds: 162.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 712 (time 1787400037.63 total 10.00 last 71.19) - open/close 1559 (time 1787400047.64 total 20.01 last 84.65) - open/close 2306 (time 1787400057.68 total 30.05 last 74.41) - open/close 3158 (time 1787400067.68 total 40.05 last 85.18) - open/close 4071 (time 1787400077.69 total 50.06 last 91.23) total: 4500 open/close in 54.91 seconds: 81.95 ops/second - /unlink 1491 (time 1787400097.61 total 10.00 last 149.07) - /unlink 2880 (time 1787400107.61 total 20.00 last 138.86) - /unlink 4448 (time 1787400117.61 total 30.01 last 156.77) total: 4500 /unlink in 30.39 seconds: 148.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 818 (time 1787400132.57 total 10.01 last 81.74) - open/close 1633 (time 1787400142.57 total 20.01 last 81.47) - open/close 2475 (time 1787400152.58 total 30.02 last 84.13) - open/close 3328 (time 1787400162.58 total 40.02 last 85.29) - open/close 4085 (time 1787400172.60 total 50.04 last 75.59) total: 4500 open/close in 54.28 seconds: 82.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1576 (time 1787400191.51 total 10.00 last 157.55) - /unlink 3123 (time 1787400201.51 total 20.00 last 154.68) total: 4500 /unlink in 29.29 seconds: 153.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 954 (time 1787400225.11 total 10.00 last 95.38) - open/close 1805 (time 1787400235.11 total 20.00 last 85.09) - open/close 2732 (time 1787400245.11 total 30.01 last 92.66) - open/close 3686 (time 1787400255.12 total 40.01 last 95.36) total: 4500 open/close in 48.79 seconds: 92.24 ops/second - /unlink 1582 (time 1787400278.46 total 10.00 last 158.15) - /unlink 3095 (time 1787400288.46 total 20.01 last 151.25) total: 4500 /unlink in 29.18 seconds: 154.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 962 (time 1787400312.70 total 10.02 last 96.03) - open/close 1583 (time 1787400322.73 total 20.05 last 61.89) - open/close 2322 (time 1787400332.74 total 30.06 last 73.85) - open/close 3219 (time 1787400342.74 total 40.06 last 89.66) - open/close 3968 (time 1787400352.75 total 50.07 last 74.86) total: 4500 open/close in 56.67 seconds: 79.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1623 (time 1787400373.38 total 10.00 last 162.29) - /unlink 3276 (time 1787400383.38 total 20.00 last 165.25) total: 4500 /unlink in 27.54 seconds: 163.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 839 (time 1787400406.05 total 10.00 last 83.86) - open/close 1695 (time 1787400416.06 total 20.01 last 85.55) - open/close 2584 (time 1787400426.08 total 30.03 last 88.74) - open/close 3300 (time 1787400436.09 total 40.04 last 71.53) - open/close 4091 (time 1787400446.10 total 50.06 last 78.96) total: 4500 open/close in 54.78 seconds: 82.15 ops/second - /unlink 1771 (time 1787400464.91 total 10.00 last 177.04) - /unlink 3247 (time 1787400474.91 total 20.01 last 147.56) total: 4500 /unlink in 28.39 seconds: 158.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 823 (time 1787400498.24 total 10.01 last 82.25) - open/close 1694 (time 1787400508.24 total 20.01 last 87.09) - open/close 2491 (time 1787400518.25 total 30.02 last 79.59) - open/close 3251 (time 1787400528.26 total 40.02 last 75.98) - open/close 4024 (time 1787400538.26 total 50.03 last 77.24) total: 4500 open/close in 58.33 seconds: 77.14 ops/second - /unlink 1324 (time 1787400563.96 total 10.00 last 132.38) - /unlink 2372 (time 1787400573.97 total 20.01 last 104.72) - /unlink 4018 (time 1787400583.97 total 30.01 last 164.57) total: 4500 /unlink in 32.98 seconds: 136.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 866 (time 1787400601.05 total 10.00 last 86.58) - open/close 1752 (time 1787400611.06 total 20.01 last 88.54) - open/close 2698 (time 1787400621.06 total 30.01 last 94.60) - open/close 3655 (time 1787400631.06 total 40.02 last 95.64) total: 4500 open/close in 49.01 seconds: 91.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1478 (time 1787400653.96 total 10.00 last 147.74) - /unlink 3192 (time 1787400663.96 total 20.01 last 171.37) total: 4500 /unlink in 28.84 seconds: 156.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 900 (time 1787400687.97 total 10.01 last 89.92) - open/close 1856 (time 1787400697.98 total 20.02 last 95.54) - open/close 2809 (time 1787400707.98 total 30.02 last 95.28) - open/close 3698 (time 1787400717.99 total 40.02 last 88.85) total: 4500 open/close in 49.58 seconds: 90.77 ops/second - /unlink 1740 (time 1787400741.10 total 10.01 last 173.91) - /unlink 3294 (time 1787400751.11 total 20.01 last 155.28) total: 4500 /unlink in 28.02 seconds: 160.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 742 (time 1787400775.24 total 10.01 last 74.14) - open/close 1342 (time 1787400785.25 total 20.01 last 59.96) - open/close 2141 (time 1787400795.25 total 30.02 last 79.88) - open/close 2931 (time 1787400805.26 total 40.03 last 78.93) - open/close 3814 (time 1787400815.26 total 50.03 last 88.24) total: 4500 open/close in 57.67 seconds: 78.03 ops/second - /unlink 1588 (time 1787400837.14 total 10.00 last 158.73) - /unlink 3144 (time 1787400847.14 total 20.01 last 155.53) total: 4500 /unlink in 29.09 seconds: 154.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 898 (time 1787400871.40 total 10.01 last 89.73) - open/close 1662 (time 1787400881.42 total 20.02 last 76.27) - open/close 2484 (time 1787400891.42 total 30.02 last 82.20) - open/close 3326 (time 1787400901.44 total 40.05 last 84.00) - open/close 4176 (time 1787400911.46 total 50.07 last 84.83) total: 4500 open/close in 54.22 seconds: 82.99 ops/second - /unlink 1469 (time 1787400929.84 total 10.01 last 146.78) - /unlink 3027 (time 1787400939.84 total 20.01 last 155.75) - /unlink 4231 (time 1787400949.85 total 30.01 last 120.36) total: 4500 /unlink in 31.76 seconds: 141.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 766 (time 1787400966.75 total 10.00 last 76.57) - open/close 1591 (time 1787400976.75 total 20.01 last 82.48) - open/close 2490 (time 1787400986.75 total 30.01 last 89.86) - open/close 3332 (time 1787400996.77 total 40.03 last 84.07) - open/close 4273 (time 1787401006.79 total 50.04 last 93.94) total: 4500 open/close in 53.01 seconds: 84.89 ops/second - /unlink 1543 (time 1787401024.58 total 10.00 last 154.29) - /unlink 3106 (time 1787401034.58 total 20.01 last 156.22) total: 4500 /unlink in 30.00 seconds: 149.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 759 (time 1787401058.78 total 10.00 last 75.88) - open/close 1507 (time 1787401068.79 total 20.01 last 74.76) - open/close 2334 (time 1787401078.79 total 30.01 last 82.66) - open/close 2983 (time 1787401088.82 total 40.04 last 64.73) - open/close 3930 (time 1787401098.83 total 50.05 last 94.62) total: 4500 open/close in 57.22 seconds: 78.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1720 (time 1787401119.81 total 10.00 last 171.96) - /unlink 3135 (time 1787401129.81 total 20.00 last 141.47) - /unlink 4461 (time 1787401139.82 total 30.01 last 132.55) total: 4500 /unlink in 30.25 seconds: 148.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 779 (time 1787401154.63 total 10.01 last 77.83) - open/close 1449 (time 1787401164.63 total 20.01 last 67.00) - open/close 2341 (time 1787401174.64 total 30.01 last 89.18) - open/close 3005 (time 1787401184.64 total 40.01 last 66.40) - open/close 3569 (time 1787401194.65 total 50.02 last 56.34) - open/close 4031 (time 1787401204.67 total 60.04 last 46.10) total: 4500 open/close in 67.90 seconds: 66.27 ops/second - /unlink 1319 (time 1787401227.90 total 10.00 last 131.90) - /unlink 2872 (time 1787401237.91 total 20.01 last 155.11) - /unlink 3929 (time 1787401247.92 total 30.02 last 105.60) total: 4500 /unlink in 35.55 seconds: 126.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 636 (time 1787401269.78 total 10.00 last 63.59) - open/close 1319 (time 1787401279.79 total 20.01 last 68.25) - open/close 2014 (time 1787401289.80 total 30.01 last 69.45) - open/close 2785 (time 1787401299.80 total 40.02 last 77.07) - open/close 3612 (time 1787401309.82 total 50.04 last 82.50) - open/close 4322 (time 1787401319.85 total 60.07 last 70.81) total: 4500 open/close in 62.08 seconds: 72.49 ops/second - /unlink 1445 (time 1787401336.17 total 10.02 last 144.27) - /unlink 2971 (time 1787401346.17 total 20.02 last 152.55) - /unlink 4337 (time 1787401356.18 total 30.03 last 136.49) total: 4500 /unlink in 31.52 seconds: 142.74 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95090 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg156-server unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: modules unloaded.