== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:36:20 (1774877780) Stopping clients: oleg627-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg627-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg627-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg627-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg627-server oleg627-server: oleg627-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing load_modules_local oleg627-server: Loading modules from /home/green/git/lustre-release/lustre oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg627-server: detected 4 online CPUs by sysfs oleg627-server: MODOPTS_LIBCFS= oleg627-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg627-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg627-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg627-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 833 (time 1774877955.98 total 10.00 last 83.29) - open/close 1597 (time 1774877965.99 total 20.01 last 76.36) - open/close 2463 (time 1774877975.99 total 30.01 last 86.58) - open/close 3329 (time 1774877985.99 total 40.01 last 86.57) - open/close 4175 (time 1774877996.01 total 50.03 last 84.49) total: 4500 open/close in 53.81 seconds: 83.63 ops/second - /unlink 1556 (time 1774878013.16 total 10.00 last 155.57) - /unlink 2932 (time 1774878023.17 total 20.01 last 137.55) - /unlink 4434 (time 1774878033.17 total 30.01 last 150.18) total: 4500 /unlink in 30.58 seconds: 147.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 864 (time 1774878047.35 total 10.00 last 86.38) - open/close 1685 (time 1774878057.35 total 20.00 last 82.09) - open/close 2499 (time 1774878067.35 total 30.00 last 81.40) - open/close 3303 (time 1774878077.37 total 40.02 last 80.29) - open/close 4030 (time 1774878087.38 total 50.03 last 72.61) total: 4500 open/close in 56.64 seconds: 79.45 ops/second - /unlink 1201 (time 1774878110.29 total 10.00 last 120.04) - /unlink 2701 (time 1774878120.29 total 20.01 last 149.96) - /unlink 4121 (time 1774878130.30 total 30.01 last 141.92) total: 4500 /unlink in 32.63 seconds: 137.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 700 (time 1774878147.88 total 10.01 last 69.94) - open/close 1499 (time 1774878157.89 total 20.02 last 79.84) - open/close 2179 (time 1774878167.90 total 30.03 last 67.93) - open/close 2947 (time 1774878177.91 total 40.04 last 76.66) - open/close 3738 (time 1774878187.93 total 50.06 last 78.97) total: 4500 open/close in 59.87 seconds: 75.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1548 (time 1774878211.35 total 10.00 last 154.77) - /unlink 2921 (time 1774878221.36 total 20.00 last 137.28) - /unlink 4439 (time 1774878231.36 total 30.00 last 151.79) total: 4500 /unlink in 30.47 seconds: 147.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 805 (time 1774878245.43 total 10.01 last 80.45) - open/close 1622 (time 1774878255.43 total 20.01 last 81.66) - open/close 2418 (time 1774878265.45 total 30.03 last 79.46) - open/close 3199 (time 1774878275.45 total 40.04 last 78.04) - open/close 4044 (time 1774878285.46 total 50.04 last 84.48) total: 4500 open/close in 55.59 seconds: 80.95 ops/second - /unlink 1475 (time 1774878304.51 total 10.00 last 147.48) - /unlink 2835 (time 1774878314.51 total 20.00 last 135.99) - /unlink 4356 (time 1774878324.51 total 30.00 last 152.09) total: 4500 /unlink in 30.91 seconds: 145.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 940 (time 1774878339.03 total 10.01 last 93.95) - open/close 1703 (time 1774878349.07 total 20.04 last 76.06) - open/close 2475 (time 1774878359.07 total 30.04 last 77.14) - open/close 3359 (time 1774878369.08 total 40.05 last 88.35) - open/close 4224 (time 1774878379.08 total 50.05 last 86.46) total: 4500 open/close in 53.55 seconds: 84.03 ops/second - /unlink 1542 (time 1774878395.76 total 10.01 last 154.12) - /unlink 3095 (time 1774878405.76 total 20.01 last 155.28) total: 4500 /unlink in 29.21 seconds: 154.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 767 (time 1774878428.67 total 10.00 last 76.66) - open/close 1671 (time 1774878438.68 total 20.01 last 90.34) - open/close 2501 (time 1774878448.68 total 30.02 last 82.97) - open/close 3390 (time 1774878458.69 total 40.02 last 88.85) - open/close 4263 (time 1774878468.71 total 50.04 last 87.12) total: 4500 open/close in 53.23 seconds: 84.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1621 (time 1774878485.21 total 10.01 last 162.02) - /unlink 3200 (time 1774878495.22 total 20.01 last 157.83) total: 4500 /unlink in 28.38 seconds: 158.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 895 (time 1774878517.10 total 10.04 last 89.13) - open/close 1697 (time 1774878527.13 total 20.07 last 79.95) - open/close 2543 (time 1774878537.14 total 30.08 last 84.52) - open/close 3387 (time 1774878547.17 total 40.12 last 84.11) - open/close 4193 (time 1774878557.17 total 50.12 last 80.57) total: 4500 open/close in 53.86 seconds: 83.55 ops/second - /unlink 1746 (time 1774878574.22 total 10.00 last 174.55) - /unlink 3400 (time 1774878584.22 total 20.01 last 165.35) total: 4500 /unlink in 26.53 seconds: 169.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 865 (time 1774878603.74 total 10.00 last 86.49) - open/close 1699 (time 1774878613.75 total 20.01 last 83.34) - open/close 2575 (time 1774878623.76 total 30.02 last 87.53) - open/close 3467 (time 1774878633.77 total 40.03 last 89.11) - open/close 4338 (time 1774878643.77 total 50.03 last 87.07) total: 4500 open/close in 51.80 seconds: 86.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1503 (time 1774878658.87 total 10.01 last 150.17) - /unlink 2989 (time 1774878668.87 total 20.01 last 148.56) total: 4500 /unlink in 29.22 seconds: 154.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 915 (time 1774878691.29 total 10.01 last 91.45) - open/close 1808 (time 1774878701.29 total 20.01 last 89.29) - open/close 2733 (time 1774878711.29 total 30.01 last 92.49) - open/close 3555 (time 1774878721.30 total 40.01 last 82.16) - open/close 4411 (time 1774878731.30 total 50.02 last 85.55) total: 4500 open/close in 50.94 seconds: 88.33 ops/second - /unlink 1429 (time 1774878745.36 total 10.00 last 142.85) - /unlink 3042 (time 1774878755.36 total 20.01 last 161.24) total: 4500 /unlink in 29.35 seconds: 153.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 763 (time 1774878777.90 total 10.00 last 76.29) - open/close 1584 (time 1774878787.90 total 20.01 last 82.06) - open/close 2485 (time 1774878797.91 total 30.01 last 90.03) - open/close 3419 (time 1774878807.91 total 40.02 last 93.39) - open/close 4403 (time 1774878817.91 total 50.02 last 98.36) total: 4500 open/close in 51.09 seconds: 88.09 ops/second - /unlink 1562 (time 1774878832.16 total 10.00 last 156.15) - /unlink 3126 (time 1774878842.16 total 20.00 last 156.37) total: 4500 /unlink in 28.60 seconds: 157.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 813 (time 1774878865.22 total 10.00 last 81.28) - open/close 1602 (time 1774878875.23 total 20.01 last 78.84) - open/close 2463 (time 1774878885.23 total 30.01 last 86.06) - open/close 3268 (time 1774878895.23 total 40.02 last 80.48) - open/close 4090 (time 1774878905.24 total 50.02 last 82.17) total: 4500 open/close in 55.02 seconds: 81.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1463 (time 1774878924.34 total 10.02 last 146.04) - /unlink 2961 (time 1774878934.34 total 20.02 last 149.78) total: 4500 /unlink in 29.89 seconds: 150.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 812 (time 1774878957.94 total 10.00 last 81.20) - open/close 1568 (time 1774878967.95 total 20.01 last 75.53) - open/close 2445 (time 1774878977.96 total 30.01 last 87.67) - open/close 3252 (time 1774878987.96 total 40.01 last 80.69) - open/close 4088 (time 1774878997.96 total 50.02 last 83.56) total: 4500 open/close in 55.08 seconds: 81.69 ops/second - /unlink 1568 (time 1774879016.39 total 10.02 last 156.46) - /unlink 3038 (time 1774879026.40 total 20.03 last 146.91) total: 4500 /unlink in 29.23 seconds: 153.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 780 (time 1774879048.91 total 10.01 last 77.95) - open/close 1674 (time 1774879058.91 total 20.01 last 89.37) - open/close 2504 (time 1774879068.91 total 30.01 last 82.98) - open/close 3382 (time 1774879078.92 total 40.01 last 87.78) - open/close 4290 (time 1774879088.93 total 50.03 last 90.69) total: 4500 open/close in 51.99 seconds: 86.55 ops/second - /unlink 1428 (time 1774879104.76 total 10.00 last 142.79) - /unlink 2824 (time 1774879114.76 total 20.00 last 139.56) - /unlink 4283 (time 1774879124.76 total 30.01 last 145.85) total: 4500 /unlink in 31.33 seconds: 143.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 905 (time 1774879138.77 total 10.00 last 90.46) - open/close 1830 (time 1774879148.77 total 20.01 last 92.45) - open/close 2745 (time 1774879158.77 total 30.01 last 91.48) - open/close 3699 (time 1774879168.78 total 40.02 last 95.36) - open/close 4477 (time 1774879178.78 total 50.02 last 77.78) total: 4500 open/close in 50.38 seconds: 89.32 ops/second - /unlink 1261 (time 1774879193.53 total 10.01 last 126.00) - /unlink 2709 (time 1774879203.54 total 20.01 last 144.79) - /unlink 4047 (time 1774879213.54 total 30.01 last 133.74) total: 4500 /unlink in 32.94 seconds: 136.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 874 (time 1774879230.40 total 10.00 last 87.39) - open/close 1734 (time 1774879240.41 total 20.01 last 85.94) - open/close 2757 (time 1774879250.42 total 30.02 last 102.16) - open/close 3716 (time 1774879260.43 total 40.03 last 95.86) total: 4500 open/close in 48.06 seconds: 93.63 ops/second - /unlink 1837 (time 1774879281.21 total 10.00 last 183.68) - /unlink 3534 (time 1774879291.21 total 20.00 last 169.65) total: 4500 /unlink in 26.29 seconds: 171.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 902 (time 1774879310.56 total 10.00 last 90.16) - open/close 1753 (time 1774879320.58 total 20.02 last 84.99) - open/close 2585 (time 1774879330.58 total 30.02 last 83.14) - open/close 3483 (time 1774879340.59 total 40.03 last 89.72) - open/close 4327 (time 1774879350.61 total 50.05 last 84.26) total: 4500 open/close in 52.09 seconds: 86.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1813 (time 1774879365.59 total 10.00 last 181.27) - /unlink 3720 (time 1774879375.59 total 20.00 last 190.66) total: 4500 /unlink in 23.97 seconds: 187.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1211 (time 1774879392.24 total 10.01 last 121.03) - open/close 2321 (time 1774879402.24 total 20.01 last 110.99) - open/close 3272 (time 1774879412.25 total 30.01 last 95.04) - open/close 4113 (time 1774879422.25 total 40.02 last 84.07) total: 4500 open/close in 43.94 seconds: 102.41 ops/second - /unlink 1592 (time 1774879439.14 total 10.01 last 159.11) - /unlink 3179 (time 1774879449.15 total 20.01 last 158.57) total: 4500 /unlink in 28.43 seconds: 158.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 767 (time 1774879471.19 total 10.02 last 76.52) - open/close 1518 (time 1774879481.20 total 20.03 last 75.08) - open/close 2355 (time 1774879491.20 total 30.03 last 83.67) - open/close 3241 (time 1774879501.21 total 40.04 last 88.48) - open/close 4083 (time 1774879511.22 total 50.05 last 84.18) total: 4500 open/close in 55.18 seconds: 81.56 ops/second - /unlink 1403 (time 1774879529.68 total 10.02 last 140.05) - /unlink 2861 (time 1774879539.68 total 20.02 last 145.74) - /unlink 4452 (time 1774879549.69 total 30.03 last 159.01) total: 4500 /unlink in 30.37 seconds: 148.17 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11170: 93904 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg627-server unloading modules via unload_modules_local on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing unload_modules_local oleg627-server: modules unloaded.