== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:44:00 (1789130640) Stopping clients: oleg337-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg337-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg337-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg337-server oleg337-server: oleg337-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg337-server: detected 4 online CPUs by sysfs oleg337-server: MODOPTS_LIBCFS= oleg337-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg337-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg337-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg337-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 841 (time 1789130846.54 total 10.01 last 84.05) - open/close 1601 (time 1789130856.57 total 20.03 last 75.83) - open/close 2210 (time 1789130866.60 total 30.06 last 60.71) - open/close 2769 (time 1789130876.61 total 40.07 last 55.85) - open/close 3438 (time 1789130886.62 total 50.08 last 66.84) - open/close 4264 (time 1789130896.62 total 60.08 last 82.57) total: 4500 open/close in 63.17 seconds: 71.24 ops/second - /unlink 1630 (time 1789130914.35 total 10.00 last 162.97) - /unlink 2983 (time 1789130924.35 total 20.00 last 135.29) - /unlink 4439 (time 1789130934.36 total 30.01 last 145.56) total: 4500 /unlink in 30.48 seconds: 147.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 835 (time 1789130948.83 total 10.01 last 83.41) - open/close 1698 (time 1789130958.85 total 20.02 last 86.19) - open/close 2258 (time 1789130968.85 total 30.03 last 55.97) - open/close 2991 (time 1789130978.86 total 40.04 last 73.24) - open/close 3759 (time 1789130988.87 total 50.04 last 76.74) total: 4500 open/close in 57.62 seconds: 78.10 ops/second - /unlink 1632 (time 1789131011.36 total 10.00 last 163.17) - /unlink 3197 (time 1789131021.37 total 20.01 last 156.45) total: 4500 /unlink in 29.13 seconds: 154.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 694 (time 1789131045.78 total 10.02 last 69.28) - open/close 1283 (time 1789131055.78 total 20.02 last 58.87) - open/close 2072 (time 1789131065.79 total 30.03 last 78.82) - open/close 2933 (time 1789131075.80 total 40.04 last 86.05) - open/close 3777 (time 1789131085.81 total 50.05 last 84.34) total: 4500 open/close in 57.33 seconds: 78.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1574 (time 1789131107.25 total 10.00 last 157.36) - /unlink 3138 (time 1789131117.25 total 20.00 last 156.40) total: 4500 /unlink in 28.89 seconds: 155.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 817 (time 1789131140.78 total 10.00 last 81.69) - open/close 1664 (time 1789131150.78 total 20.00 last 84.69) - open/close 2517 (time 1789131160.79 total 30.01 last 85.24) - open/close 3327 (time 1789131170.80 total 40.02 last 80.95) - open/close 4050 (time 1789131180.80 total 50.02 last 72.26) total: 4500 open/close in 55.11 seconds: 81.65 ops/second - /unlink 1490 (time 1789131200.56 total 10.00 last 148.93) - /unlink 3062 (time 1789131210.57 total 20.01 last 157.09) total: 4500 /unlink in 29.30 seconds: 153.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 827 (time 1789131234.77 total 10.00 last 82.70) - open/close 1687 (time 1789131244.78 total 20.01 last 85.94) - open/close 2538 (time 1789131254.79 total 30.02 last 85.01) - open/close 3447 (time 1789131264.80 total 40.03 last 90.82) - open/close 4304 (time 1789131274.82 total 50.04 last 85.57) total: 4500 open/close in 52.56 seconds: 85.62 ops/second - /unlink 1465 (time 1789131291.52 total 10.00 last 146.44) - /unlink 2958 (time 1789131301.52 total 20.01 last 149.26) total: 4500 /unlink in 29.48 seconds: 152.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 819 (time 1789131325.25 total 10.01 last 81.78) - open/close 1672 (time 1789131335.26 total 20.02 last 85.26) - open/close 2562 (time 1789131345.27 total 30.03 last 88.90) - open/close 3383 (time 1789131355.27 total 40.04 last 82.05) - open/close 4266 (time 1789131365.29 total 50.05 last 88.20) total: 4500 open/close in 52.64 seconds: 85.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1603 (time 1789131382.17 total 10.00 last 160.24) - /unlink 3118 (time 1789131392.18 total 20.01 last 151.35) total: 4500 /unlink in 28.79 seconds: 156.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 780 (time 1789131416.24 total 10.01 last 77.96) - open/close 1496 (time 1789131426.25 total 20.01 last 71.54) - open/close 2122 (time 1789131436.25 total 30.01 last 62.60) - open/close 2958 (time 1789131446.26 total 40.02 last 83.52) - open/close 3781 (time 1789131456.26 total 50.03 last 82.24) total: 4500 open/close in 58.96 seconds: 76.32 ops/second - /unlink 1597 (time 1789131479.40 total 10.00 last 159.65) - /unlink 3219 (time 1789131489.41 total 20.01 last 162.13) total: 4500 /unlink in 27.83 seconds: 161.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 894 (time 1789131511.69 total 10.00 last 89.39) - open/close 1665 (time 1789131521.69 total 20.00 last 77.09) - open/close 2460 (time 1789131531.70 total 30.01 last 79.42) - open/close 3365 (time 1789131541.71 total 40.02 last 90.43) - open/close 4187 (time 1789131551.72 total 50.03 last 82.10) total: 4500 open/close in 53.94 seconds: 83.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1577 (time 1789131570.54 total 10.00 last 157.65) - /unlink 3136 (time 1789131580.54 total 20.00 last 155.87) total: 4500 /unlink in 28.75 seconds: 156.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 804 (time 1789131604.03 total 10.02 last 80.25) - open/close 1550 (time 1789131614.04 total 20.03 last 74.50) - open/close 2318 (time 1789131624.07 total 30.06 last 76.59) - open/close 3145 (time 1789131634.07 total 40.06 last 82.70) - open/close 3927 (time 1789131644.07 total 50.07 last 78.16) total: 4500 open/close in 56.99 seconds: 78.95 ops/second - /unlink 1495 (time 1789131664.80 total 10.00 last 149.49) - /unlink 3032 (time 1789131674.80 total 20.00 last 153.66) - /unlink 4474 (time 1789131684.80 total 30.01 last 144.16) total: 4500 /unlink in 30.15 seconds: 149.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 858 (time 1789131698.76 total 10.01 last 85.74) - open/close 1665 (time 1789131708.76 total 20.01 last 80.68) - open/close 2557 (time 1789131718.77 total 30.01 last 89.18) - open/close 3317 (time 1789131728.77 total 40.01 last 75.98) - open/close 4116 (time 1789131738.77 total 50.02 last 79.88) total: 4500 open/close in 54.49 seconds: 82.59 ops/second - /unlink 1506 (time 1789131757.70 total 10.00 last 150.56) - /unlink 3057 (time 1789131767.70 total 20.00 last 155.10) total: 4500 /unlink in 28.85 seconds: 155.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 798 (time 1789131790.41 total 10.01 last 79.72) - open/close 1606 (time 1789131800.43 total 20.03 last 80.66) - open/close 2449 (time 1789131810.43 total 30.04 last 84.23) - open/close 3220 (time 1789131820.44 total 40.04 last 77.09) - open/close 4010 (time 1789131830.44 total 50.04 last 78.96) total: 4500 open/close in 55.92 seconds: 80.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1492 (time 1789131850.33 total 10.01 last 149.09) - /unlink 2761 (time 1789131860.34 total 20.01 last 126.84) - /unlink 3979 (time 1789131870.34 total 30.01 last 121.80) total: 4500 /unlink in 33.29 seconds: 135.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 648 (time 1789131888.12 total 10.01 last 64.75) - open/close 1422 (time 1789131898.13 total 20.02 last 77.34) - open/close 2142 (time 1789131908.13 total 30.02 last 72.00) - open/close 3034 (time 1789131918.14 total 40.03 last 89.09) - open/close 3876 (time 1789131928.15 total 50.04 last 84.12) total: 4500 open/close in 58.68 seconds: 76.69 ops/second - /unlink 1516 (time 1789131951.07 total 10.00 last 151.54) - /unlink 3156 (time 1789131961.07 total 20.01 last 163.94) total: 4500 /unlink in 28.48 seconds: 158.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 843 (time 1789131983.18 total 10.00 last 84.27) - open/close 1720 (time 1789131993.18 total 20.00 last 87.69) - open/close 2598 (time 1789132003.19 total 30.01 last 87.72) - open/close 3385 (time 1789132013.19 total 40.02 last 78.68) - open/close 4202 (time 1789132023.20 total 50.03 last 81.62) total: 4500 open/close in 54.37 seconds: 82.76 ops/second - /unlink 1529 (time 1789132041.80 total 10.00 last 152.83) - /unlink 3172 (time 1789132051.80 total 20.01 last 164.25) total: 4500 /unlink in 28.44 seconds: 158.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 752 (time 1789132075.76 total 10.01 last 75.16) - open/close 1433 (time 1789132085.80 total 20.04 last 67.85) - open/close 2301 (time 1789132095.80 total 30.05 last 86.74) - open/close 2979 (time 1789132105.82 total 40.07 last 67.68) - open/close 3865 (time 1789132115.83 total 50.07 last 88.54) total: 4500 open/close in 58.11 seconds: 77.44 ops/second - /unlink 1654 (time 1789132138.47 total 10.01 last 165.19) - /unlink 2985 (time 1789132148.47 total 20.02 last 133.05) - /unlink 4352 (time 1789132158.47 total 30.02 last 136.68) total: 4500 /unlink in 31.06 seconds: 144.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 833 (time 1789132174.11 total 10.01 last 83.23) - open/close 1561 (time 1789132184.13 total 20.03 last 72.63) - open/close 2418 (time 1789132194.14 total 30.04 last 85.65) - open/close 3290 (time 1789132204.15 total 40.05 last 87.09) - open/close 4147 (time 1789132214.16 total 50.06 last 85.64) total: 4500 open/close in 53.87 seconds: 83.53 ops/second - /unlink 1576 (time 1789132231.81 total 10.00 last 157.58) - /unlink 3065 (time 1789132241.81 total 20.00 last 148.87) total: 4500 /unlink in 29.33 seconds: 153.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 870 (time 1789132265.39 total 10.01 last 86.95) - open/close 1634 (time 1789132275.39 total 20.01 last 76.37) - open/close 2536 (time 1789132285.40 total 30.01 last 90.16) - open/close 3363 (time 1789132295.40 total 40.02 last 82.66) - open/close 4253 (time 1789132305.41 total 50.03 last 88.89) total: 4500 open/close in 52.62 seconds: 85.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1653 (time 1789132321.64 total 10.00 last 165.27) - /unlink 3160 (time 1789132331.64 total 20.01 last 150.64) total: 4500 /unlink in 28.08 seconds: 160.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 778 (time 1789132354.35 total 10.01 last 77.75) - open/close 1597 (time 1789132364.36 total 20.01 last 81.87) - open/close 2295 (time 1789132374.39 total 30.04 last 69.60) - open/close 3104 (time 1789132384.39 total 40.04 last 80.89) - open/close 3857 (time 1789132394.39 total 50.04 last 75.29) total: 4500 open/close in 57.17 seconds: 78.71 ops/second - /unlink 1656 (time 1789132415.56 total 10.00 last 165.57) - /unlink 3283 (time 1789132425.56 total 20.00 last 162.66) total: 4500 /unlink in 28.12 seconds: 160.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 754 (time 1789132448.68 total 10.01 last 75.35) - open/close 1616 (time 1789132458.69 total 20.02 last 86.08) - open/close 2368 (time 1789132468.70 total 30.02 last 75.19) - open/close 3248 (time 1789132478.70 total 40.03 last 87.95) - open/close 4123 (time 1789132488.70 total 50.03 last 87.49) total: 4500 open/close in 54.39 seconds: 82.74 ops/second - /unlink 1592 (time 1789132508.08 total 10.00 last 159.18) - /unlink 2992 (time 1789132518.08 total 20.00 last 139.99) total: 4500 /unlink in 28.73 seconds: 156.63 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95319 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg337-server unloading modules via unload_modules_local on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local oleg337-server: modules unloaded.