== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:25:29 (1787405129) Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg210-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg210-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg210-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg210-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 859 (time 1787405343.12 total 10.00 last 85.88) - open/close 1719 (time 1787405353.12 total 20.00 last 85.98) - open/close 2491 (time 1787405363.15 total 30.03 last 77.00) - open/close 3171 (time 1787405373.17 total 40.05 last 67.87) - open/close 3927 (time 1787405383.17 total 50.05 last 75.57) total: 4500 open/close in 56.34 seconds: 79.87 ops/second - /unlink 1315 (time 1787405404.92 total 10.01 last 131.33) - /unlink 2766 (time 1787405414.93 total 20.02 last 145.01) - /unlink 4116 (time 1787405424.93 total 30.02 last 134.93) total: 4500 /unlink in 34.29 seconds: 131.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 937 (time 1787405444.30 total 10.01 last 93.64) - open/close 1574 (time 1787405454.30 total 20.01 last 63.69) - open/close 2214 (time 1787405464.31 total 30.01 last 63.95) - open/close 3106 (time 1787405474.32 total 40.03 last 89.05) - open/close 3761 (time 1787405484.33 total 50.03 last 65.48) total: 4500 open/close in 59.26 seconds: 75.94 ops/second - /unlink 1608 (time 1787405507.87 total 10.00 last 160.72) - /unlink 3476 (time 1787405517.87 total 20.01 last 186.77) total: 4500 /unlink in 26.38 seconds: 170.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 877 (time 1787405538.59 total 10.01 last 87.62) - open/close 1699 (time 1787405548.60 total 20.01 last 82.18) - open/close 2502 (time 1787405558.61 total 30.02 last 80.21) - open/close 3206 (time 1787405568.62 total 40.03 last 70.36) - open/close 3998 (time 1787405578.62 total 50.04 last 79.14) total: 4500 open/close in 57.60 seconds: 78.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1641 (time 1787405600.69 total 10.00 last 164.05) - /unlink 2988 (time 1787405610.69 total 20.01 last 134.67) - /unlink 4454 (time 1787405620.70 total 30.01 last 146.55) total: 4500 /unlink in 30.35 seconds: 148.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 903 (time 1787405635.81 total 10.00 last 90.27) - open/close 1553 (time 1787405645.82 total 20.02 last 64.92) - open/close 2308 (time 1787405655.83 total 30.03 last 75.42) - open/close 3247 (time 1787405665.84 total 40.04 last 93.80) - open/close 4121 (time 1787405675.85 total 50.05 last 87.31) total: 4500 open/close in 54.05 seconds: 83.25 ops/second - /unlink 1706 (time 1787405694.01 total 10.01 last 170.48) - /unlink 3284 (time 1787405704.01 total 20.01 last 157.77) total: 4500 /unlink in 27.30 seconds: 164.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 806 (time 1787405726.39 total 10.01 last 80.53) - open/close 1604 (time 1787405736.41 total 20.02 last 79.69) - open/close 2326 (time 1787405746.41 total 30.02 last 72.19) - open/close 2951 (time 1787405756.41 total 40.02 last 62.49) - open/close 3743 (time 1787405766.42 total 50.03 last 79.17) total: 4500 open/close in 58.20 seconds: 77.32 ops/second - /unlink 1540 (time 1787405789.20 total 10.01 last 153.84) - /unlink 3045 (time 1787405799.20 total 20.01 last 150.50) total: 4500 /unlink in 29.60 seconds: 152.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 757 (time 1787405823.86 total 10.01 last 75.64) - open/close 1493 (time 1787405833.87 total 20.02 last 73.55) - open/close 2274 (time 1787405843.87 total 30.02 last 78.04) - open/close 2940 (time 1787405853.89 total 40.04 last 66.49) - open/close 3801 (time 1787405863.89 total 50.04 last 86.06) total: 4500 open/close in 57.53 seconds: 78.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1455 (time 1787405885.70 total 10.00 last 145.50) - /unlink 3219 (time 1787405895.70 total 20.00 last 176.34) total: 4500 /unlink in 28.45 seconds: 158.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 817 (time 1787405920.13 total 10.00 last 81.67) - open/close 1579 (time 1787405930.16 total 20.03 last 75.97) - open/close 2178 (time 1787405940.18 total 30.05 last 59.81) - open/close 2961 (time 1787405950.18 total 40.05 last 78.26) - open/close 3799 (time 1787405960.19 total 50.06 last 83.71) total: 4500 open/close in 57.65 seconds: 78.06 ops/second - /unlink 1494 (time 1787405982.17 total 10.00 last 149.38) - /unlink 2971 (time 1787405992.18 total 20.00 last 147.68) total: 4500 /unlink in 29.70 seconds: 151.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 866 (time 1787406017.30 total 10.01 last 86.48) - open/close 1743 (time 1787406027.31 total 20.02 last 87.63) - open/close 2630 (time 1787406037.34 total 30.04 last 88.50) - open/close 3415 (time 1787406047.34 total 40.05 last 78.50) - open/close 4008 (time 1787406057.36 total 50.07 last 59.15) total: 4500 open/close in 57.95 seconds: 77.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1204 (time 1787406080.69 total 10.00 last 120.37) - /unlink 2566 (time 1787406090.69 total 20.01 last 136.15) - /unlink 4269 (time 1787406100.70 total 30.01 last 170.23) total: 4500 /unlink in 31.10 seconds: 144.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 801 (time 1787406115.79 total 10.00 last 80.09) - open/close 1599 (time 1787406125.79 total 20.00 last 79.79) - open/close 2402 (time 1787406135.81 total 30.02 last 80.12) - open/close 3202 (time 1787406145.82 total 40.03 last 79.93) - open/close 3965 (time 1787406155.83 total 50.04 last 76.22) total: 4500 open/close in 59.02 seconds: 76.25 ops/second - /unlink 1490 (time 1787406180.36 total 10.00 last 148.95) - /unlink 3050 (time 1787406190.36 total 20.00 last 155.99) total: 4500 /unlink in 29.09 seconds: 154.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 995 (time 1787406213.68 total 10.01 last 99.44) - open/close 2006 (time 1787406223.68 total 20.01 last 101.09) - open/close 2825 (time 1787406233.69 total 30.02 last 81.81) - open/close 3504 (time 1787406243.69 total 40.02 last 67.90) - open/close 4277 (time 1787406253.71 total 50.04 last 77.15) total: 4500 open/close in 53.97 seconds: 83.38 ops/second - /unlink 1342 (time 1787406273.30 total 10.00 last 134.18) - /unlink 2829 (time 1787406283.30 total 20.00 last 148.66) - /unlink 4421 (time 1787406293.30 total 30.01 last 159.19) total: 4500 /unlink in 30.74 seconds: 146.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 857 (time 1787406309.35 total 10.04 last 85.37) - open/close 1779 (time 1787406319.35 total 20.04 last 92.17) - open/close 2698 (time 1787406329.36 total 30.04 last 91.89) - open/close 3550 (time 1787406339.36 total 40.04 last 85.20) - open/close 4346 (time 1787406349.36 total 50.05 last 79.56) total: 4500 open/close in 51.64 seconds: 87.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1413 (time 1787406365.68 total 10.00 last 141.26) - /unlink 2454 (time 1787406375.69 total 20.01 last 104.00) - /unlink 3813 (time 1787406385.70 total 30.02 last 135.81) total: 4500 /unlink in 34.55 seconds: 130.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 844 (time 1787406405.94 total 10.01 last 84.35) - open/close 1642 (time 1787406415.96 total 20.03 last 79.64) - open/close 2477 (time 1787406425.96 total 30.03 last 83.46) - open/close 3480 (time 1787406435.96 total 40.03 last 100.29) - open/close 4355 (time 1787406445.97 total 50.04 last 87.47) total: 4500 open/close in 52.34 seconds: 85.98 ops/second - /unlink 1230 (time 1787406463.17 total 10.03 last 122.66) - /unlink 2611 (time 1787406473.18 total 20.03 last 138.01) - /unlink 4094 (time 1787406483.18 total 30.04 last 148.24) total: 4500 /unlink in 32.42 seconds: 138.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 745 (time 1787406500.10 total 10.00 last 74.47) - open/close 1620 (time 1787406510.11 total 20.01 last 87.45) - open/close 2362 (time 1787406520.12 total 30.02 last 74.16) - open/close 2901 (time 1787406530.12 total 40.02 last 53.86) - open/close 3477 (time 1787406540.15 total 50.05 last 57.47) - open/close 4318 (time 1787406550.15 total 60.05 last 84.09) total: 4500 open/close in 62.91 seconds: 71.53 ops/second - /unlink 1584 (time 1787406568.06 total 10.00 last 158.35) - /unlink 3187 (time 1787406578.06 total 20.01 last 160.26) total: 4500 /unlink in 28.88 seconds: 155.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 822 (time 1787406602.07 total 10.00 last 82.17) - open/close 1622 (time 1787406612.07 total 20.01 last 79.96) - open/close 2472 (time 1787406622.07 total 30.01 last 84.99) - open/close 3238 (time 1787406632.08 total 40.01 last 76.57) - open/close 3976 (time 1787406642.09 total 50.02 last 73.75) total: 4500 open/close in 55.67 seconds: 80.84 ops/second - /unlink 1255 (time 1787406662.14 total 10.01 last 125.32) - /unlink 2889 (time 1787406672.14 total 20.02 last 163.35) total: 4500 /unlink in 29.43 seconds: 152.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 807 (time 1787406696.13 total 10.00 last 80.68) - open/close 1522 (time 1787406706.14 total 20.01 last 71.47) - open/close 2366 (time 1787406716.14 total 30.02 last 84.33) - open/close 3158 (time 1787406726.17 total 40.04 last 79.02) - open/close 3824 (time 1787406736.18 total 50.05 last 66.55) total: 4500 open/close in 59.90 seconds: 75.12 ops/second - /unlink 1522 (time 1787406761.64 total 10.00 last 152.15) - /unlink 3087 (time 1787406771.64 total 20.01 last 156.40) total: 4500 /unlink in 28.97 seconds: 155.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 823 (time 1787406796.36 total 10.01 last 82.24) - open/close 1630 (time 1787406806.36 total 20.01 last 80.68) - open/close 2401 (time 1787406816.36 total 30.01 last 77.09) - open/close 3232 (time 1787406826.36 total 40.01 last 83.08) - open/close 4158 (time 1787406836.37 total 50.02 last 92.58) total: 4500 open/close in 54.07 seconds: 83.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1594 (time 1787406854.48 total 10.00 last 159.38) - /unlink 3210 (time 1787406864.48 total 20.00 last 161.58) total: 4500 /unlink in 28.37 seconds: 158.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 657 (time 1787406888.81 total 10.00 last 65.69) - open/close 1363 (time 1787406898.81 total 20.00 last 70.58) - open/close 2204 (time 1787406908.81 total 30.00 last 84.10) - open/close 3025 (time 1787406918.82 total 40.01 last 82.03) - open/close 3996 (time 1787406928.83 total 50.02 last 97.00) total: 4500 open/close in 56.14 seconds: 80.16 ops/second - /unlink 1649 (time 1787406949.40 total 10.00 last 164.89) - /unlink 3204 (time 1787406959.40 total 20.00 last 155.47) total: 4500 /unlink in 27.62 seconds: 162.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 863 (time 1787406981.44 total 10.00 last 86.29) - open/close 1633 (time 1787406991.44 total 20.00 last 76.99) - open/close 2240 (time 1787407001.45 total 30.01 last 60.67) - open/close 2999 (time 1787407011.45 total 40.01 last 75.86) - open/close 3872 (time 1787407021.46 total 50.02 last 87.28) total: 4500 open/close in 56.94 seconds: 79.04 ops/second - /unlink 1591 (time 1787407042.12 total 10.00 last 159.05) - /unlink 3231 (time 1787407052.12 total 20.00 last 163.97) total: 4500 /unlink in 28.55 seconds: 157.63 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94920 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local oleg210-server: modules unloaded.