== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:11:47 (1788361907) Stopping clients: oleg125-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg125-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg125-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg125-server oleg125-server: oleg125-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg125-server: detected 4 online CPUs by sysfs oleg125-server: MODOPTS_LIBCFS= oleg125-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg125-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg125-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 887 (time 1788362054.30 total 10.01 last 88.63) - open/close 1793 (time 1788362064.31 total 20.02 last 90.53) - open/close 2671 (time 1788362074.32 total 30.03 last 87.71) - open/close 3549 (time 1788362084.32 total 40.03 last 87.74) - open/close 4400 (time 1788362094.34 total 50.05 last 84.96) total: 4500 open/close in 51.22 seconds: 87.85 ops/second - /unlink 1603 (time 1788362109.13 total 10.00 last 160.28) - /unlink 3240 (time 1788362119.13 total 20.00 last 163.67) total: 4500 /unlink in 27.31 seconds: 164.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 833 (time 1788362139.20 total 10.00 last 83.30) - open/close 1788 (time 1788362149.21 total 20.01 last 95.43) - open/close 2801 (time 1788362159.21 total 30.01 last 101.25) - open/close 3822 (time 1788362169.22 total 40.02 last 102.07) total: 4500 open/close in 46.93 seconds: 95.89 ops/second - /unlink 1901 (time 1788362188.86 total 10.00 last 190.03) - /unlink 3794 (time 1788362198.86 total 20.01 last 189.16) total: 4500 /unlink in 23.85 seconds: 188.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 1005 (time 1788362216.28 total 10.01 last 100.41) - open/close 1983 (time 1788362226.28 total 20.01 last 97.80) - open/close 2957 (time 1788362236.29 total 30.02 last 97.33) - open/close 3968 (time 1788362246.29 total 40.02 last 101.07) total: 4500 open/close in 45.35 seconds: 99.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1455 (time 1788362264.72 total 10.00 last 145.45) - /unlink 2972 (time 1788362274.72 total 20.01 last 151.59) total: 4500 /unlink in 28.33 seconds: 158.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 920 (time 1788362295.94 total 10.00 last 91.98) - open/close 1795 (time 1788362305.95 total 20.01 last 87.43) - open/close 2731 (time 1788362315.96 total 30.02 last 93.47) - open/close 3610 (time 1788362325.96 total 40.02 last 87.89) total: 4500 open/close in 49.62 seconds: 90.69 ops/second - /unlink 1896 (time 1788362348.76 total 10.00 last 189.57) - /unlink 3878 (time 1788362358.77 total 20.00 last 198.16) total: 4500 /unlink in 23.38 seconds: 192.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 1086 (time 1788362374.90 total 10.01 last 108.45) - open/close 2158 (time 1788362384.90 total 20.01 last 107.19) - open/close 3148 (time 1788362394.90 total 30.02 last 98.99) - open/close 4133 (time 1788362404.91 total 40.02 last 98.41) total: 4500 open/close in 44.05 seconds: 102.15 ops/second - /unlink 1662 (time 1788362422.15 total 10.01 last 166.11) - /unlink 3396 (time 1788362432.15 total 20.01 last 173.39) total: 4500 /unlink in 26.51 seconds: 169.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 995 (time 1788362451.78 total 10.01 last 99.43) - open/close 1982 (time 1788362461.78 total 20.01 last 98.69) - open/close 2988 (time 1788362471.78 total 30.01 last 100.59) - open/close 3950 (time 1788362481.78 total 40.01 last 96.19) total: 4500 open/close in 45.25 seconds: 99.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1928 (time 1788362500.00 total 10.00 last 192.74) - /unlink 3763 (time 1788362510.00 total 20.00 last 183.47) total: 4500 /unlink in 24.11 seconds: 186.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 1157 (time 1788362527.42 total 10.01 last 115.61) - open/close 2211 (time 1788362537.43 total 20.02 last 105.30) - open/close 3401 (time 1788362547.43 total 30.02 last 118.99) total: 4500 open/close in 39.98 seconds: 112.56 ops/second - /unlink 2043 (time 1788362569.89 total 10.00 last 204.25) - /unlink 3906 (time 1788362579.90 total 20.01 last 186.15) total: 4500 /unlink in 23.31 seconds: 193.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 1043 (time 1788362596.50 total 10.01 last 104.24) - open/close 2218 (time 1788362606.51 total 20.01 last 117.44) - open/close 3327 (time 1788362616.51 total 30.01 last 110.87) - open/close 4453 (time 1788362626.51 total 40.02 last 112.53) total: 4500 open/close in 40.39 seconds: 111.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 2394 (time 1788362639.12 total 10.00 last 239.37) total: 4500 /unlink in 15.18 seconds: 296.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1603 (time 1788362656.00 total 10.00 last 160.29) - open/close 3017 (time 1788362666.01 total 20.01 last 141.24) total: 4500 open/close in 27.77 seconds: 162.06 ops/second - /unlink 2623 (time 1788362685.42 total 10.02 last 261.86) - /unlink 4100 (time 1788362695.43 total 20.03 last 147.56) total: 4500 /unlink in 22.87 seconds: 196.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 813 (time 1788362712.81 total 10.01 last 81.22) - open/close 1543 (time 1788362722.81 total 20.01 last 72.98) - open/close 2331 (time 1788362732.82 total 30.03 last 78.70) - open/close 3193 (time 1788362742.82 total 40.03 last 86.19) - open/close 4067 (time 1788362752.83 total 50.03 last 87.37) total: 4500 open/close in 54.49 seconds: 82.58 ops/second - /unlink 1839 (time 1788362770.01 total 10.01 last 183.79) - /unlink 3869 (time 1788362780.01 total 20.01 last 203.00) total: 4500 /unlink in 23.21 seconds: 193.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1035 (time 1788362795.78 total 10.00 last 103.48) - open/close 1992 (time 1788362805.79 total 20.00 last 95.67) - open/close 2909 (time 1788362815.79 total 30.01 last 91.67) - open/close 3912 (time 1788362825.80 total 40.02 last 100.18) total: 4500 open/close in 46.52 seconds: 96.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1932 (time 1788362845.45 total 10.00 last 193.14) - /unlink 3712 (time 1788362855.45 total 20.00 last 177.97) total: 4500 /unlink in 24.43 seconds: 184.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 911 (time 1788362873.16 total 10.01 last 91.05) - open/close 1875 (time 1788362883.17 total 20.01 last 96.36) - open/close 2843 (time 1788362893.17 total 30.01 last 96.80) - open/close 3815 (time 1788362903.17 total 40.01 last 97.17) total: 4500 open/close in 46.67 seconds: 96.41 ops/second - /unlink 1898 (time 1788362922.54 total 10.00 last 189.76) - /unlink 3709 (time 1788362932.54 total 20.01 last 181.02) total: 4500 /unlink in 24.41 seconds: 184.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 930 (time 1788362950.00 total 10.00 last 92.99) - open/close 1967 (time 1788362960.00 total 20.00 last 103.67) - open/close 2891 (time 1788362970.01 total 30.02 last 92.30) - open/close 3836 (time 1788362980.01 total 40.02 last 94.50) total: 4500 open/close in 46.61 seconds: 96.54 ops/second - /unlink 1760 (time 1788363000.00 total 10.01 last 175.88) - /unlink 3579 (time 1788363010.00 total 20.01 last 181.90) total: 4500 /unlink in 25.06 seconds: 179.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 995 (time 1788363027.99 total 10.01 last 99.39) - open/close 1891 (time 1788363038.00 total 20.01 last 89.58) - open/close 2699 (time 1788363048.01 total 30.03 last 80.68) - open/close 3689 (time 1788363058.01 total 40.03 last 98.97) total: 4500 open/close in 49.38 seconds: 91.14 ops/second - /unlink 1727 (time 1788363080.43 total 10.00 last 172.63) - /unlink 3320 (time 1788363090.44 total 20.01 last 159.25) total: 4500 /unlink in 27.06 seconds: 166.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 880 (time 1788363109.98 total 10.01 last 87.93) - open/close 1751 (time 1788363119.99 total 20.02 last 87.03) - open/close 2678 (time 1788363129.99 total 30.02 last 92.65) - open/close 3472 (time 1788363140.00 total 40.02 last 79.37) - open/close 4328 (time 1788363150.01 total 50.03 last 85.52) total: 4500 open/close in 52.15 seconds: 86.28 ops/second - /unlink 1716 (time 1788363165.80 total 10.00 last 171.60) - /unlink 3497 (time 1788363175.81 total 20.01 last 177.99) total: 4500 /unlink in 25.60 seconds: 175.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 973 (time 1788363194.30 total 10.01 last 97.22) - open/close 1878 (time 1788363204.31 total 20.02 last 90.42) - open/close 2812 (time 1788363214.32 total 30.03 last 93.31) - open/close 3708 (time 1788363224.33 total 40.04 last 89.48) total: 4500 open/close in 48.38 seconds: 93.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2001 (time 1788363244.96 total 10.00 last 200.04) - /unlink 3665 (time 1788363254.97 total 20.01 last 166.29) total: 4500 /unlink in 24.54 seconds: 183.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1081 (time 1788363272.04 total 10.00 last 108.05) - open/close 2126 (time 1788363282.06 total 20.02 last 104.29) - open/close 3048 (time 1788363292.06 total 30.03 last 92.17) - open/close 4037 (time 1788363302.07 total 40.04 last 98.79) total: 4500 open/close in 45.01 seconds: 99.98 ops/second - /unlink 1855 (time 1788363319.81 total 10.00 last 185.43) - /unlink 3535 (time 1788363329.81 total 20.01 last 167.97) total: 4500 /unlink in 26.21 seconds: 171.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 932 (time 1788363349.33 total 10.00 last 93.18) - open/close 1876 (time 1788363359.34 total 20.01 last 94.34) - open/close 2827 (time 1788363369.34 total 30.01 last 95.04) - open/close 3784 (time 1788363379.35 total 40.02 last 95.61) total: 4500 open/close in 46.44 seconds: 96.89 ops/second - /unlink 1781 (time 1788363398.61 total 10.00 last 178.07) - /unlink 3681 (time 1788363408.61 total 20.00 last 189.95) total: 4500 /unlink in 24.44 seconds: 184.10 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 94942 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules via unload_modules_local on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local oleg125-server: modules unloaded.