== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:07:57 (1781237277) Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg418-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg418-server oleg418-server: oleg418-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg418-server: detected 4 online CPUs by sysfs oleg418-server: MODOPTS_LIBCFS= oleg418-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg418-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg418-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 588 (time 1781237453.32 total 10.00 last 58.79) - open/close 1125 (time 1781237463.35 total 20.03 last 53.56) - open/close 1620 (time 1781237473.35 total 30.03 last 49.49) - open/close 2160 (time 1781237483.36 total 40.04 last 53.96) - open/close 2760 (time 1781237493.39 total 50.07 last 59.81) - open/close 3296 (time 1781237503.40 total 60.08 last 53.54) - open/close 3862 (time 1781237513.41 total 70.09 last 56.55) - open/close 4457 (time 1781237523.43 total 80.10 last 59.43) total: 4500 open/close in 80.84 seconds: 55.67 ops/second - /unlink 1076 (time 1781237539.24 total 10.01 last 107.54) - /unlink 2109 (time 1781237549.24 total 20.01 last 103.29) - /unlink 3067 (time 1781237559.26 total 30.03 last 95.59) - /unlink 4055 (time 1781237569.27 total 40.03 last 98.76) total: 4500 /unlink in 44.38 seconds: 101.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 526 (time 1781237588.85 total 10.01 last 52.54) - open/close 1143 (time 1781237598.86 total 20.02 last 61.62) - open/close 1734 (time 1781237608.86 total 30.03 last 59.08) - open/close 2305 (time 1781237618.87 total 40.04 last 57.05) - open/close 2878 (time 1781237628.88 total 50.05 last 57.23) - open/close 3407 (time 1781237638.89 total 60.06 last 52.87) - open/close 3958 (time 1781237648.90 total 70.07 last 55.02) - open/close 4471 (time 1781237658.91 total 80.08 last 51.24) total: 4500 open/close in 80.62 seconds: 55.82 ops/second - /unlink 952 (time 1781237675.71 total 10.02 last 95.03) - /unlink 1902 (time 1781237685.71 total 20.02 last 94.98) - /unlink 2805 (time 1781237695.72 total 30.03 last 90.25) - /unlink 3719 (time 1781237705.72 total 40.03 last 91.36) total: 4500 /unlink in 49.05 seconds: 91.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 480 (time 1781237731.23 total 10.02 last 47.92) - open/close 1028 (time 1781237741.23 total 20.02 last 54.80) - open/close 1564 (time 1781237751.24 total 30.03 last 53.56) - open/close 2134 (time 1781237761.25 total 40.04 last 56.91) - open/close 2599 (time 1781237771.26 total 50.05 last 46.47) - open/close 3113 (time 1781237781.27 total 60.06 last 51.36) - open/close 3649 (time 1781237791.28 total 70.07 last 53.55) - open/close 4234 (time 1781237801.28 total 80.07 last 58.46) total: 4500 open/close in 84.74 seconds: 53.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1047 (time 1781237820.64 total 10.01 last 104.63) - /unlink 2051 (time 1781237830.65 total 20.02 last 100.28) - /unlink 3073 (time 1781237840.66 total 30.03 last 102.10) - /unlink 4019 (time 1781237850.66 total 40.03 last 94.56) total: 4500 /unlink in 44.53 seconds: 101.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 612 (time 1781237871.10 total 10.01 last 61.16) - open/close 1171 (time 1781237881.11 total 20.02 last 55.84) - open/close 1749 (time 1781237891.12 total 30.03 last 57.73) - open/close 2346 (time 1781237901.13 total 40.04 last 59.66) - open/close 2916 (time 1781237911.13 total 50.04 last 56.97) - open/close 3556 (time 1781237921.15 total 60.06 last 63.88) - open/close 4141 (time 1781237931.15 total 70.06 last 58.48) total: 4500 open/close in 76.33 seconds: 58.95 ops/second - /unlink 995 (time 1781237952.98 total 10.00 last 99.48) - /unlink 2004 (time 1781237962.99 total 20.01 last 100.83) - /unlink 2911 (time 1781237973.00 total 30.02 last 90.63) - /unlink 3953 (time 1781237983.00 total 40.02 last 104.15) total: 4500 /unlink in 45.87 seconds: 98.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 453 (time 1781238003.46 total 10.01 last 45.25) - open/close 1007 (time 1781238013.48 total 20.03 last 55.32) - open/close 1546 (time 1781238023.50 total 30.04 last 53.81) - open/close 2125 (time 1781238033.51 total 40.06 last 57.81) - open/close 2743 (time 1781238043.53 total 50.08 last 61.67) - open/close 3291 (time 1781238053.55 total 60.09 last 54.73) - open/close 3986 (time 1781238063.55 total 70.10 last 69.46) total: 4500 open/close in 78.69 seconds: 57.19 ops/second - /unlink 1027 (time 1781238087.09 total 10.01 last 102.58) - /unlink 2177 (time 1781238097.09 total 20.01 last 114.97) - /unlink 3204 (time 1781238107.10 total 30.02 last 102.65) - /unlink 4197 (time 1781238117.10 total 40.03 last 99.23) total: 4500 /unlink in 43.26 seconds: 104.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 592 (time 1781238135.96 total 10.00 last 59.20) - open/close 1210 (time 1781238145.98 total 20.02 last 61.68) - open/close 1812 (time 1781238156.01 total 30.05 last 60.04) - open/close 2467 (time 1781238166.01 total 40.05 last 65.50) - open/close 3107 (time 1781238176.01 total 50.05 last 63.99) - open/close 3678 (time 1781238186.02 total 60.05 last 57.08) - open/close 4282 (time 1781238196.02 total 70.06 last 60.37) total: 4500 open/close in 74.09 seconds: 60.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 916 (time 1781238216.20 total 10.01 last 91.51) - /unlink 1930 (time 1781238226.21 total 20.02 last 101.32) - /unlink 2953 (time 1781238236.21 total 30.02 last 102.26) - /unlink 3986 (time 1781238246.22 total 40.03 last 103.24) total: 4500 /unlink in 44.84 seconds: 100.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 584 (time 1781238266.18 total 10.01 last 58.34) - open/close 1222 (time 1781238276.18 total 20.02 last 63.76) - open/close 1752 (time 1781238286.19 total 30.03 last 52.95) - open/close 2289 (time 1781238296.20 total 40.03 last 53.67) - open/close 2855 (time 1781238306.22 total 50.05 last 56.47) - open/close 3356 (time 1781238316.24 total 60.08 last 49.99) - open/close 3983 (time 1781238326.25 total 70.08 last 62.65) - open/close 4460 (time 1781238336.25 total 80.09 last 47.69) total: 4500 open/close in 80.75 seconds: 55.73 ops/second - /unlink 1029 (time 1781238352.95 total 10.02 last 102.67) - /unlink 2027 (time 1781238362.95 total 20.03 last 99.73) - /unlink 3108 (time 1781238372.96 total 30.03 last 108.08) - /unlink 4179 (time 1781238382.96 total 40.03 last 107.08) total: 4500 /unlink in 43.11 seconds: 104.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 605 (time 1781238400.63 total 10.01 last 60.47) - open/close 1153 (time 1781238410.65 total 20.02 last 54.71) - open/close 1722 (time 1781238420.66 total 30.03 last 56.82) - open/close 2339 (time 1781238430.67 total 40.04 last 61.67) - open/close 2952 (time 1781238440.68 total 50.05 last 61.22) - open/close 3577 (time 1781238450.68 total 60.06 last 62.49) - open/close 4182 (time 1781238460.70 total 70.07 last 60.41) total: 4500 open/close in 75.14 seconds: 59.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1040 (time 1781238480.81 total 10.00 last 103.96) - /unlink 2063 (time 1781238490.82 total 20.02 last 102.18) - /unlink 2890 (time 1781238500.84 total 30.03 last 82.57) - /unlink 3765 (time 1781238510.84 total 40.04 last 87.46) total: 4500 /unlink in 47.89 seconds: 93.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 605 (time 1781238534.34 total 10.01 last 60.44) - open/close 1220 (time 1781238544.34 total 20.01 last 61.49) - open/close 1820 (time 1781238554.35 total 30.02 last 59.94) - open/close 2484 (time 1781238564.36 total 40.03 last 66.37) - open/close 3116 (time 1781238574.37 total 50.04 last 63.13) - open/close 3756 (time 1781238584.37 total 60.04 last 63.98) - open/close 4337 (time 1781238594.40 total 70.07 last 57.94) total: 4500 open/close in 72.94 seconds: 61.69 ops/second - /unlink 1087 (time 1781238612.50 total 10.00 last 108.68) - /unlink 2154 (time 1781238622.50 total 20.00 last 106.70) - /unlink 3179 (time 1781238632.50 total 30.01 last 102.47) - /unlink 4204 (time 1781238642.51 total 40.01 last 102.45) total: 4500 /unlink in 43.06 seconds: 104.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 609 (time 1781238660.24 total 10.00 last 60.88) - open/close 1178 (time 1781238670.27 total 20.03 last 56.76) - open/close 1703 (time 1781238680.27 total 30.03 last 52.48) - open/close 2281 (time 1781238690.28 total 40.03 last 57.78) - open/close 2857 (time 1781238700.30 total 50.06 last 57.45) - open/close 3477 (time 1781238710.31 total 60.07 last 61.96) - open/close 4076 (time 1781238720.33 total 70.09 last 59.76) total: 4500 open/close in 76.95 seconds: 58.48 ops/second - /unlink 1006 (time 1781238742.17 total 10.00 last 100.56) - /unlink 1973 (time 1781238752.18 total 20.01 last 96.61) - /unlink 2949 (time 1781238762.19 total 30.03 last 97.46) - /unlink 4029 (time 1781238772.21 total 40.05 last 107.78) total: 4500 /unlink in 44.21 seconds: 101.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 563 (time 1781238792.19 total 10.01 last 56.27) - open/close 1152 (time 1781238802.20 total 20.02 last 58.84) - open/close 1738 (time 1781238812.20 total 30.02 last 58.57) - open/close 2288 (time 1781238822.21 total 40.02 last 54.98) - open/close 2845 (time 1781238832.22 total 50.03 last 55.65) - open/close 3406 (time 1781238842.23 total 60.05 last 56.00) - open/close 3964 (time 1781238852.24 total 70.05 last 55.78) total: 4500 open/close in 78.85 seconds: 57.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1021 (time 1781238875.53 total 10.01 last 101.98) - /unlink 2048 (time 1781238885.53 total 20.02 last 102.63) - /unlink 3046 (time 1781238895.54 total 30.02 last 99.79) - /unlink 4034 (time 1781238905.54 total 40.02 last 98.79) total: 4500 /unlink in 44.68 seconds: 100.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 576 (time 1781238925.25 total 10.04 last 57.39) - open/close 1165 (time 1781238935.27 total 20.05 last 58.81) - open/close 1786 (time 1781238945.28 total 30.06 last 62.04) - open/close 2361 (time 1781238955.29 total 40.07 last 57.47) - open/close 2963 (time 1781238965.31 total 50.09 last 60.06) - open/close 3576 (time 1781238975.32 total 60.10 last 61.26) - open/close 4191 (time 1781238985.34 total 70.12 last 61.35) total: 4500 open/close in 75.41 seconds: 59.68 ops/second - /unlink 1078 (time 1781239005.95 total 10.00 last 107.76) - /unlink 2078 (time 1781239015.95 total 20.01 last 99.94) - /unlink 3045 (time 1781239025.95 total 30.01 last 96.69) - /unlink 3925 (time 1781239035.95 total 40.01 last 87.99) total: 4500 /unlink in 46.46 seconds: 96.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 559 (time 1781239057.65 total 10.01 last 55.83) - open/close 1125 (time 1781239067.65 total 20.02 last 56.58) - open/close 1745 (time 1781239077.67 total 30.03 last 61.89) - open/close 2345 (time 1781239087.67 total 40.04 last 59.95) - open/close 2906 (time 1781239097.69 total 50.06 last 55.99) - open/close 3458 (time 1781239107.72 total 60.09 last 55.07) - open/close 4044 (time 1781239117.74 total 70.11 last 58.46) total: 4500 open/close in 78.87 seconds: 57.06 ops/second - /unlink 1022 (time 1781239141.58 total 10.00 last 102.15) - /unlink 1942 (time 1781239151.59 total 20.01 last 91.98) - /unlink 2757 (time 1781239161.60 total 30.02 last 81.36) - /unlink 3422 (time 1781239171.62 total 40.04 last 66.41) - /unlink 4321 (time 1781239181.62 total 50.04 last 89.88) total: 4500 /unlink in 52.13 seconds: 86.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 573 (time 1781239199.62 total 10.01 last 57.25) - open/close 1168 (time 1781239209.62 total 20.01 last 59.50) - open/close 1746 (time 1781239219.64 total 30.03 last 57.70) - open/close 2343 (time 1781239229.64 total 40.03 last 59.70) - open/close 2950 (time 1781239239.65 total 50.04 last 60.65) - open/close 3547 (time 1781239249.65 total 60.04 last 59.66) - open/close 4083 (time 1781239259.67 total 70.06 last 53.53) total: 4500 open/close in 76.73 seconds: 58.65 ops/second - /unlink 982 (time 1781239281.58 total 10.01 last 98.14) - /unlink 1933 (time 1781239291.58 total 20.01 last 95.09) - /unlink 2901 (time 1781239301.59 total 30.02 last 96.70) - /unlink 3940 (time 1781239311.60 total 40.03 last 103.82) total: 4500 /unlink in 45.34 seconds: 99.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 523 (time 1781239332.38 total 10.01 last 52.26) - open/close 1083 (time 1781239342.39 total 20.02 last 55.95) - open/close 1668 (time 1781239352.41 total 30.04 last 58.37) - open/close 2148 (time 1781239362.42 total 40.04 last 47.99) - open/close 2726 (time 1781239372.47 total 50.09 last 57.49) - open/close 3264 (time 1781239382.49 total 60.11 last 53.68) - open/close 3814 (time 1781239392.49 total 70.12 last 54.99) - open/close 4388 (time 1781239402.50 total 80.12 last 57.39) total: 4500 open/close in 82.33 seconds: 54.66 ops/second - /unlink 953 (time 1781239420.39 total 10.01 last 95.18) - /unlink 1963 (time 1781239430.39 total 20.02 last 100.95) - /unlink 2887 (time 1781239440.40 total 30.03 last 92.32) - /unlink 3934 (time 1781239450.41 total 40.03 last 104.63) total: 4500 /unlink in 46.35 seconds: 97.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 578 (time 1781239471.43 total 10.01 last 57.73) - open/close 1147 (time 1781239481.43 total 20.01 last 56.89) - open/close 1714 (time 1781239491.44 total 30.02 last 56.67) - open/close 2243 (time 1781239501.45 total 40.03 last 52.87) - open/close 2802 (time 1781239511.45 total 50.03 last 55.89) - open/close 3332 (time 1781239521.46 total 60.04 last 52.95) - open/close 3872 (time 1781239531.47 total 70.05 last 53.93) - open/close 4405 (time 1781239541.49 total 80.07 last 53.21) total: 4500 open/close in 81.71 seconds: 55.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 957 (time 1781239558.73 total 10.01 last 95.60) - /unlink 1951 (time 1781239568.73 total 20.01 last 99.39) - /unlink 2918 (time 1781239578.74 total 30.02 last 96.63) - /unlink 3812 (time 1781239588.74 total 40.03 last 89.34) total: 4500 /unlink in 47.70 seconds: 94.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 513 (time 1781239611.92 total 10.01 last 51.26) - open/close 1086 (time 1781239621.92 total 20.01 last 57.29) - open/close 1684 (time 1781239631.93 total 30.02 last 59.75) - open/close 2286 (time 1781239641.94 total 40.03 last 60.14) - open/close 2893 (time 1781239651.95 total 50.04 last 60.66) - open/close 3503 (time 1781239661.96 total 60.05 last 60.95) - open/close 4091 (time 1781239671.96 total 70.05 last 58.78) total: 4500 open/close in 77.49 seconds: 58.07 ops/second - /unlink 1072 (time 1781239694.21 total 10.01 last 107.09) - /unlink 2037 (time 1781239704.22 total 20.02 last 96.42) - /unlink 3089 (time 1781239714.22 total 30.02 last 105.20) - /unlink 4111 (time 1781239724.22 total 40.02 last 102.15) total: 4500 /unlink in 43.75 seconds: 102.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 578 (time 1781239743.54 total 10.03 last 57.62) - open/close 1092 (time 1781239753.56 total 20.05 last 51.32) - open/close 1583 (time 1781239763.57 total 30.06 last 49.05) - open/close 2068 (time 1781239773.57 total 40.06 last 48.49) - open/close 2628 (time 1781239783.58 total 50.07 last 55.96) - open/close 3147 (time 1781239793.58 total 60.07 last 51.88) - open/close 3686 (time 1781239803.58 total 70.07 last 53.89) - open/close 4169 (time 1781239813.62 total 80.11 last 48.13) total: 4500 open/close in 89.29 seconds: 50.40 ops/second - /unlink 955 (time 1781239839.54 total 10.00 last 95.45) - /unlink 1746 (time 1781239849.55 total 20.01 last 79.06) - /unlink 2665 (time 1781239859.55 total 30.01 last 91.86) - /unlink 3613 (time 1781239869.55 total 40.02 last 94.76) total: 4500 /unlink in 49.07 seconds: 91.70 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11209: 74821 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server unloading modules via unload_modules_local on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local oleg418-server: modules unloaded.