== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:51:01 (1786567861) Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg242-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg242-server oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg242-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 854 (time 1786568059.55 total 10.02 last 85.20) - open/close 1649 (time 1786568069.55 total 20.02 last 79.50) - open/close 2397 (time 1786568079.58 total 30.05 last 74.64) - open/close 3182 (time 1786568089.59 total 40.06 last 78.38) - open/close 3948 (time 1786568099.59 total 50.06 last 76.59) total: 4500 open/close in 58.25 seconds: 77.25 ops/second - /unlink 1302 (time 1786568121.91 total 10.01 last 130.12) - /unlink 2839 (time 1786568131.91 total 20.01 last 153.68) - /unlink 4221 (time 1786568141.91 total 30.01 last 138.18) total: 4500 /unlink in 32.01 seconds: 140.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 711 (time 1786568158.33 total 10.00 last 71.08) - open/close 1535 (time 1786568168.33 total 20.00 last 82.39) - open/close 2359 (time 1786568178.34 total 30.01 last 82.33) - open/close 3141 (time 1786568188.35 total 40.02 last 78.11) - open/close 3973 (time 1786568198.35 total 50.03 last 83.16) total: 4500 open/close in 56.18 seconds: 80.10 ops/second - /unlink 1401 (time 1786568218.92 total 10.00 last 140.04) - /unlink 2763 (time 1786568228.92 total 20.01 last 136.18) - /unlink 4101 (time 1786568238.93 total 30.01 last 133.74) total: 4500 /unlink in 33.15 seconds: 135.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 698 (time 1786568257.34 total 10.01 last 69.73) - open/close 1330 (time 1786568267.34 total 20.01 last 63.19) - open/close 2117 (time 1786568277.35 total 30.02 last 78.63) - open/close 2781 (time 1786568287.36 total 40.03 last 66.38) - open/close 3507 (time 1786568297.37 total 50.03 last 72.53) - open/close 4328 (time 1786568307.37 total 60.04 last 82.07) total: 4500 open/close in 61.87 seconds: 72.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1798 (time 1786568323.01 total 10.01 last 179.54) - /unlink 3494 (time 1786568333.02 total 20.02 last 169.46) total: 4500 /unlink in 27.14 seconds: 165.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 664 (time 1786568353.75 total 10.00 last 66.39) - open/close 1558 (time 1786568363.75 total 20.00 last 89.37) - open/close 2398 (time 1786568373.76 total 30.02 last 83.90) - open/close 3300 (time 1786568383.77 total 40.02 last 90.17) - open/close 4202 (time 1786568393.77 total 50.02 last 90.18) total: 4500 open/close in 53.33 seconds: 84.38 ops/second - /unlink 1662 (time 1786568411.53 total 10.01 last 166.10) - /unlink 3284 (time 1786568421.53 total 20.01 last 162.20) total: 4500 /unlink in 26.10 seconds: 172.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 973 (time 1786568440.60 total 10.00 last 97.26) - open/close 1968 (time 1786568450.63 total 20.03 last 99.26) - open/close 2861 (time 1786568460.63 total 30.03 last 89.26) - open/close 3771 (time 1786568470.64 total 40.04 last 90.96) total: 4500 open/close in 47.47 seconds: 94.80 ops/second - /unlink 1896 (time 1786568491.53 total 10.00 last 189.55) - /unlink 3676 (time 1786568501.53 total 20.00 last 177.96) total: 4500 /unlink in 24.39 seconds: 184.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 1025 (time 1786568519.24 total 10.00 last 102.47) - open/close 1996 (time 1786568529.25 total 20.01 last 97.00) - open/close 2975 (time 1786568539.26 total 30.01 last 97.88) - open/close 4037 (time 1786568549.26 total 40.02 last 106.17) total: 4500 open/close in 44.89 seconds: 100.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1608 (time 1786568567.14 total 10.00 last 160.77) - /unlink 3255 (time 1786568577.14 total 20.01 last 164.60) total: 4500 /unlink in 27.26 seconds: 165.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 1088 (time 1786568597.37 total 10.01 last 108.74) - open/close 2066 (time 1786568607.39 total 20.02 last 97.65) - open/close 3014 (time 1786568617.39 total 30.02 last 94.76) - open/close 3802 (time 1786568627.41 total 40.04 last 78.69) total: 4500 open/close in 48.41 seconds: 92.95 ops/second - /unlink 1972 (time 1786568649.27 total 10.00 last 197.19) - /unlink 3672 (time 1786568659.27 total 20.00 last 170.00) total: 4500 /unlink in 24.94 seconds: 180.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 928 (time 1786568678.38 total 10.00 last 92.77) - open/close 1861 (time 1786568688.39 total 20.01 last 93.23) - open/close 2703 (time 1786568698.39 total 30.02 last 84.16) - open/close 3570 (time 1786568708.40 total 40.03 last 86.58) - open/close 4476 (time 1786568718.41 total 50.04 last 90.53) total: 4500 open/close in 50.25 seconds: 89.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1594 (time 1786568732.34 total 10.00 last 159.32) - /unlink 3297 (time 1786568742.34 total 20.01 last 170.27) total: 4500 /unlink in 27.06 seconds: 166.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 926 (time 1786568762.97 total 10.00 last 92.59) - open/close 1850 (time 1786568772.97 total 20.00 last 92.39) - open/close 2820 (time 1786568782.98 total 30.01 last 96.93) - open/close 3759 (time 1786568792.98 total 40.01 last 93.87) total: 4500 open/close in 48.02 seconds: 93.71 ops/second - /unlink 1876 (time 1786568814.30 total 10.01 last 187.49) - /unlink 3661 (time 1786568824.30 total 20.01 last 178.45) total: 4500 /unlink in 24.74 seconds: 181.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 853 (time 1786568842.17 total 10.00 last 85.28) - open/close 1805 (time 1786568852.18 total 20.01 last 95.12) - open/close 2840 (time 1786568862.19 total 30.02 last 103.41) - open/close 3862 (time 1786568872.20 total 40.03 last 102.06) total: 4500 open/close in 46.30 seconds: 97.19 ops/second - /unlink 1727 (time 1786568891.43 total 10.00 last 172.66) - /unlink 3636 (time 1786568901.43 total 20.00 last 190.87) total: 4500 /unlink in 24.17 seconds: 186.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 973 (time 1786568919.79 total 10.01 last 97.23) - open/close 1941 (time 1786568929.79 total 20.01 last 96.76) - open/close 3000 (time 1786568939.79 total 30.01 last 105.87) - open/close 4046 (time 1786568949.80 total 40.02 last 104.54) total: 4500 open/close in 44.28 seconds: 101.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2115 (time 1786568966.96 total 10.01 last 211.28) - /unlink 3993 (time 1786568976.96 total 20.01 last 187.76) total: 4500 /unlink in 22.83 seconds: 197.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 934 (time 1786568993.04 total 10.01 last 93.34) - open/close 1885 (time 1786569003.05 total 20.01 last 95.05) - open/close 2909 (time 1786569013.05 total 30.01 last 102.39) - open/close 3872 (time 1786569023.05 total 40.02 last 96.25) total: 4500 open/close in 46.18 seconds: 97.45 ops/second - /unlink 2083 (time 1786569042.07 total 10.01 last 208.19) - /unlink 4075 (time 1786569052.07 total 20.01 last 199.14) total: 4500 /unlink in 22.08 seconds: 203.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 998 (time 1786569067.56 total 10.00 last 99.75) - open/close 2072 (time 1786569077.56 total 20.01 last 107.34) - open/close 2974 (time 1786569087.57 total 30.01 last 90.18) - open/close 3927 (time 1786569097.57 total 40.01 last 95.28) total: 4500 open/close in 45.39 seconds: 99.15 ops/second - /unlink 1905 (time 1786569116.15 total 10.00 last 190.44) - /unlink 3947 (time 1786569126.15 total 20.00 last 204.18) total: 4500 /unlink in 22.56 seconds: 199.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1059 (time 1786569141.19 total 10.01 last 105.81) - open/close 2193 (time 1786569151.20 total 20.02 last 113.28) - open/close 3343 (time 1786569161.20 total 30.02 last 114.99) - open/close 4351 (time 1786569171.21 total 40.03 last 100.72) total: 4500 open/close in 41.64 seconds: 108.08 ops/second - /unlink 1937 (time 1786569186.28 total 10.00 last 193.67) - /unlink 3825 (time 1786569196.29 total 20.01 last 188.69) total: 4500 /unlink in 23.38 seconds: 192.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 975 (time 1786569213.14 total 10.01 last 97.42) - open/close 2074 (time 1786569223.15 total 20.02 last 109.80) - open/close 3166 (time 1786569233.15 total 30.02 last 109.18) - open/close 4275 (time 1786569243.16 total 40.02 last 110.83) total: 4500 open/close in 42.24 seconds: 106.54 ops/second - /unlink 2142 (time 1786569258.23 total 10.00 last 214.20) - /unlink 4107 (time 1786569268.23 total 20.00 last 196.47) total: 4500 /unlink in 21.95 seconds: 204.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1081 (time 1786569282.98 total 10.01 last 108.03) - open/close 2227 (time 1786569292.98 total 20.01 last 114.55) - open/close 3251 (time 1786569302.99 total 30.02 last 102.35) - open/close 4181 (time 1786569313.00 total 40.02 last 92.93) total: 4500 open/close in 43.12 seconds: 104.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2046 (time 1786569329.01 total 10.00 last 204.56) - /unlink 3996 (time 1786569339.01 total 20.01 last 194.93) total: 4500 /unlink in 22.41 seconds: 200.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1081 (time 1786569354.52 total 10.01 last 107.98) - open/close 2100 (time 1786569364.53 total 20.02 last 101.85) - open/close 3093 (time 1786569374.53 total 30.02 last 99.28) - open/close 4225 (time 1786569384.54 total 40.03 last 113.06) total: 4500 open/close in 42.39 seconds: 106.15 ops/second - /unlink 2056 (time 1786569400.05 total 10.00 last 205.53) - /unlink 3936 (time 1786569410.06 total 20.01 last 187.95) total: 4500 /unlink in 23.20 seconds: 193.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 903 (time 1786569426.41 total 10.00 last 90.29) - open/close 2033 (time 1786569436.42 total 20.00 last 112.96) - open/close 2958 (time 1786569446.42 total 30.01 last 92.49) - open/close 3744 (time 1786569456.42 total 40.01 last 78.60) total: 4500 open/close in 48.50 seconds: 92.79 ops/second - /unlink 2005 (time 1786569478.22 total 10.00 last 200.42) - /unlink 4226 (time 1786569488.23 total 20.01 last 222.06) total: 4500 /unlink in 21.42 seconds: 210.08 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95206 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: modules unloaded.