== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:02:42 (1787835762) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg344-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg344-server oleg344-server: oleg344-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg344-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg344-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg344-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 713 (time 1787835964.93 total 10.01 last 71.25) - open/close 1380 (time 1787835974.94 total 20.02 last 66.64) - open/close 2041 (time 1787835984.95 total 30.03 last 65.99) - open/close 2815 (time 1787835994.97 total 40.05 last 77.28) - open/close 3471 (time 1787836004.99 total 50.07 last 65.47) - open/close 4121 (time 1787836015.00 total 60.07 last 64.96) total: 4500 open/close in 65.83 seconds: 68.36 ops/second - /unlink 1346 (time 1787836035.80 total 10.00 last 134.56) - /unlink 2585 (time 1787836045.81 total 20.01 last 123.83) - /unlink 3927 (time 1787836055.83 total 30.03 last 133.97) total: 4500 /unlink in 35.16 seconds: 127.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 674 (time 1787836075.93 total 10.00 last 67.37) - open/close 1229 (time 1787836085.95 total 20.02 last 55.39) - open/close 1923 (time 1787836095.95 total 30.03 last 69.39) - open/close 2523 (time 1787836105.97 total 40.04 last 59.91) - open/close 3183 (time 1787836115.98 total 50.05 last 65.92) - open/close 3827 (time 1787836125.98 total 60.06 last 64.38) total: 4500 open/close in 69.95 seconds: 64.34 ops/second - /unlink 1154 (time 1787836150.70 total 10.00 last 115.35) - /unlink 2092 (time 1787836160.71 total 20.00 last 93.80) - /unlink 3193 (time 1787836170.71 total 30.01 last 110.09) - /unlink 4353 (time 1787836180.73 total 40.03 last 115.77) total: 4500 /unlink in 41.84 seconds: 107.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 590 (time 1787836198.59 total 10.02 last 58.86) - open/close 1143 (time 1787836208.60 total 20.03 last 55.25) - open/close 1661 (time 1787836218.62 total 30.05 last 51.69) - open/close 2219 (time 1787836228.64 total 40.07 last 55.70) - open/close 2839 (time 1787836238.66 total 50.09 last 61.89) - open/close 3520 (time 1787836248.66 total 60.10 last 68.07) - open/close 4200 (time 1787836258.67 total 70.10 last 67.99) total: 4500 open/close in 74.33 seconds: 60.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1239 (time 1787836277.26 total 10.00 last 123.89) - /unlink 2290 (time 1787836287.26 total 20.01 last 105.02) - /unlink 3474 (time 1787836297.28 total 30.03 last 118.17) total: 4500 /unlink in 38.04 seconds: 118.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 707 (time 1787836320.15 total 10.00 last 70.69) - open/close 1314 (time 1787836330.15 total 20.01 last 60.65) - open/close 2014 (time 1787836340.17 total 30.02 last 69.90) - open/close 2656 (time 1787836350.17 total 40.03 last 64.19) - open/close 3365 (time 1787836360.17 total 50.03 last 70.88) - open/close 4028 (time 1787836370.18 total 60.03 last 66.28) total: 4500 open/close in 67.02 seconds: 67.14 ops/second - /unlink 1242 (time 1787836392.22 total 10.01 last 124.11) - /unlink 2309 (time 1787836402.22 total 20.01 last 106.67) - /unlink 3440 (time 1787836412.23 total 30.01 last 113.05) - /unlink 4490 (time 1787836422.23 total 40.02 last 104.96) total: 4500 /unlink in 40.13 seconds: 112.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 537 (time 1787836438.78 total 10.00 last 53.70) - open/close 1063 (time 1787836448.80 total 20.01 last 52.54) - open/close 1631 (time 1787836458.81 total 30.03 last 56.71) - open/close 2166 (time 1787836468.82 total 40.03 last 53.46) - open/close 2828 (time 1787836478.82 total 50.04 last 66.16) - open/close 3457 (time 1787836488.85 total 60.07 last 62.75) - open/close 4129 (time 1787836498.85 total 70.07 last 67.18) total: 4500 open/close in 76.30 seconds: 58.98 ops/second - /unlink 1191 (time 1787836519.85 total 10.01 last 118.93) - /unlink 2265 (time 1787836529.86 total 20.02 last 107.30) - /unlink 3375 (time 1787836539.86 total 30.03 last 110.97) - /unlink 4404 (time 1787836549.89 total 40.05 last 102.64) total: 4500 /unlink in 41.44 seconds: 108.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 555 (time 1787836566.90 total 10.01 last 55.46) - open/close 1108 (time 1787836576.92 total 20.02 last 55.24) - open/close 1703 (time 1787836586.93 total 30.03 last 59.43) - open/close 2361 (time 1787836596.94 total 40.04 last 65.74) - open/close 3028 (time 1787836606.96 total 50.06 last 66.56) - open/close 3655 (time 1787836616.96 total 60.06 last 62.68) - open/close 4322 (time 1787836626.97 total 70.07 last 66.62) total: 4500 open/close in 73.07 seconds: 61.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1159 (time 1787836644.99 total 10.00 last 115.87) - /unlink 2205 (time 1787836655.00 total 20.01 last 104.52) - /unlink 3377 (time 1787836665.01 total 30.01 last 117.16) total: 4500 /unlink in 39.67 seconds: 113.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 691 (time 1787836690.37 total 10.04 last 68.85) - open/close 1289 (time 1787836700.37 total 20.04 last 59.79) - open/close 1909 (time 1787836710.37 total 30.04 last 62.00) - open/close 2559 (time 1787836720.39 total 40.05 last 64.91) - open/close 3250 (time 1787836730.39 total 50.05 last 69.08) - open/close 3897 (time 1787836740.41 total 60.07 last 64.60) total: 4500 open/close in 68.40 seconds: 65.79 ops/second - /unlink 1231 (time 1787836764.26 total 10.00 last 123.08) - /unlink 2558 (time 1787836774.27 total 20.02 last 132.50) - /unlink 3900 (time 1787836784.28 total 30.02 last 134.13) total: 4500 /unlink in 34.66 seconds: 129.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 717 (time 1787836803.07 total 10.01 last 71.62) - open/close 1384 (time 1787836813.07 total 20.02 last 66.67) - open/close 2047 (time 1787836823.07 total 30.02 last 66.30) - open/close 2675 (time 1787836833.08 total 40.03 last 62.74) - open/close 3372 (time 1787836843.09 total 50.03 last 69.66) - open/close 4048 (time 1787836853.09 total 60.04 last 67.57) total: 4500 open/close in 66.00 seconds: 68.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1223 (time 1787836873.67 total 10.00 last 122.27) - /unlink 2455 (time 1787836883.69 total 20.02 last 122.97) - /unlink 3714 (time 1787836893.69 total 30.02 last 125.86) total: 4500 /unlink in 36.10 seconds: 124.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 712 (time 1787836914.68 total 10.01 last 71.16) - open/close 1500 (time 1787836924.68 total 20.01 last 78.78) - open/close 2190 (time 1787836934.69 total 30.01 last 68.96) - open/close 2878 (time 1787836944.71 total 40.03 last 68.65) - open/close 3534 (time 1787836954.71 total 50.04 last 65.58) - open/close 4227 (time 1787836964.72 total 60.04 last 69.25) total: 4500 open/close in 63.93 seconds: 70.39 ops/second - /unlink 1257 (time 1787836983.14 total 10.00 last 125.64) - /unlink 2474 (time 1787836993.15 total 20.02 last 121.52) - /unlink 3780 (time 1787837003.15 total 30.02 last 130.56) total: 4500 /unlink in 36.00 seconds: 125.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 702 (time 1787837024.37 total 10.02 last 70.08) - open/close 1378 (time 1787837034.38 total 20.03 last 67.54) - open/close 2034 (time 1787837044.38 total 30.03 last 65.59) - open/close 2685 (time 1787837054.38 total 40.03 last 65.09) - open/close 3333 (time 1787837064.39 total 50.04 last 64.76) - open/close 4007 (time 1787837074.40 total 60.05 last 67.33) total: 4500 open/close in 68.04 seconds: 66.14 ops/second - /unlink 1273 (time 1787837097.17 total 10.01 last 127.21) - /unlink 2513 (time 1787837107.17 total 20.01 last 123.96) - /unlink 3823 (time 1787837117.17 total 30.01 last 130.97) total: 4500 /unlink in 35.55 seconds: 126.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 565 (time 1787837137.01 total 10.00 last 56.48) - open/close 1131 (time 1787837147.02 total 20.01 last 56.54) - open/close 1616 (time 1787837157.03 total 30.03 last 48.41) - open/close 2170 (time 1787837167.04 total 40.04 last 55.37) - open/close 2808 (time 1787837177.06 total 50.06 last 63.68) - open/close 3374 (time 1787837187.07 total 60.07 last 56.55) - open/close 4016 (time 1787837197.07 total 70.07 last 64.16) total: 4500 open/close in 76.85 seconds: 58.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1209 (time 1787837218.60 total 10.00 last 120.85) - /unlink 2375 (time 1787837228.62 total 20.02 last 116.38) - /unlink 3615 (time 1787837238.62 total 30.02 last 123.98) total: 4500 /unlink in 37.46 seconds: 120.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 661 (time 1787837261.86 total 10.01 last 66.05) - open/close 1333 (time 1787837271.86 total 20.01 last 67.17) - open/close 2020 (time 1787837281.86 total 30.01 last 68.68) - open/close 2677 (time 1787837291.89 total 40.04 last 65.55) - open/close 3283 (time 1787837301.90 total 50.05 last 60.51) - open/close 3900 (time 1787837311.91 total 60.06 last 61.66) total: 4500 open/close in 68.06 seconds: 66.12 ops/second - /unlink 1374 (time 1787837334.19 total 10.01 last 137.31) - /unlink 2558 (time 1787837344.19 total 20.01 last 118.39) - /unlink 3799 (time 1787837354.19 total 30.01 last 124.09) total: 4500 /unlink in 36.74 seconds: 122.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 563 (time 1787837378.72 total 10.01 last 56.25) - open/close 1197 (time 1787837388.73 total 20.02 last 63.36) - open/close 1842 (time 1787837398.73 total 30.02 last 64.49) - open/close 2543 (time 1787837408.73 total 40.02 last 70.06) - open/close 3266 (time 1787837418.74 total 50.03 last 72.29) - open/close 4028 (time 1787837428.74 total 60.03 last 76.13) total: 4500 open/close in 67.16 seconds: 67.01 ops/second - /unlink 1223 (time 1787837450.75 total 10.00 last 122.25) - /unlink 2474 (time 1787837460.75 total 20.01 last 125.04) - /unlink 3781 (time 1787837470.76 total 30.02 last 130.60) total: 4500 /unlink in 35.50 seconds: 126.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 739 (time 1787837490.46 total 10.00 last 73.89) - open/close 1421 (time 1787837500.47 total 20.01 last 68.13) - open/close 2072 (time 1787837510.48 total 30.02 last 65.03) - open/close 2629 (time 1787837520.49 total 40.03 last 55.65) - open/close 3306 (time 1787837530.50 total 50.03 last 67.68) - open/close 3958 (time 1787837540.50 total 60.03 last 65.18) total: 4500 open/close in 68.68 seconds: 65.52 ops/second - /unlink 1385 (time 1787837563.98 total 10.00 last 138.47) - /unlink 2692 (time 1787837573.99 total 20.01 last 130.60) - /unlink 3964 (time 1787837583.99 total 30.01 last 127.15) total: 4500 /unlink in 33.86 seconds: 132.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 750 (time 1787837603.19 total 10.00 last 75.00) - open/close 1425 (time 1787837613.20 total 20.01 last 67.46) - open/close 2069 (time 1787837623.21 total 30.02 last 64.33) - open/close 2747 (time 1787837633.24 total 40.05 last 67.56) - open/close 3332 (time 1787837643.25 total 50.05 last 58.48) - open/close 3976 (time 1787837653.25 total 60.06 last 64.37) total: 4500 open/close in 67.61 seconds: 66.56 ops/second - /unlink 1301 (time 1787837675.50 total 10.00 last 130.06) - /unlink 2524 (time 1787837685.51 total 20.01 last 122.25) - /unlink 3864 (time 1787837695.51 total 30.01 last 133.99) total: 4500 /unlink in 34.68 seconds: 129.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 664 (time 1787837714.59 total 10.00 last 66.39) - open/close 1447 (time 1787837724.60 total 20.01 last 78.24) - open/close 2154 (time 1787837734.60 total 30.01 last 70.66) - open/close 2821 (time 1787837744.61 total 40.02 last 66.65) - open/close 3562 (time 1787837754.63 total 50.05 last 73.92) - open/close 4289 (time 1787837764.64 total 60.05 last 72.65) total: 4500 open/close in 63.00 seconds: 71.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1359 (time 1787837782.23 total 10.00 last 135.88) - /unlink 2659 (time 1787837792.23 total 20.00 last 129.98) - /unlink 4036 (time 1787837802.23 total 30.01 last 137.66) total: 4500 /unlink in 33.39 seconds: 134.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 734 (time 1787837820.51 total 10.01 last 73.29) - open/close 1474 (time 1787837830.52 total 20.02 last 73.94) - open/close 2197 (time 1787837840.54 total 30.04 last 72.16) - open/close 2924 (time 1787837850.54 total 40.05 last 72.64) - open/close 3569 (time 1787837860.55 total 50.06 last 64.45) - open/close 4299 (time 1787837870.57 total 60.08 last 72.85) total: 4500 open/close in 62.83 seconds: 71.63 ops/second - /unlink 1298 (time 1787837888.15 total 10.01 last 129.63) - /unlink 2577 (time 1787837898.15 total 20.02 last 127.88) - /unlink 3832 (time 1787837908.17 total 30.03 last 125.36) total: 4500 /unlink in 35.34 seconds: 127.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 638 (time 1787837928.95 total 10.01 last 63.76) - open/close 1350 (time 1787837938.96 total 20.02 last 71.13) - open/close 2050 (time 1787837948.97 total 30.02 last 69.96) - open/close 2698 (time 1787837958.98 total 40.03 last 64.72) - open/close 3375 (time 1787837968.99 total 50.04 last 67.64) - open/close 4015 (time 1787837979.00 total 60.05 last 63.96) total: 4500 open/close in 66.93 seconds: 67.24 ops/second - /unlink 1321 (time 1787838000.48 total 10.01 last 132.03) - /unlink 2626 (time 1787838010.48 total 20.01 last 130.46) - /unlink 3941 (time 1787838020.49 total 30.01 last 131.44) total: 4500 /unlink in 34.63 seconds: 129.93 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95267 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server unloading modules via unload_modules_local on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local oleg344-server: modules unloaded.