== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:00:12 (1788156012) Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg250-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg250-server oleg250-server: oleg250-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: MODOPTS_LIBCFS= oleg250-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg250-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg250-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg250-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 638 (time 1788156216.85 total 10.01 last 63.71) - open/close 1209 (time 1788156226.87 total 20.04 last 56.96) - open/close 1943 (time 1788156236.87 total 30.04 last 73.36) - open/close 2748 (time 1788156246.88 total 40.05 last 80.47) - open/close 3675 (time 1788156256.88 total 50.05 last 92.66) total: 4500 open/close in 58.68 seconds: 76.69 ops/second - /unlink 1368 (time 1788156279.79 total 10.00 last 136.79) - /unlink 2915 (time 1788156289.79 total 20.00 last 154.67) total: 4500 /unlink in 29.79 seconds: 151.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 646 (time 1788156314.54 total 10.00 last 64.58) - open/close 1558 (time 1788156324.55 total 20.01 last 91.10) - open/close 2632 (time 1788156334.56 total 30.02 last 107.32) - open/close 3501 (time 1788156344.56 total 40.03 last 86.85) - open/close 4396 (time 1788156354.60 total 50.07 last 89.14) total: 4500 open/close in 51.76 seconds: 86.95 ops/second - /unlink 1695 (time 1788156370.68 total 10.00 last 169.42) - /unlink 3384 (time 1788156380.68 total 20.01 last 168.86) total: 4500 /unlink in 26.90 seconds: 167.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 692 (time 1788156401.66 total 10.00 last 69.19) - open/close 1487 (time 1788156411.67 total 20.00 last 79.48) - open/close 2408 (time 1788156421.67 total 30.01 last 92.09) - open/close 3374 (time 1788156431.68 total 40.02 last 96.49) - open/close 4167 (time 1788156441.69 total 50.03 last 79.23) total: 4500 open/close in 53.82 seconds: 83.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1296 (time 1788156460.22 total 10.00 last 129.57) - /unlink 2561 (time 1788156470.22 total 20.01 last 126.46) - /unlink 4191 (time 1788156480.22 total 30.01 last 163.00) total: 4500 /unlink in 31.91 seconds: 141.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 582 (time 1788156498.45 total 10.01 last 58.14) - open/close 1361 (time 1788156508.46 total 20.01 last 77.87) - open/close 2208 (time 1788156518.46 total 30.02 last 84.68) - open/close 3042 (time 1788156528.46 total 40.02 last 83.38) - open/close 3883 (time 1788156538.47 total 50.03 last 84.00) total: 4500 open/close in 57.63 seconds: 78.09 ops/second - /unlink 1390 (time 1788156562.01 total 10.00 last 138.96) - /unlink 2895 (time 1788156572.02 total 20.01 last 150.44) total: 4500 /unlink in 28.99 seconds: 155.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 874 (time 1788156596.47 total 10.00 last 87.40) - open/close 1811 (time 1788156606.47 total 20.00 last 93.66) - open/close 2852 (time 1788156616.48 total 30.01 last 104.08) - open/close 3662 (time 1788156626.48 total 40.01 last 80.96) - open/close 4385 (time 1788156636.48 total 50.01 last 72.29) total: 4500 open/close in 51.37 seconds: 87.60 ops/second - /unlink 1796 (time 1788156652.32 total 10.00 last 179.57) - /unlink 3452 (time 1788156662.32 total 20.00 last 165.55) total: 4500 /unlink in 26.29 seconds: 171.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 687 (time 1788156684.32 total 10.00 last 68.69) - open/close 1528 (time 1788156694.33 total 20.01 last 84.03) - open/close 2172 (time 1788156704.34 total 30.02 last 64.34) - open/close 2792 (time 1788156714.34 total 40.02 last 61.99) - open/close 3353 (time 1788156724.36 total 50.04 last 55.98) - open/close 3960 (time 1788156734.37 total 60.05 last 60.66) total: 4500 open/close in 68.15 seconds: 66.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1438 (time 1788156757.91 total 10.00 last 143.76) - /unlink 2725 (time 1788156767.91 total 20.00 last 128.69) - /unlink 4348 (time 1788156777.91 total 30.01 last 162.22) total: 4500 /unlink in 31.11 seconds: 144.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 938 (time 1788156793.73 total 10.00 last 93.79) - open/close 1789 (time 1788156803.74 total 20.01 last 85.06) - open/close 2688 (time 1788156813.74 total 30.01 last 89.88) - open/close 3645 (time 1788156823.74 total 40.01 last 95.66) - open/close 4384 (time 1788156833.74 total 50.01 last 73.89) total: 4500 open/close in 51.73 seconds: 87.00 ops/second - /unlink 1603 (time 1788156849.20 total 10.00 last 160.25) - /unlink 3090 (time 1788156859.21 total 20.01 last 148.56) total: 4500 /unlink in 27.69 seconds: 162.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 955 (time 1788156881.34 total 10.00 last 95.45) - open/close 1780 (time 1788156891.35 total 20.02 last 82.39) - open/close 2723 (time 1788156901.36 total 30.02 last 94.25) - open/close 3654 (time 1788156911.37 total 40.03 last 93.04) - open/close 4460 (time 1788156921.37 total 50.03 last 80.56) total: 4500 open/close in 50.43 seconds: 89.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1206 (time 1788156936.71 total 10.01 last 120.53) - /unlink 2255 (time 1788156946.71 total 20.01 last 104.87) - /unlink 3820 (time 1788156956.72 total 30.01 last 156.46) total: 4500 /unlink in 35.38 seconds: 127.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 568 (time 1788156977.38 total 10.01 last 56.76) - open/close 1109 (time 1788156987.39 total 20.02 last 54.05) - open/close 1684 (time 1788156997.39 total 30.02 last 57.47) - open/close 2182 (time 1788157007.40 total 40.03 last 49.78) - open/close 3058 (time 1788157017.40 total 50.03 last 87.56) - open/close 3886 (time 1788157027.41 total 60.04 last 82.76) total: 4500 open/close in 67.38 seconds: 66.79 ops/second - /unlink 1696 (time 1788157049.64 total 10.00 last 169.59) - /unlink 3238 (time 1788157059.64 total 20.00 last 154.18) total: 4500 /unlink in 28.03 seconds: 160.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 947 (time 1788157082.83 total 10.00 last 94.70) - open/close 1807 (time 1788157092.84 total 20.01 last 85.95) - open/close 2586 (time 1788157102.84 total 30.01 last 77.88) - open/close 3469 (time 1788157112.85 total 40.02 last 88.24) - open/close 4098 (time 1788157122.87 total 50.03 last 62.79) total: 4500 open/close in 54.76 seconds: 82.18 ops/second - /unlink 1562 (time 1788157142.11 total 10.02 last 155.93) - /unlink 2769 (time 1788157152.12 total 20.03 last 120.58) - /unlink 4032 (time 1788157162.13 total 30.03 last 126.20) total: 4500 /unlink in 34.65 seconds: 129.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 725 (time 1788157183.82 total 10.00 last 72.48) - open/close 1470 (time 1788157193.82 total 20.00 last 74.49) - open/close 2289 (time 1788157203.83 total 30.01 last 81.85) - open/close 3138 (time 1788157213.83 total 40.01 last 84.90) - open/close 3921 (time 1788157223.83 total 50.01 last 78.27) total: 4500 open/close in 56.63 seconds: 79.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1385 (time 1788157245.69 total 10.00 last 138.46) - /unlink 2728 (time 1788157255.69 total 20.01 last 134.23) - /unlink 3923 (time 1788157265.69 total 30.01 last 119.48) total: 4500 /unlink in 34.52 seconds: 130.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 702 (time 1788157285.61 total 10.01 last 70.13) - open/close 1562 (time 1788157295.63 total 20.03 last 85.85) - open/close 2425 (time 1788157305.63 total 30.03 last 86.29) - open/close 3290 (time 1788157315.63 total 40.03 last 86.47) - open/close 4103 (time 1788157325.64 total 50.04 last 81.22) total: 4500 open/close in 54.54 seconds: 82.51 ops/second - /unlink 1571 (time 1788157344.62 total 10.00 last 157.04) - /unlink 3135 (time 1788157354.63 total 20.01 last 156.32) - /unlink 4483 (time 1788157364.63 total 30.01 last 134.76) total: 4500 /unlink in 30.20 seconds: 149.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 914 (time 1788157379.67 total 10.00 last 91.37) - open/close 1841 (time 1788157389.67 total 20.01 last 92.63) - open/close 2767 (time 1788157399.68 total 30.02 last 92.53) - open/close 3621 (time 1788157409.68 total 40.02 last 85.40) - open/close 4465 (time 1788157419.69 total 50.02 last 84.36) total: 4500 open/close in 50.48 seconds: 89.14 ops/second - /unlink 1732 (time 1788157434.13 total 10.00 last 173.15) - /unlink 3340 (time 1788157444.13 total 20.00 last 160.78) total: 4500 /unlink in 26.39 seconds: 170.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 526 (time 1788157466.27 total 10.01 last 52.57) - open/close 991 (time 1788157476.29 total 20.03 last 46.39) - open/close 1484 (time 1788157486.31 total 30.04 last 49.23) - open/close 2093 (time 1788157496.31 total 40.04 last 60.89) - open/close 2750 (time 1788157506.32 total 50.06 last 65.60) - open/close 3535 (time 1788157516.36 total 60.10 last 78.17) - open/close 4159 (time 1788157526.37 total 70.11 last 62.35) total: 4500 open/close in 76.76 seconds: 58.62 ops/second - /unlink 1564 (time 1788157547.63 total 10.00 last 156.38) - /unlink 3041 (time 1788157557.63 total 20.01 last 147.64) - /unlink 4497 (time 1788157567.63 total 30.01 last 145.57) total: 4500 /unlink in 30.02 seconds: 149.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 659 (time 1788157582.29 total 10.01 last 65.84) - open/close 1413 (time 1788157592.30 total 20.01 last 75.37) - open/close 2304 (time 1788157602.30 total 30.02 last 89.04) - open/close 3004 (time 1788157612.30 total 40.02 last 69.99) - open/close 3775 (time 1788157622.32 total 50.03 last 77.00) - open/close 4495 (time 1788157632.32 total 60.04 last 71.98) total: 4500 open/close in 60.13 seconds: 74.84 ops/second - /unlink 1576 (time 1788157647.24 total 10.00 last 157.56) - /unlink 3377 (time 1788157657.24 total 20.00 last 180.09) total: 4500 /unlink in 26.11 seconds: 172.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 909 (time 1788157677.83 total 10.00 last 90.90) - open/close 1657 (time 1788157687.84 total 20.00 last 74.78) - open/close 2428 (time 1788157697.84 total 30.01 last 77.08) - open/close 3089 (time 1788157707.85 total 40.02 last 66.02) - open/close 3982 (time 1788157717.87 total 50.04 last 89.12) total: 4500 open/close in 57.55 seconds: 78.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1623 (time 1788157740.70 total 10.00 last 162.27) - /unlink 3110 (time 1788157750.70 total 20.01 last 148.62) total: 4500 /unlink in 28.30 seconds: 159.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 914 (time 1788157773.74 total 10.01 last 91.31) - open/close 1771 (time 1788157783.75 total 20.02 last 85.59) - open/close 2674 (time 1788157793.75 total 30.02 last 90.29) - open/close 3471 (time 1788157803.76 total 40.03 last 79.61) - open/close 4332 (time 1788157813.77 total 50.04 last 86.02) total: 4500 open/close in 52.31 seconds: 86.03 ops/second - /unlink 1617 (time 1788157830.96 total 10.00 last 161.67) - /unlink 2985 (time 1788157840.96 total 20.01 last 136.68) - /unlink 4430 (time 1788157850.97 total 30.02 last 144.43) total: 4500 /unlink in 30.71 seconds: 146.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 775 (time 1788157866.37 total 10.01 last 77.44) - open/close 1596 (time 1788157876.38 total 20.02 last 82.01) - open/close 2315 (time 1788157886.38 total 30.02 last 71.87) - open/close 3126 (time 1788157896.39 total 40.04 last 81.00) - open/close 3861 (time 1788157906.39 total 50.04 last 73.50) total: 4500 open/close in 58.28 seconds: 77.21 ops/second - /unlink 1449 (time 1788157929.62 total 10.00 last 144.88) - /unlink 3143 (time 1788157939.63 total 20.00 last 169.36) total: 4500 /unlink in 28.54 seconds: 157.69 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95271 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg250-server unloading modules via unload_modules_local on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local oleg250-server: modules unloaded.