== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:50:40 (1785081040) Stopping clients: oleg405-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg405-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg405-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg405-server oleg405-server: oleg405-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg405-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm Registered 1 changelog users: 'cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 760 (time 1785081191.51 total 10.02 last 75.85) - open/close 1539 (time 1785081201.52 total 20.03 last 77.83) - open/close 2310 (time 1785081211.52 total 30.03 last 77.07) - open/close 3057 (time 1785081221.53 total 40.04 last 74.62) - open/close 3819 (time 1785081231.54 total 50.05 last 76.14) total: 4500 open/close in 59.34 seconds: 75.83 ops/second - /unlink 1413 (time 1785081255.40 total 10.00 last 141.26) - /unlink 2659 (time 1785081265.41 total 20.01 last 124.50) - /unlink 4016 (time 1785081275.42 total 30.02 last 135.61) total: 4500 /unlink in 33.58 seconds: 134.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 701 (time 1785081294.38 total 10.02 last 69.97) - open/close 1410 (time 1785081304.38 total 20.02 last 70.88) - open/close 2063 (time 1785081314.38 total 30.02 last 65.28) - open/close 2677 (time 1785081324.40 total 40.04 last 61.30) - open/close 3411 (time 1785081334.40 total 50.05 last 73.36) - open/close 4045 (time 1785081344.42 total 60.06 last 63.29) total: 4500 open/close in 67.29 seconds: 66.87 ops/second - /unlink 1343 (time 1785081367.97 total 10.00 last 134.29) - /unlink 2483 (time 1785081377.97 total 20.00 last 113.97) - /unlink 3758 (time 1785081387.97 total 30.01 last 127.45) total: 4500 /unlink in 35.87 seconds: 125.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 632 (time 1785081409.02 total 10.01 last 63.15) - open/close 1320 (time 1785081419.02 total 20.01 last 68.77) - open/close 1910 (time 1785081429.04 total 30.03 last 58.89) - open/close 2543 (time 1785081439.04 total 40.04 last 63.28) - open/close 3188 (time 1785081449.05 total 50.04 last 64.46) - open/close 3870 (time 1785081459.05 total 60.04 last 68.19) total: 4500 open/close in 69.02 seconds: 65.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1263 (time 1785081482.91 total 10.00 last 126.25) - /unlink 2436 (time 1785081492.91 total 20.01 last 117.24) - /unlink 3632 (time 1785081502.92 total 30.02 last 119.48) total: 4500 /unlink in 36.67 seconds: 122.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 670 (time 1785081524.30 total 10.00 last 66.97) - open/close 1397 (time 1785081534.31 total 20.02 last 72.61) - open/close 2149 (time 1785081544.31 total 30.02 last 75.19) - open/close 2931 (time 1785081554.32 total 40.02 last 78.16) - open/close 3672 (time 1785081564.32 total 50.03 last 74.07) - open/close 4322 (time 1785081574.34 total 60.05 last 64.88) total: 4500 open/close in 62.44 seconds: 72.07 ops/second - /unlink 1272 (time 1785081591.10 total 10.01 last 127.06) - /unlink 2596 (time 1785081601.10 total 20.01 last 132.37) - /unlink 3873 (time 1785081611.11 total 30.02 last 127.66) total: 4500 /unlink in 34.96 seconds: 128.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 705 (time 1785081631.09 total 10.00 last 70.50) - open/close 1384 (time 1785081641.09 total 20.00 last 67.89) - open/close 2017 (time 1785081651.10 total 30.00 last 63.29) - open/close 2758 (time 1785081661.10 total 40.00 last 74.09) - open/close 3462 (time 1785081671.10 total 50.01 last 70.39) - open/close 4129 (time 1785081681.11 total 60.02 last 66.61) total: 4500 open/close in 65.46 seconds: 68.74 ops/second - /unlink 1290 (time 1785081701.96 total 10.00 last 128.98) - /unlink 2634 (time 1785081711.96 total 20.00 last 134.38) - /unlink 3980 (time 1785081721.96 total 30.01 last 134.52) total: 4500 /unlink in 33.73 seconds: 133.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 812 (time 1785081740.50 total 10.00 last 81.19) - open/close 1535 (time 1785081750.51 total 20.01 last 72.22) - open/close 2286 (time 1785081760.52 total 30.02 last 75.07) - open/close 3069 (time 1785081770.52 total 40.02 last 78.28) - open/close 3832 (time 1785081780.54 total 50.04 last 76.17) total: 4500 open/close in 58.83 seconds: 76.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1266 (time 1785081803.49 total 10.00 last 126.56) - /unlink 2560 (time 1785081813.50 total 20.01 last 129.33) - /unlink 4014 (time 1785081823.50 total 30.01 last 145.36) total: 4500 /unlink in 33.79 seconds: 133.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 751 (time 1785081841.34 total 10.01 last 75.00) - open/close 1433 (time 1785081851.34 total 20.02 last 68.17) - open/close 2208 (time 1785081861.35 total 30.03 last 77.44) - open/close 2870 (time 1785081871.36 total 40.03 last 66.14) - open/close 3651 (time 1785081881.36 total 50.04 last 78.05) - open/close 4444 (time 1785081891.37 total 60.05 last 79.22) total: 4500 open/close in 60.74 seconds: 74.08 ops/second - /unlink 1376 (time 1785081906.01 total 10.01 last 137.52) - /unlink 2522 (time 1785081916.02 total 20.01 last 114.56) - /unlink 3740 (time 1785081926.03 total 30.02 last 121.63) total: 4500 /unlink in 36.07 seconds: 124.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 718 (time 1785081946.25 total 10.01 last 71.72) - open/close 1465 (time 1785081956.26 total 20.02 last 74.67) - open/close 2303 (time 1785081966.26 total 30.02 last 83.79) - open/close 3058 (time 1785081976.27 total 40.02 last 75.45) - open/close 3777 (time 1785081986.27 total 50.03 last 71.88) total: 4500 open/close in 59.34 seconds: 75.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1247 (time 1785082010.21 total 10.00 last 124.70) - /unlink 2575 (time 1785082020.22 total 20.01 last 132.72) - /unlink 3869 (time 1785082030.22 total 30.01 last 129.37) total: 4500 /unlink in 34.57 seconds: 130.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 761 (time 1785082049.49 total 10.01 last 76.05) - open/close 1498 (time 1785082059.50 total 20.02 last 73.63) - open/close 2261 (time 1785082069.51 total 30.03 last 76.22) - open/close 3047 (time 1785082079.52 total 40.03 last 78.55) - open/close 3769 (time 1785082089.54 total 50.06 last 72.04) - open/close 4481 (time 1785082099.54 total 60.06 last 71.20) total: 4500 open/close in 60.29 seconds: 74.63 ops/second - /unlink 1369 (time 1785082113.69 total 10.00 last 136.88) - /unlink 2661 (time 1785082123.69 total 20.00 last 129.15) - /unlink 4068 (time 1785082133.69 total 30.01 last 140.68) total: 4500 /unlink in 33.40 seconds: 134.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 716 (time 1785082151.55 total 10.01 last 71.53) - open/close 1485 (time 1785082161.55 total 20.01 last 76.89) - open/close 2259 (time 1785082171.56 total 30.02 last 77.30) - open/close 2992 (time 1785082181.58 total 40.05 last 73.13) - open/close 3682 (time 1785082191.60 total 50.07 last 68.87) - open/close 4301 (time 1785082201.61 total 60.07 last 61.89) total: 4500 open/close in 62.67 seconds: 71.81 ops/second - /unlink 1169 (time 1785082218.94 total 10.00 last 116.85) - /unlink 2461 (time 1785082228.94 total 20.01 last 129.17) - /unlink 3844 (time 1785082238.94 total 30.01 last 138.28) total: 4500 /unlink in 35.23 seconds: 127.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 688 (time 1785082258.46 total 10.01 last 68.73) - open/close 1351 (time 1785082268.47 total 20.02 last 66.25) - open/close 2071 (time 1785082278.47 total 30.02 last 71.98) - open/close 2671 (time 1785082288.47 total 40.03 last 59.97) - open/close 3382 (time 1785082298.48 total 50.03 last 71.05) - open/close 4110 (time 1785082308.49 total 60.04 last 72.75) total: 4500 open/close in 65.39 seconds: 68.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1277 (time 1785082327.95 total 10.00 last 127.64) - /unlink 2521 (time 1785082337.95 total 20.01 last 124.39) - /unlink 3964 (time 1785082347.96 total 30.02 last 144.14) total: 4500 /unlink in 33.85 seconds: 132.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 753 (time 1785082367.15 total 10.01 last 75.22) - open/close 1521 (time 1785082377.15 total 20.01 last 76.79) - open/close 2242 (time 1785082387.17 total 30.04 last 71.92) - open/close 2986 (time 1785082397.18 total 40.04 last 74.35) - open/close 3738 (time 1785082407.19 total 50.05 last 75.15) - open/close 4469 (time 1785082417.20 total 60.06 last 73.04) total: 4500 open/close in 60.57 seconds: 74.30 ops/second - /unlink 1296 (time 1785082432.02 total 10.00 last 129.54) - /unlink 2551 (time 1785082442.02 total 20.01 last 125.49) - /unlink 3810 (time 1785082452.02 total 30.01 last 125.81) total: 4500 /unlink in 34.95 seconds: 128.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 800 (time 1785082471.73 total 10.02 last 79.86) - open/close 1615 (time 1785082481.75 total 20.04 last 81.34) - open/close 2413 (time 1785082491.76 total 30.05 last 79.68) - open/close 3182 (time 1785082501.77 total 40.05 last 76.88) - open/close 3929 (time 1785082511.77 total 50.06 last 74.65) total: 4500 open/close in 57.54 seconds: 78.21 ops/second - /unlink 1341 (time 1785082533.28 total 10.00 last 134.08) - /unlink 2681 (time 1785082543.29 total 20.00 last 133.97) - /unlink 3942 (time 1785082553.29 total 30.01 last 126.02) total: 4500 /unlink in 34.00 seconds: 132.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 668 (time 1785082572.19 total 10.00 last 66.78) - open/close 1255 (time 1785082582.21 total 20.02 last 58.61) - open/close 1788 (time 1785082592.24 total 30.05 last 53.16) - open/close 2466 (time 1785082602.25 total 40.06 last 67.68) - open/close 3262 (time 1785082612.27 total 50.08 last 79.48) - open/close 4003 (time 1785082622.28 total 60.09 last 74.03) total: 4500 open/close in 66.35 seconds: 67.82 ops/second - /unlink 1404 (time 1785082642.41 total 10.00 last 140.39) - /unlink 2769 (time 1785082652.41 total 20.00 last 136.49) - /unlink 3975 (time 1785082662.42 total 30.01 last 120.46) total: 4500 /unlink in 34.08 seconds: 132.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 638 (time 1785082681.72 total 10.00 last 63.78) - open/close 1252 (time 1785082691.73 total 20.01 last 61.37) - open/close 1989 (time 1785082701.73 total 30.01 last 73.68) - open/close 2794 (time 1785082711.75 total 40.04 last 80.30) - open/close 3528 (time 1785082721.77 total 50.05 last 73.28) - open/close 4289 (time 1785082731.78 total 60.06 last 76.04) total: 4500 open/close in 62.97 seconds: 71.46 ops/second - /unlink 1360 (time 1785082749.34 total 10.00 last 135.96) - /unlink 2695 (time 1785082759.35 total 20.01 last 133.45) - /unlink 4058 (time 1785082769.35 total 30.01 last 136.26) total: 4500 /unlink in 33.80 seconds: 133.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 788 (time 1785082788.38 total 10.01 last 78.68) - open/close 1470 (time 1785082798.38 total 20.02 last 68.20) - open/close 2168 (time 1785082808.40 total 30.03 last 69.68) - open/close 2962 (time 1785082818.40 total 40.03 last 79.38) - open/close 3703 (time 1785082828.41 total 50.04 last 74.07) - open/close 4257 (time 1785082838.41 total 60.04 last 55.39) total: 4500 open/close in 63.22 seconds: 71.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1394 (time 1785082855.70 total 10.01 last 139.30) - /unlink 2571 (time 1785082865.70 total 20.01 last 117.63) - /unlink 3889 (time 1785082875.71 total 30.02 last 131.77) total: 4500 /unlink in 35.36 seconds: 127.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 736 (time 1785082895.17 total 10.01 last 73.53) - open/close 1444 (time 1785082905.18 total 20.01 last 70.79) - open/close 2130 (time 1785082915.18 total 30.01 last 68.60) - open/close 2905 (time 1785082925.18 total 40.02 last 77.45) - open/close 3708 (time 1785082935.18 total 50.02 last 80.28) - open/close 4457 (time 1785082945.19 total 60.03 last 74.83) total: 4500 open/close in 60.68 seconds: 74.16 ops/second - /unlink 1493 (time 1785082959.54 total 10.01 last 149.21) - /unlink 2778 (time 1785082969.54 total 20.01 last 128.45) - /unlink 4338 (time 1785082979.55 total 30.02 last 155.81) total: 4500 /unlink in 31.38 seconds: 143.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 828 (time 1785082995.01 total 10.00 last 82.78) - open/close 1611 (time 1785083005.02 total 20.01 last 78.22) - open/close 2366 (time 1785083015.02 total 30.02 last 75.47) - open/close 3176 (time 1785083025.03 total 40.02 last 80.99) - open/close 3909 (time 1785083035.04 total 50.03 last 73.22) total: 4500 open/close in 56.93 seconds: 79.04 ops/second - /unlink 1454 (time 1785083055.90 total 10.01 last 145.32) - /unlink 2883 (time 1785083065.91 total 20.01 last 142.76) - /unlink 4245 (time 1785083075.91 total 30.02 last 136.17) total: 4500 /unlink in 31.89 seconds: 141.13 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 75523 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: modules unloaded.