== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:33:39 (1787830419) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg118-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg118-server oleg118-server: oleg118-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 677 (time 1787830658.87 total 10.00 last 67.68) - open/close 1504 (time 1787830668.88 total 20.01 last 82.65) - open/close 2173 (time 1787830678.89 total 30.02 last 66.81) - open/close 2817 (time 1787830688.91 total 40.05 last 64.26) - open/close 3536 (time 1787830698.91 total 50.05 last 71.88) - open/close 4234 (time 1787830708.92 total 60.05 last 69.77) total: 4500 open/close in 64.16 seconds: 70.14 ops/second - /unlink 1448 (time 1787830727.72 total 10.00 last 144.76) - /unlink 2715 (time 1787830737.72 total 20.00 last 126.68) - /unlink 3929 (time 1787830747.73 total 30.01 last 121.32) total: 4500 /unlink in 34.66 seconds: 129.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 659 (time 1787830768.91 total 10.01 last 65.86) - open/close 1458 (time 1787830778.91 total 20.01 last 79.90) - open/close 2170 (time 1787830788.91 total 30.01 last 71.20) - open/close 2870 (time 1787830798.91 total 40.01 last 69.98) - open/close 3557 (time 1787830808.93 total 50.02 last 68.61) - open/close 4182 (time 1787830818.94 total 60.03 last 62.44) total: 4500 open/close in 63.99 seconds: 70.33 ops/second - /unlink 1233 (time 1787830837.99 total 10.01 last 123.24) - /unlink 2294 (time 1787830847.99 total 20.01 last 106.07) - /unlink 3666 (time 1787830858.00 total 30.01 last 137.12) total: 4500 /unlink in 38.30 seconds: 117.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 671 (time 1787830882.21 total 10.00 last 67.08) - open/close 1310 (time 1787830892.21 total 20.01 last 63.87) - open/close 2048 (time 1787830902.25 total 30.05 last 73.50) - open/close 2762 (time 1787830912.26 total 40.05 last 71.36) - open/close 3486 (time 1787830922.26 total 50.06 last 72.37) - open/close 4198 (time 1787830932.28 total 60.07 last 71.09) total: 4500 open/close in 63.89 seconds: 70.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1339 (time 1787830951.29 total 10.00 last 133.84) - /unlink 2586 (time 1787830961.29 total 20.01 last 124.66) - /unlink 3707 (time 1787830971.29 total 30.01 last 112.06) total: 4500 /unlink in 36.44 seconds: 123.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 671 (time 1787830993.14 total 10.01 last 67.06) - open/close 1367 (time 1787831003.15 total 20.01 last 69.54) - open/close 2128 (time 1787831013.16 total 30.02 last 76.06) - open/close 2800 (time 1787831023.16 total 40.03 last 67.16) - open/close 3476 (time 1787831033.19 total 50.05 last 67.43) - open/close 4100 (time 1787831043.21 total 60.07 last 62.29) total: 4500 open/close in 65.51 seconds: 68.69 ops/second - /unlink 1262 (time 1787831063.63 total 10.00 last 126.18) - /unlink 2452 (time 1787831073.64 total 20.01 last 118.90) - /unlink 3802 (time 1787831083.64 total 30.01 last 134.99) total: 4500 /unlink in 35.37 seconds: 127.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 668 (time 1787831104.80 total 10.00 last 66.78) - open/close 1277 (time 1787831114.82 total 20.01 last 60.83) - open/close 1947 (time 1787831124.82 total 30.02 last 66.96) - open/close 2619 (time 1787831134.82 total 40.02 last 67.19) - open/close 3278 (time 1787831144.83 total 50.03 last 65.83) - open/close 3987 (time 1787831154.84 total 60.03 last 70.88) total: 4500 open/close in 67.21 seconds: 66.96 ops/second - /unlink 1441 (time 1787831176.76 total 10.01 last 144.01) - /unlink 2894 (time 1787831186.76 total 20.01 last 145.30) - /unlink 4211 (time 1787831196.78 total 30.02 last 131.54) total: 4500 /unlink in 32.57 seconds: 138.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 614 (time 1787831215.11 total 10.01 last 61.37) - open/close 1126 (time 1787831225.12 total 20.02 last 51.13) - open/close 1729 (time 1787831235.14 total 30.04 last 60.19) - open/close 2203 (time 1787831245.14 total 40.04 last 47.39) - open/close 2828 (time 1787831255.15 total 50.05 last 62.45) - open/close 3512 (time 1787831265.15 total 60.05 last 68.39) - open/close 4155 (time 1787831275.17 total 70.06 last 64.19) total: 4500 open/close in 75.00 seconds: 60.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1235 (time 1787831295.30 total 10.02 last 123.30) - /unlink 2441 (time 1787831305.31 total 20.02 last 120.53) - /unlink 3619 (time 1787831315.31 total 30.03 last 117.76) total: 4500 /unlink in 37.36 seconds: 120.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 626 (time 1787831338.26 total 10.01 last 62.56) - open/close 1236 (time 1787831348.26 total 20.01 last 60.99) - open/close 1845 (time 1787831358.28 total 30.03 last 60.79) - open/close 2536 (time 1787831368.29 total 40.03 last 69.08) - open/close 3272 (time 1787831378.29 total 50.04 last 73.53) - open/close 3988 (time 1787831388.30 total 60.05 last 71.54) total: 4500 open/close in 69.19 seconds: 65.04 ops/second - /unlink 1429 (time 1787831412.79 total 10.00 last 142.85) - /unlink 2691 (time 1787831422.80 total 20.01 last 126.14) - /unlink 3947 (time 1787831432.80 total 30.01 last 125.57) total: 4500 /unlink in 33.55 seconds: 134.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 641 (time 1787831451.85 total 10.01 last 64.05) - open/close 1318 (time 1787831461.86 total 20.01 last 67.68) - open/close 1973 (time 1787831471.86 total 30.02 last 65.46) - open/close 2717 (time 1787831481.87 total 40.02 last 74.35) - open/close 3471 (time 1787831491.88 total 50.03 last 75.33) - open/close 4110 (time 1787831501.88 total 60.03 last 63.90) total: 4500 open/close in 66.25 seconds: 67.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1502 (time 1787831524.22 total 10.01 last 149.98) - /unlink 2872 (time 1787831534.23 total 20.02 last 136.95) - /unlink 4046 (time 1787831544.23 total 30.02 last 117.39) total: 4500 /unlink in 33.62 seconds: 133.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 703 (time 1787831562.89 total 10.02 last 70.17) - open/close 1318 (time 1787831572.91 total 20.04 last 61.38) - open/close 2058 (time 1787831582.91 total 30.04 last 73.98) - open/close 2696 (time 1787831592.92 total 40.05 last 63.75) - open/close 3435 (time 1787831602.95 total 50.08 last 73.67) - open/close 4083 (time 1787831612.95 total 60.09 last 64.76) total: 4500 open/close in 66.50 seconds: 67.67 ops/second - /unlink 1279 (time 1787831634.98 total 10.00 last 127.85) - /unlink 2507 (time 1787831644.98 total 20.00 last 122.79) - /unlink 3708 (time 1787831654.99 total 30.01 last 120.07) total: 4500 /unlink in 36.36 seconds: 123.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 611 (time 1787831676.97 total 10.01 last 61.05) - open/close 1290 (time 1787831686.98 total 20.01 last 67.89) - open/close 2022 (time 1787831696.98 total 30.02 last 73.16) - open/close 2748 (time 1787831706.99 total 40.02 last 72.56) - open/close 3442 (time 1787831716.99 total 50.02 last 69.39) - open/close 4142 (time 1787831726.99 total 60.02 last 69.99) total: 4500 open/close in 64.88 seconds: 69.36 ops/second - /unlink 1627 (time 1787831746.12 total 10.00 last 162.68) - /unlink 3009 (time 1787831756.13 total 20.01 last 138.10) - /unlink 3992 (time 1787831766.13 total 30.01 last 98.25) total: 4500 /unlink in 34.30 seconds: 131.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 697 (time 1787831785.77 total 10.00 last 69.69) - open/close 1423 (time 1787831795.78 total 20.01 last 72.50) - open/close 2333 (time 1787831805.79 total 30.02 last 90.92) - open/close 3174 (time 1787831815.80 total 40.03 last 84.04) - open/close 3854 (time 1787831825.80 total 50.03 last 67.99) total: 4500 open/close in 58.95 seconds: 76.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1532 (time 1787831849.57 total 10.00 last 153.18) - /unlink 2960 (time 1787831859.57 total 20.00 last 142.78) - /unlink 4143 (time 1787831869.58 total 30.00 last 118.28) total: 4500 /unlink in 32.41 seconds: 138.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 712 (time 1787831886.43 total 10.01 last 71.15) - open/close 1444 (time 1787831896.46 total 20.03 last 73.02) - open/close 2138 (time 1787831906.47 total 30.04 last 69.34) - open/close 2825 (time 1787831916.48 total 40.05 last 68.61) - open/close 3476 (time 1787831926.48 total 50.05 last 65.10) - open/close 4334 (time 1787831936.48 total 60.06 last 85.75) total: 4500 open/close in 62.84 seconds: 71.61 ops/second - /unlink 1490 (time 1787831953.37 total 10.00 last 149.00) - /unlink 2928 (time 1787831963.37 total 20.00 last 143.77) - /unlink 4208 (time 1787831973.37 total 30.00 last 127.98) total: 4500 /unlink in 32.37 seconds: 139.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 736 (time 1787831990.79 total 10.00 last 73.58) - open/close 1511 (time 1787832000.79 total 20.01 last 77.44) - open/close 2053 (time 1787832010.79 total 30.01 last 54.19) - open/close 2697 (time 1787832020.81 total 40.02 last 64.33) - open/close 3544 (time 1787832030.81 total 50.03 last 84.63) - open/close 4287 (time 1787832040.81 total 60.03 last 74.30) total: 4500 open/close in 62.60 seconds: 71.89 ops/second - /unlink 1456 (time 1787832058.19 total 10.02 last 145.35) - /unlink 2922 (time 1787832068.24 total 20.07 last 145.89) - /unlink 4170 (time 1787832078.24 total 30.07 last 124.75) total: 4500 /unlink in 32.05 seconds: 140.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 631 (time 1787832096.86 total 10.00 last 63.09) - open/close 1401 (time 1787832106.87 total 20.01 last 76.96) - open/close 2191 (time 1787832116.90 total 30.04 last 78.72) - open/close 2925 (time 1787832126.90 total 40.04 last 73.40) - open/close 3662 (time 1787832136.92 total 50.06 last 73.58) - open/close 4319 (time 1787832146.92 total 60.06 last 65.69) total: 4500 open/close in 63.07 seconds: 71.35 ops/second - /unlink 1344 (time 1787832164.57 total 10.00 last 134.36) - /unlink 2762 (time 1787832174.57 total 20.00 last 141.77) - /unlink 4213 (time 1787832184.57 total 30.01 last 145.06) total: 4500 /unlink in 32.52 seconds: 138.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 719 (time 1787832202.33 total 10.02 last 71.76) - open/close 1284 (time 1787832212.34 total 20.02 last 56.48) - open/close 2041 (time 1787832222.35 total 30.04 last 75.61) - open/close 2820 (time 1787832232.36 total 40.04 last 77.83) - open/close 3357 (time 1787832242.36 total 50.05 last 53.69) - open/close 3953 (time 1787832252.36 total 60.05 last 59.59) - open/close 4454 (time 1787832262.36 total 70.05 last 50.07) total: 4500 open/close in 71.18 seconds: 63.22 ops/second - /unlink 1399 (time 1787832279.54 total 10.00 last 139.83) - /unlink 2810 (time 1787832289.55 total 20.01 last 140.99) - /unlink 4305 (time 1787832299.55 total 30.02 last 149.46) total: 4500 /unlink in 31.61 seconds: 142.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 746 (time 1787832315.86 total 10.01 last 74.53) - open/close 1348 (time 1787832325.86 total 20.01 last 60.19) - open/close 1969 (time 1787832335.86 total 30.01 last 62.08) - open/close 2646 (time 1787832345.88 total 40.03 last 67.56) - open/close 3357 (time 1787832355.90 total 50.06 last 70.93) - open/close 4068 (time 1787832365.91 total 60.07 last 71.05) total: 4500 open/close in 65.57 seconds: 68.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1250 (time 1787832386.61 total 10.01 last 124.93) - /unlink 2563 (time 1787832396.61 total 20.01 last 131.25) - /unlink 3702 (time 1787832406.62 total 30.01 last 113.85) total: 4500 /unlink in 36.52 seconds: 123.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 740 (time 1787832429.27 total 10.00 last 73.99) - open/close 1472 (time 1787832439.28 total 20.01 last 73.13) - open/close 2108 (time 1787832449.29 total 30.03 last 63.49) - open/close 2891 (time 1787832459.30 total 40.03 last 78.28) - open/close 3636 (time 1787832469.30 total 50.03 last 74.50) - open/close 4332 (time 1787832479.30 total 60.03 last 69.58) total: 4500 open/close in 62.41 seconds: 72.10 ops/second - /unlink 1213 (time 1787832497.09 total 10.00 last 121.25) - /unlink 2070 (time 1787832507.11 total 20.02 last 85.58) - /unlink 3024 (time 1787832517.12 total 30.03 last 95.29) - /unlink 4256 (time 1787832527.12 total 40.03 last 123.15) total: 4500 /unlink in 42.16 seconds: 106.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 668 (time 1787832545.03 total 10.01 last 66.73) - open/close 1335 (time 1787832555.04 total 20.02 last 66.64) - open/close 1791 (time 1787832565.05 total 30.03 last 45.55) - open/close 2358 (time 1787832575.05 total 40.04 last 56.67) - open/close 3080 (time 1787832585.07 total 50.06 last 72.03) - open/close 3768 (time 1787832595.08 total 60.07 last 68.74) - open/close 4355 (time 1787832605.09 total 70.08 last 58.65) total: 4500 open/close in 72.31 seconds: 62.23 ops/second - /unlink 1320 (time 1787832622.22 total 10.01 last 131.83) - /unlink 2353 (time 1787832632.23 total 20.02 last 103.24) - /unlink 3660 (time 1787832642.23 total 30.02 last 130.65) total: 4500 /unlink in 37.48 seconds: 120.08 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11330: 95291 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: modules unloaded.