== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:45:47 (1787737547) Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg319-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg319-server oleg319-server: oleg319-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg319-server: detected 4 online CPUs by sysfs oleg319-server: MODOPTS_LIBCFS= oleg319-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg319-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg319-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg319-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 699 (time 1787737734.70 total 10.00 last 69.90) - open/close 1318 (time 1787737744.70 total 20.01 last 61.86) - open/close 2081 (time 1787737754.72 total 30.02 last 76.21) - open/close 2800 (time 1787737764.72 total 40.02 last 71.87) - open/close 3499 (time 1787737774.74 total 50.04 last 69.76) - open/close 4160 (time 1787737784.74 total 60.05 last 66.09) total: 4500 open/close in 64.80 seconds: 69.44 ops/second - /unlink 1237 (time 1787737803.82 total 10.00 last 123.65) - /unlink 2492 (time 1787737813.84 total 20.02 last 125.28) - /unlink 3727 (time 1787737823.84 total 30.02 last 123.46) total: 4500 /unlink in 37.25 seconds: 120.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 544 (time 1787737847.44 total 10.01 last 54.35) - open/close 1188 (time 1787737857.44 total 20.01 last 64.39) - open/close 1880 (time 1787737867.46 total 30.03 last 69.10) - open/close 2547 (time 1787737877.50 total 40.07 last 66.43) - open/close 3199 (time 1787737887.51 total 50.08 last 65.14) - open/close 3749 (time 1787737897.52 total 60.09 last 54.93) - open/close 4455 (time 1787737907.52 total 70.09 last 70.56) total: 4500 open/close in 70.63 seconds: 63.72 ops/second - /unlink 1220 (time 1787737922.31 total 10.00 last 121.96) - /unlink 2313 (time 1787737932.33 total 20.02 last 109.16) - /unlink 3472 (time 1787737942.33 total 30.02 last 115.81) total: 4500 /unlink in 38.39 seconds: 117.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 728 (time 1787737965.62 total 10.00 last 72.78) - open/close 1417 (time 1787737975.62 total 20.01 last 68.86) - open/close 2113 (time 1787737985.63 total 30.01 last 69.56) - open/close 2785 (time 1787737995.64 total 40.02 last 67.15) - open/close 3457 (time 1787738005.64 total 50.03 last 67.15) - open/close 4146 (time 1787738015.66 total 60.04 last 68.82) total: 4500 open/close in 65.03 seconds: 69.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1232 (time 1787738034.75 total 10.01 last 123.08) - /unlink 2586 (time 1787738044.75 total 20.02 last 135.32) - /unlink 3873 (time 1787738054.76 total 30.02 last 128.60) total: 4500 /unlink in 36.01 seconds: 124.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 664 (time 1787738074.95 total 10.01 last 66.35) - open/close 1389 (time 1787738084.96 total 20.01 last 72.45) - open/close 2074 (time 1787738094.97 total 30.02 last 68.44) - open/close 2795 (time 1787738104.97 total 40.03 last 72.08) - open/close 3491 (time 1787738114.98 total 50.03 last 69.55) - open/close 4191 (time 1787738124.99 total 60.04 last 69.93) total: 4500 open/close in 63.82 seconds: 70.51 ops/second - /unlink 1319 (time 1787738143.03 total 10.01 last 131.78) - /unlink 2626 (time 1787738153.03 total 20.01 last 130.68) - /unlink 3885 (time 1787738163.03 total 30.01 last 125.90) total: 4500 /unlink in 35.02 seconds: 128.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 713 (time 1787738182.27 total 10.00 last 71.30) - open/close 1401 (time 1787738192.28 total 20.00 last 68.78) - open/close 2045 (time 1787738202.29 total 30.02 last 64.31) - open/close 2690 (time 1787738212.29 total 40.02 last 64.49) - open/close 3343 (time 1787738222.30 total 50.02 last 65.29) - open/close 4100 (time 1787738232.30 total 60.03 last 75.65) total: 4500 open/close in 65.85 seconds: 68.33 ops/second - /unlink 1305 (time 1787738252.79 total 10.01 last 130.43) - /unlink 2443 (time 1787738262.80 total 20.01 last 113.76) - /unlink 3669 (time 1787738272.80 total 30.01 last 122.58) total: 4500 /unlink in 36.10 seconds: 124.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 704 (time 1787738293.14 total 10.01 last 70.36) - open/close 1430 (time 1787738303.15 total 20.01 last 72.54) - open/close 2125 (time 1787738313.16 total 30.03 last 69.40) - open/close 2814 (time 1787738323.17 total 40.04 last 68.84) - open/close 3520 (time 1787738333.17 total 50.04 last 70.59) - open/close 4194 (time 1787738343.17 total 60.04 last 67.38) total: 4500 open/close in 62.31 seconds: 72.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1327 (time 1787738359.72 total 10.00 last 132.65) - /unlink 2556 (time 1787738369.73 total 20.01 last 122.83) - /unlink 3885 (time 1787738379.73 total 30.01 last 132.86) total: 4500 /unlink in 34.91 seconds: 128.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 716 (time 1787738399.12 total 10.00 last 71.57) - open/close 1390 (time 1787738409.12 total 20.01 last 67.37) - open/close 2055 (time 1787738419.15 total 30.04 last 66.31) - open/close 2717 (time 1787738429.16 total 40.05 last 66.12) - open/close 3317 (time 1787738439.17 total 50.06 last 59.95) - open/close 4107 (time 1787738449.18 total 60.07 last 78.91) total: 4500 open/close in 65.20 seconds: 69.01 ops/second - /unlink 1423 (time 1787738468.10 total 10.00 last 142.29) - /unlink 2715 (time 1787738478.11 total 20.01 last 129.05) - /unlink 3927 (time 1787738488.12 total 30.02 last 121.07) total: 4500 /unlink in 35.01 seconds: 128.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 683 (time 1787738508.35 total 10.00 last 68.27) - open/close 1315 (time 1787738518.35 total 20.01 last 63.18) - open/close 2024 (time 1787738528.37 total 30.02 last 70.81) - open/close 2775 (time 1787738538.38 total 40.04 last 74.97) - open/close 3515 (time 1787738548.39 total 50.04 last 73.98) - open/close 4260 (time 1787738558.40 total 60.06 last 74.38) total: 4500 open/close in 63.41 seconds: 70.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1391 (time 1787738575.44 total 10.01 last 138.99) - /unlink 2741 (time 1787738585.44 total 20.01 last 134.92) - /unlink 4027 (time 1787738595.45 total 30.02 last 128.57) total: 4500 /unlink in 33.79 seconds: 133.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 551 (time 1787738614.42 total 10.01 last 55.05) - open/close 1104 (time 1787738624.42 total 20.01 last 55.29) - open/close 1735 (time 1787738634.42 total 30.01 last 63.10) - open/close 2433 (time 1787738644.43 total 40.02 last 69.74) - open/close 3117 (time 1787738654.43 total 50.02 last 68.38) - open/close 3747 (time 1787738664.44 total 60.03 last 62.96) - open/close 4362 (time 1787738674.44 total 70.03 last 61.47) total: 4500 open/close in 71.91 seconds: 62.57 ops/second - /unlink 1295 (time 1787738690.44 total 10.00 last 129.44) - /unlink 2589 (time 1787738700.44 total 20.01 last 129.37) - /unlink 3840 (time 1787738710.45 total 30.02 last 124.98) total: 4500 /unlink in 35.23 seconds: 127.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 641 (time 1787738730.28 total 10.01 last 64.05) - open/close 1365 (time 1787738740.28 total 20.01 last 72.37) - open/close 2060 (time 1787738750.29 total 30.02 last 69.48) - open/close 2795 (time 1787738760.30 total 40.03 last 73.39) - open/close 3481 (time 1787738770.30 total 50.03 last 68.58) - open/close 4186 (time 1787738780.33 total 60.06 last 70.32) total: 4500 open/close in 66.01 seconds: 68.17 ops/second - /unlink 1362 (time 1787738801.08 total 10.00 last 136.14) - /unlink 2668 (time 1787738811.09 total 20.01 last 130.50) - /unlink 4027 (time 1787738821.09 total 30.02 last 135.83) total: 4500 /unlink in 33.41 seconds: 134.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 674 (time 1787738839.01 total 10.00 last 67.38) - open/close 1334 (time 1787738849.01 total 20.01 last 65.97) - open/close 2163 (time 1787738859.01 total 30.01 last 82.87) - open/close 2837 (time 1787738869.03 total 40.03 last 67.31) - open/close 3490 (time 1787738879.03 total 50.03 last 65.29) - open/close 4151 (time 1787738889.04 total 60.04 last 66.04) total: 4500 open/close in 65.52 seconds: 68.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1344 (time 1787738908.96 total 10.00 last 134.35) - /unlink 2736 (time 1787738918.97 total 20.01 last 139.08) - /unlink 4084 (time 1787738928.97 total 30.01 last 134.77) total: 4500 /unlink in 33.51 seconds: 134.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 760 (time 1787738947.75 total 10.01 last 75.96) - open/close 1515 (time 1787738957.76 total 20.02 last 75.42) - open/close 2242 (time 1787738967.77 total 30.03 last 72.63) - open/close 2863 (time 1787738977.78 total 40.03 last 62.06) - open/close 3506 (time 1787738987.80 total 50.06 last 64.11) - open/close 4097 (time 1787738997.81 total 60.07 last 59.06) total: 4500 open/close in 66.07 seconds: 68.10 ops/second - /unlink 1270 (time 1787739017.90 total 10.01 last 126.93) - /unlink 2492 (time 1787739027.91 total 20.01 last 122.13) - /unlink 3561 (time 1787739037.91 total 30.02 last 106.85) total: 4500 /unlink in 38.61 seconds: 116.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 721 (time 1787739061.02 total 10.00 last 72.08) - open/close 1411 (time 1787739071.02 total 20.01 last 68.97) - open/close 2096 (time 1787739081.02 total 30.01 last 68.48) - open/close 2811 (time 1787739091.04 total 40.03 last 71.34) - open/close 3437 (time 1787739101.05 total 50.03 last 62.59) - open/close 4122 (time 1787739111.05 total 60.04 last 68.47) total: 4500 open/close in 65.75 seconds: 68.44 ops/second - /unlink 1276 (time 1787739130.97 total 10.00 last 127.58) - /unlink 2689 (time 1787739140.97 total 20.00 last 141.27) - /unlink 3978 (time 1787739150.97 total 30.01 last 128.84) total: 4500 /unlink in 34.35 seconds: 131.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 744 (time 1787739169.56 total 10.01 last 74.36) - open/close 1427 (time 1787739179.57 total 20.01 last 68.27) - open/close 2134 (time 1787739189.58 total 30.02 last 70.66) - open/close 2864 (time 1787739199.58 total 40.02 last 72.96) - open/close 3647 (time 1787739209.60 total 50.04 last 78.15) - open/close 4358 (time 1787739219.60 total 60.04 last 71.09) total: 4500 open/close in 61.97 seconds: 72.62 ops/second - /unlink 1330 (time 1787739235.79 total 10.01 last 132.87) - /unlink 2642 (time 1787739245.80 total 20.01 last 131.16) - /unlink 3945 (time 1787739255.80 total 30.01 last 130.28) total: 4500 /unlink in 34.17 seconds: 131.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 707 (time 1787739275.03 total 10.01 last 70.64) - open/close 1207 (time 1787739285.04 total 20.01 last 49.99) - open/close 1671 (time 1787739295.06 total 30.03 last 46.31) - open/close 2332 (time 1787739305.06 total 40.03 last 66.07) - open/close 3018 (time 1787739315.07 total 50.05 last 68.52) - open/close 3765 (time 1787739325.08 total 60.06 last 74.61) - open/close 4399 (time 1787739335.08 total 70.06 last 63.40) total: 4500 open/close in 71.41 seconds: 63.02 ops/second - /unlink 1307 (time 1787739351.54 total 10.00 last 130.64) - /unlink 2602 (time 1787739361.54 total 20.01 last 129.49) - /unlink 4018 (time 1787739371.55 total 30.01 last 141.54) total: 4500 /unlink in 33.49 seconds: 134.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 702 (time 1787739389.64 total 10.02 last 70.08) - open/close 1409 (time 1787739399.65 total 20.03 last 70.63) - open/close 2148 (time 1787739409.67 total 30.04 last 73.78) - open/close 2829 (time 1787739419.67 total 40.05 last 68.09) - open/close 3479 (time 1787739429.67 total 50.05 last 64.98) - open/close 4150 (time 1787739439.68 total 60.05 last 67.09) total: 4500 open/close in 65.70 seconds: 68.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1343 (time 1787739460.48 total 10.00 last 134.25) - /unlink 2547 (time 1787739470.48 total 20.00 last 120.39) - /unlink 3804 (time 1787739480.49 total 30.02 last 125.57) total: 4500 /unlink in 35.00 seconds: 128.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 647 (time 1787739499.90 total 10.01 last 64.66) - open/close 1217 (time 1787739509.90 total 20.01 last 56.97) - open/close 1873 (time 1787739519.91 total 30.02 last 65.56) - open/close 2597 (time 1787739529.91 total 40.02 last 72.39) - open/close 3261 (time 1787739539.92 total 50.03 last 66.34) - open/close 3993 (time 1787739549.94 total 60.05 last 73.04) total: 4500 open/close in 67.63 seconds: 66.53 ops/second - /unlink 1120 (time 1787739571.49 total 10.01 last 111.92) - /unlink 2306 (time 1787739581.49 total 20.01 last 118.53) - /unlink 3438 (time 1787739591.50 total 30.02 last 113.10) total: 4500 /unlink in 39.01 seconds: 115.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 719 (time 1787739615.79 total 10.01 last 71.86) - open/close 1314 (time 1787739625.79 total 20.01 last 59.50) - open/close 1886 (time 1787739635.79 total 30.01 last 57.16) - open/close 2487 (time 1787739645.81 total 40.03 last 59.99) - open/close 3110 (time 1787739655.83 total 50.05 last 62.17) - open/close 3771 (time 1787739665.83 total 60.05 last 66.10) total: 4500 open/close in 69.26 seconds: 64.97 ops/second - /unlink 1283 (time 1787739689.48 total 10.01 last 128.16) - /unlink 2566 (time 1787739699.48 total 20.01 last 128.25) - /unlink 3868 (time 1787739709.48 total 30.02 last 130.18) total: 4500 /unlink in 34.91 seconds: 128.92 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95229 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server unloading modules via unload_modules_local on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local oleg319-server: modules unloaded.