== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:15:45 (1782231345) Stopping clients: oleg354-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg354-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg354-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg354-server oleg354-server: oleg354-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing load_modules_local oleg354-server: Loading modules from /home/green/git/lustre-release/lustre oleg354-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg354-server: detected 4 online CPUs by sysfs oleg354-server: MODOPTS_LIBCFS= oleg354-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg354-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg354-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 843 (time 1782231499.71 total 10.00 last 84.27) - open/close 1624 (time 1782231509.72 total 20.01 last 78.05) - open/close 2434 (time 1782231519.73 total 30.02 last 80.91) - open/close 3220 (time 1782231529.73 total 40.02 last 78.60) - open/close 4030 (time 1782231539.73 total 50.03 last 80.95) total: 4500 open/close in 55.60 seconds: 80.93 ops/second - /unlink 1569 (time 1782231558.40 total 10.00 last 156.86) - /unlink 3060 (time 1782231568.40 total 20.00 last 149.10) total: 4500 /unlink in 29.03 seconds: 155.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 850 (time 1782231590.79 total 10.01 last 84.90) - open/close 1706 (time 1782231600.80 total 20.02 last 85.54) - open/close 2557 (time 1782231610.80 total 30.02 last 85.07) - open/close 3362 (time 1782231620.82 total 40.04 last 80.39) - open/close 4232 (time 1782231630.82 total 50.04 last 86.93) total: 4500 open/close in 52.95 seconds: 84.99 ops/second - /unlink 1686 (time 1782231646.93 total 10.00 last 168.52) - /unlink 3155 (time 1782231656.94 total 20.01 last 146.84) total: 4500 /unlink in 29.20 seconds: 154.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 776 (time 1782231680.32 total 10.01 last 77.52) - open/close 1575 (time 1782231690.33 total 20.02 last 79.82) - open/close 2414 (time 1782231700.33 total 30.02 last 83.86) - open/close 3291 (time 1782231710.33 total 40.03 last 87.66) - open/close 4153 (time 1782231720.34 total 50.03 last 86.17) total: 4500 open/close in 53.86 seconds: 83.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1627 (time 1782231737.28 total 10.00 last 162.64) - /unlink 3141 (time 1782231747.28 total 20.00 last 151.40) total: 4500 /unlink in 27.59 seconds: 163.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 965 (time 1782231767.62 total 10.02 last 96.34) - open/close 1797 (time 1782231777.63 total 20.03 last 83.13) - open/close 2767 (time 1782231787.63 total 30.03 last 97.00) - open/close 3667 (time 1782231797.64 total 40.03 last 89.92) total: 4500 open/close in 49.29 seconds: 91.30 ops/second - /unlink 1932 (time 1782231819.41 total 10.00 last 193.15) - /unlink 3939 (time 1782231829.41 total 20.00 last 200.69) total: 4500 /unlink in 22.79 seconds: 197.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 887 (time 1782231845.03 total 10.00 last 88.66) - open/close 1751 (time 1782231855.03 total 20.01 last 86.35) - open/close 2617 (time 1782231865.04 total 30.02 last 86.52) - open/close 3477 (time 1782231875.05 total 40.03 last 85.92) - open/close 4419 (time 1782231885.06 total 50.03 last 94.17) total: 4500 open/close in 50.98 seconds: 88.27 ops/second - /unlink 1601 (time 1782231899.17 total 10.00 last 160.08) - /unlink 3340 (time 1782231909.18 total 20.01 last 173.82) total: 4500 /unlink in 27.13 seconds: 165.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 771 (time 1782231929.25 total 10.01 last 77.02) - open/close 1518 (time 1782231939.25 total 20.01 last 74.68) - open/close 2329 (time 1782231949.26 total 30.02 last 81.06) - open/close 3206 (time 1782231959.27 total 40.02 last 87.64) - open/close 4037 (time 1782231969.27 total 50.03 last 83.05) total: 4500 open/close in 55.31 seconds: 81.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1786 (time 1782231987.40 total 10.00 last 178.59) - /unlink 3606 (time 1782231997.41 total 20.00 last 181.93) total: 4500 /unlink in 24.47 seconds: 183.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 917 (time 1782232014.75 total 10.01 last 91.62) - open/close 1944 (time 1782232024.75 total 20.01 last 102.69) - open/close 2989 (time 1782232034.76 total 30.01 last 104.46) - open/close 3982 (time 1782232044.77 total 40.02 last 99.21) total: 4500 open/close in 45.44 seconds: 99.03 ops/second - /unlink 1815 (time 1782232063.04 total 10.00 last 181.46) - /unlink 3658 (time 1782232073.04 total 20.00 last 184.26) total: 4500 /unlink in 24.61 seconds: 182.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 888 (time 1782232090.50 total 10.00 last 88.78) - open/close 1713 (time 1782232100.51 total 20.01 last 82.40) - open/close 2596 (time 1782232110.52 total 30.03 last 88.19) - open/close 3492 (time 1782232120.53 total 40.04 last 89.52) - open/close 4344 (time 1782232130.54 total 50.04 last 85.15) total: 4500 open/close in 51.74 seconds: 86.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1629 (time 1782232144.91 total 10.00 last 162.85) - /unlink 3295 (time 1782232154.91 total 20.01 last 166.53) total: 4500 /unlink in 27.30 seconds: 164.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 846 (time 1782232175.35 total 10.00 last 84.58) - open/close 1800 (time 1782232185.36 total 20.01 last 95.32) - open/close 2692 (time 1782232195.37 total 30.02 last 89.09) - open/close 3588 (time 1782232205.37 total 40.03 last 89.58) - open/close 4499 (time 1782232215.39 total 50.04 last 90.99) total: 4500 open/close in 50.04 seconds: 89.93 ops/second - /unlink 2147 (time 1782232228.25 total 10.00 last 214.64) - /unlink 3965 (time 1782232238.25 total 20.00 last 181.77) total: 4500 /unlink in 23.10 seconds: 194.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 930 (time 1782232254.29 total 10.01 last 92.93) - open/close 1916 (time 1782232264.29 total 20.01 last 98.55) - open/close 2880 (time 1782232274.30 total 30.02 last 96.34) - open/close 3892 (time 1782232284.30 total 40.02 last 101.15) total: 4500 open/close in 45.47 seconds: 98.97 ops/second - /unlink 2151 (time 1782232301.89 total 10.01 last 214.96) - /unlink 4394 (time 1782232311.89 total 20.01 last 224.28) total: 4500 /unlink in 20.46 seconds: 219.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1254 (time 1782232324.94 total 10.00 last 125.36) - open/close 2325 (time 1782232334.95 total 20.01 last 107.03) - open/close 3356 (time 1782232344.95 total 30.01 last 103.09) - open/close 4417 (time 1782232354.95 total 40.02 last 106.04) total: 4500 open/close in 40.78 seconds: 110.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1793 (time 1782232368.47 total 10.01 last 179.20) - /unlink 3576 (time 1782232378.47 total 20.01 last 178.29) total: 4500 /unlink in 24.80 seconds: 181.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 911 (time 1782232396.30 total 10.01 last 90.99) - open/close 1815 (time 1782232406.31 total 20.02 last 90.36) - open/close 2719 (time 1782232416.32 total 30.02 last 90.35) - open/close 3595 (time 1782232426.33 total 40.03 last 87.49) total: 4500 open/close in 49.46 seconds: 90.98 ops/second - /unlink 1866 (time 1782232448.24 total 10.00 last 186.59) - /unlink 3601 (time 1782232458.25 total 20.01 last 173.37) total: 4500 /unlink in 24.97 seconds: 180.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 870 (time 1782232475.96 total 10.01 last 86.93) - open/close 1726 (time 1782232485.96 total 20.01 last 85.60) - open/close 2599 (time 1782232495.97 total 30.02 last 87.22) - open/close 3478 (time 1782232505.98 total 40.02 last 87.88) - open/close 4478 (time 1782232515.98 total 50.02 last 99.96) total: 4500 open/close in 50.23 seconds: 89.58 ops/second - /unlink 2019 (time 1782232528.83 total 10.00 last 201.88) - /unlink 3854 (time 1782232538.83 total 20.00 last 183.46) total: 4500 /unlink in 23.49 seconds: 191.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 999 (time 1782232554.95 total 10.01 last 99.78) - open/close 1909 (time 1782232564.95 total 20.02 last 90.97) - open/close 2926 (time 1782232574.96 total 30.02 last 101.64) - open/close 3895 (time 1782232584.96 total 40.03 last 96.84) total: 4500 open/close in 48.13 seconds: 93.50 ops/second - /unlink 988 (time 1782232607.85 total 10.00 last 98.78) - /unlink 1950 (time 1782232617.85 total 20.00 last 96.18) - /unlink 2716 (time 1782232627.87 total 30.02 last 76.50) - /unlink 3505 (time 1782232637.87 total 40.02 last 78.90) - /unlink 4108 (time 1782232647.89 total 50.04 last 60.15) total: 4500 /unlink in 54.59 seconds: 82.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 637 (time 1782232670.10 total 10.00 last 63.68) - open/close 1309 (time 1782232680.10 total 20.01 last 67.19) - open/close 1923 (time 1782232690.10 total 30.01 last 61.39) - open/close 2612 (time 1782232700.11 total 40.01 last 68.86) - open/close 3264 (time 1782232710.12 total 50.03 last 65.12) - open/close 3936 (time 1782232720.12 total 60.03 last 67.18) total: 4500 open/close in 67.99 seconds: 66.19 ops/second - /unlink 1205 (time 1782232742.98 total 10.00 last 120.48) - /unlink 2463 (time 1782232752.98 total 20.00 last 125.78) - /unlink 3653 (time 1782232762.99 total 30.01 last 118.87) total: 4500 /unlink in 37.02 seconds: 121.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 719 (time 1782232784.53 total 10.04 last 71.65) - open/close 1340 (time 1782232794.54 total 20.04 last 62.07) - open/close 1887 (time 1782232804.57 total 30.07 last 54.52) - open/close 2534 (time 1782232814.57 total 40.07 last 64.68) - open/close 3149 (time 1782232824.58 total 50.09 last 61.43) - open/close 3807 (time 1782232834.58 total 60.09 last 65.78) - open/close 4452 (time 1782232844.59 total 70.10 last 64.45) total: 4500 open/close in 70.87 seconds: 63.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1215 (time 1782232860.32 total 10.00 last 121.47) - /unlink 2346 (time 1782232870.32 total 20.01 last 113.06) - /unlink 3511 (time 1782232880.32 total 30.01 last 116.49) total: 4500 /unlink in 38.34 seconds: 117.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 633 (time 1782232903.94 total 10.02 last 63.21) - open/close 1366 (time 1782232913.95 total 20.02 last 73.24) - open/close 2078 (time 1782232923.96 total 30.04 last 71.11) - open/close 2799 (time 1782232933.97 total 40.04 last 72.05) - open/close 3475 (time 1782232943.97 total 50.05 last 67.56) - open/close 4156 (time 1782232953.98 total 60.05 last 68.06) total: 4500 open/close in 65.37 seconds: 68.84 ops/second - /unlink 1271 (time 1782232973.94 total 10.00 last 127.08) - /unlink 2451 (time 1782232983.94 total 20.00 last 117.97) - /unlink 3574 (time 1782232993.94 total 30.01 last 112.29) total: 4500 /unlink in 37.00 seconds: 121.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 767 (time 1782233015.50 total 10.01 last 76.65) - open/close 1381 (time 1782233025.53 total 20.03 last 61.23) - open/close 1967 (time 1782233035.53 total 30.04 last 58.59) - open/close 2555 (time 1782233045.55 total 40.05 last 58.73) - open/close 3219 (time 1782233055.56 total 50.06 last 66.31) - open/close 3920 (time 1782233065.57 total 60.07 last 70.04) total: 4500 open/close in 68.24 seconds: 65.95 ops/second - /unlink 1252 (time 1782233088.49 total 10.01 last 125.13) - /unlink 2489 (time 1782233098.49 total 20.01 last 123.67) - /unlink 3691 (time 1782233108.50 total 30.01 last 120.13) total: 4500 /unlink in 37.12 seconds: 121.24 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94972 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg354-server unloading modules via unload_modules_local on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local oleg354-server: modules unloaded.