== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:08:20 (1769756900) Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg308-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg308-server oleg308-server: oleg308-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: MODOPTS_LIBCFS= oleg308-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg308-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg308-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 992 (time 1769757090.79 total 10.00 last 99.19) - open/close 1993 (time 1769757100.81 total 20.02 last 99.92) - open/close 3043 (time 1769757110.82 total 30.03 last 104.93) - open/close 3905 (time 1769757120.83 total 40.04 last 86.10) total: 4500 open/close in 47.29 seconds: 95.15 ops/second - /unlink 1816 (time 1769757142.73 total 10.00 last 181.58) - /unlink 3655 (time 1769757152.73 total 20.00 last 183.87) total: 4500 /unlink in 25.56 seconds: 176.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 914 (time 1769757172.49 total 10.01 last 91.35) - open/close 1893 (time 1769757182.49 total 20.01 last 97.87) - open/close 2763 (time 1769757192.52 total 30.04 last 86.77) - open/close 3688 (time 1769757202.53 total 40.05 last 92.38) total: 4500 open/close in 48.18 seconds: 93.40 ops/second - /unlink 1725 (time 1769757224.61 total 10.00 last 172.49) - /unlink 3290 (time 1769757234.61 total 20.01 last 156.42) total: 4500 /unlink in 27.58 seconds: 163.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 1010 (time 1769757255.65 total 10.01 last 100.93) - open/close 1845 (time 1769757265.66 total 20.01 last 83.47) - open/close 2814 (time 1769757275.66 total 30.02 last 96.84) - open/close 3712 (time 1769757285.66 total 40.02 last 89.79) total: 4500 open/close in 48.12 seconds: 93.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1652 (time 1769757308.03 total 10.00 last 165.13) - /unlink 3112 (time 1769757318.03 total 20.01 last 145.95) total: 4500 /unlink in 28.61 seconds: 157.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 935 (time 1769757340.73 total 10.01 last 93.42) - open/close 1935 (time 1769757350.74 total 20.02 last 99.92) - open/close 2933 (time 1769757360.74 total 30.02 last 99.78) - open/close 3865 (time 1769757370.75 total 40.02 last 93.14) total: 4500 open/close in 47.81 seconds: 94.13 ops/second - /unlink 1740 (time 1769757392.95 total 10.01 last 173.91) - /unlink 3318 (time 1769757402.96 total 20.01 last 157.65) total: 4500 /unlink in 28.23 seconds: 159.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 708 (time 1769757426.28 total 10.03 last 70.61) - open/close 1429 (time 1769757436.28 total 20.03 last 72.08) - open/close 2329 (time 1769757446.28 total 30.03 last 89.99) - open/close 3189 (time 1769757456.28 total 40.03 last 85.98) - open/close 4078 (time 1769757466.28 total 50.03 last 88.88) total: 4500 open/close in 54.39 seconds: 82.73 ops/second - /unlink 1558 (time 1769757484.90 total 10.00 last 155.78) - /unlink 3147 (time 1769757494.90 total 20.00 last 158.89) total: 4500 /unlink in 28.15 seconds: 159.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 932 (time 1769757517.65 total 10.01 last 93.15) - open/close 1804 (time 1769757527.65 total 20.01 last 87.19) - open/close 2661 (time 1769757537.66 total 30.02 last 85.62) - open/close 3605 (time 1769757547.66 total 40.02 last 94.40) - open/close 4484 (time 1769757557.66 total 50.02 last 87.87) total: 4500 open/close in 50.20 seconds: 89.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1884 (time 1769757571.60 total 10.01 last 188.25) - /unlink 3503 (time 1769757581.60 total 20.01 last 161.84) total: 4500 /unlink in 25.99 seconds: 173.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 978 (time 1769757601.69 total 10.01 last 97.72) - open/close 1964 (time 1769757611.70 total 20.02 last 98.50) - open/close 2985 (time 1769757621.71 total 30.02 last 102.07) - open/close 3885 (time 1769757631.72 total 40.04 last 89.86) total: 4500 open/close in 46.72 seconds: 96.32 ops/second - /unlink 1750 (time 1769757652.37 total 10.02 last 174.66) - /unlink 3429 (time 1769757662.38 total 20.02 last 167.88) total: 4500 /unlink in 26.25 seconds: 171.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 965 (time 1769757682.63 total 10.00 last 96.46) - open/close 1946 (time 1769757692.67 total 20.04 last 97.77) - open/close 2895 (time 1769757702.68 total 30.05 last 94.75) - open/close 3814 (time 1769757712.69 total 40.07 last 91.80) total: 4500 open/close in 47.81 seconds: 94.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1647 (time 1769757734.14 total 10.00 last 164.63) - /unlink 3382 (time 1769757744.14 total 20.01 last 173.49) total: 4500 /unlink in 26.82 seconds: 167.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 998 (time 1769757764.55 total 10.00 last 99.79) - open/close 2014 (time 1769757774.55 total 20.00 last 101.57) - open/close 3066 (time 1769757784.56 total 30.01 last 105.18) - open/close 4083 (time 1769757794.56 total 40.01 last 101.63) total: 4500 open/close in 43.83 seconds: 102.66 ops/second - /unlink 1459 (time 1769757811.46 total 10.00 last 145.87) - /unlink 3139 (time 1769757821.47 total 20.01 last 167.91) total: 4500 /unlink in 27.80 seconds: 161.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1133 (time 1769757842.89 total 10.00 last 113.28) - open/close 2124 (time 1769757852.89 total 20.01 last 99.02) - open/close 3045 (time 1769757862.90 total 30.01 last 92.07) - open/close 4009 (time 1769757872.91 total 40.02 last 96.33) total: 4500 open/close in 44.87 seconds: 100.29 ops/second - /unlink 1888 (time 1769757890.98 total 10.00 last 188.77) - /unlink 3791 (time 1769757901.00 total 20.01 last 190.09) total: 4500 /unlink in 24.25 seconds: 185.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 979 (time 1769757919.43 total 10.01 last 97.84) - open/close 1875 (time 1769757929.43 total 20.01 last 89.56) - open/close 2875 (time 1769757939.44 total 30.02 last 99.88) - open/close 3824 (time 1769757949.45 total 40.03 last 94.83) total: 4500 open/close in 46.79 seconds: 96.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1813 (time 1769757969.66 total 10.00 last 181.25) - /unlink 3831 (time 1769757979.67 total 20.01 last 201.66) total: 4500 /unlink in 23.02 seconds: 195.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1066 (time 1769757996.04 total 10.01 last 106.46) - open/close 2216 (time 1769758006.04 total 20.01 last 114.99) - open/close 3267 (time 1769758016.05 total 30.02 last 105.02) - open/close 4349 (time 1769758026.05 total 40.03 last 108.15) total: 4500 open/close in 41.23 seconds: 109.15 ops/second - /unlink 1786 (time 1769758040.17 total 10.00 last 178.55) - /unlink 3655 (time 1769758050.17 total 20.01 last 186.83) total: 4500 /unlink in 24.88 seconds: 180.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1020 (time 1769758068.92 total 10.00 last 101.97) - open/close 2063 (time 1769758078.92 total 20.01 last 104.28) - open/close 3200 (time 1769758088.94 total 30.02 last 113.53) - open/close 4224 (time 1769758098.94 total 40.02 last 102.38) total: 4500 open/close in 42.52 seconds: 105.83 ops/second - /unlink 1964 (time 1769758114.00 total 10.00 last 196.38) - /unlink 4160 (time 1769758124.00 total 20.01 last 219.51) total: 4500 /unlink in 21.78 seconds: 206.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 909 (time 1769758138.42 total 10.01 last 90.83) - open/close 2063 (time 1769758148.43 total 20.01 last 115.37) - open/close 3251 (time 1769758158.44 total 30.02 last 118.71) - open/close 4441 (time 1769758168.45 total 40.03 last 118.87) total: 4500 open/close in 40.65 seconds: 110.69 ops/second - /unlink 2174 (time 1769758181.78 total 10.00 last 217.37) - /unlink 4287 (time 1769758191.78 total 20.00 last 211.26) total: 4500 /unlink in 21.22 seconds: 212.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 1103 (time 1769758205.76 total 10.01 last 110.18) - open/close 2215 (time 1769758215.76 total 20.02 last 111.15) - open/close 3341 (time 1769758225.77 total 30.02 last 112.52) - open/close 4452 (time 1769758235.78 total 40.03 last 111.05) total: 4500 open/close in 40.44 seconds: 111.27 ops/second - /unlink 1977 (time 1769758249.06 total 10.00 last 197.61) - /unlink 4102 (time 1769758259.06 total 20.01 last 212.49) total: 4500 /unlink in 22.03 seconds: 204.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1229 (time 1769758273.84 total 10.00 last 122.86) - open/close 2367 (time 1769758283.84 total 20.00 last 113.78) - open/close 3570 (time 1769758293.85 total 30.01 last 120.23) total: 4500 open/close in 37.72 seconds: 119.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2422 (time 1769758314.28 total 10.00 last 242.14) - /unlink 3983 (time 1769758324.28 total 20.00 last 156.10) total: 4500 /unlink in 22.53 seconds: 199.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1543 (time 1769758340.60 total 10.00 last 154.26) - open/close 2996 (time 1769758350.60 total 20.00 last 145.29) - open/close 4160 (time 1769758360.61 total 30.01 last 116.31) total: 4500 open/close in 32.62 seconds: 137.95 ops/second - /unlink 2107 (time 1769758375.68 total 10.01 last 210.58) - /unlink 3756 (time 1769758385.69 total 20.01 last 164.75) total: 4500 /unlink in 23.83 seconds: 188.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1171 (time 1769758402.31 total 10.00 last 117.07) - open/close 2333 (time 1769758412.31 total 20.00 last 116.17) - open/close 3566 (time 1769758422.32 total 30.01 last 123.22) total: 4500 open/close in 37.82 seconds: 118.98 ops/second - /unlink 2279 (time 1769758442.73 total 10.00 last 227.87) total: 4500 /unlink in 19.53 seconds: 230.44 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 93974 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.