== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:05:03 (1783973103) Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg417-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg417-server oleg417-server: oleg417-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg417-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg417-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 620 (time 1783973325.89 total 10.01 last 61.96) - open/close 1281 (time 1783973335.89 total 20.01 last 66.08) - open/close 1922 (time 1783973345.89 total 30.01 last 64.08) - open/close 2581 (time 1783973355.90 total 40.02 last 65.86) - open/close 3235 (time 1783973365.91 total 50.02 last 65.37) - open/close 3940 (time 1783973375.91 total 60.03 last 70.49) total: 4500 open/close in 69.28 seconds: 64.95 ops/second - /unlink 1183 (time 1783973399.71 total 10.00 last 118.26) - /unlink 2328 (time 1783973409.72 total 20.01 last 114.45) - /unlink 3499 (time 1783973419.73 total 30.02 last 116.99) total: 4500 /unlink in 38.44 seconds: 117.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 684 (time 1783973443.15 total 10.01 last 68.33) - open/close 1329 (time 1783973453.16 total 20.02 last 64.43) - open/close 1973 (time 1783973463.17 total 30.03 last 64.34) - open/close 2615 (time 1783973473.18 total 40.04 last 64.17) - open/close 3181 (time 1783973483.19 total 50.05 last 56.55) - open/close 3643 (time 1783973493.19 total 60.05 last 46.19) - open/close 4194 (time 1783973503.20 total 70.06 last 55.05) total: 4500 open/close in 74.76 seconds: 60.20 ops/second - /unlink 1151 (time 1783973523.16 total 10.00 last 115.04) - /unlink 2264 (time 1783973533.17 total 20.02 last 111.12) - /unlink 3456 (time 1783973543.18 total 30.02 last 119.18) total: 4500 /unlink in 38.96 seconds: 115.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 620 (time 1783973567.17 total 10.00 last 61.99) - open/close 1280 (time 1783973577.18 total 20.01 last 65.92) - open/close 1965 (time 1783973587.19 total 30.02 last 68.44) - open/close 2570 (time 1783973597.21 total 40.05 last 60.36) - open/close 3111 (time 1783973607.22 total 50.05 last 54.07) - open/close 3652 (time 1783973617.22 total 60.06 last 54.06) - open/close 4304 (time 1783973627.23 total 70.06 last 65.18) total: 4500 open/close in 73.50 seconds: 61.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1211 (time 1783973645.14 total 10.01 last 120.95) - /unlink 2377 (time 1783973655.15 total 20.02 last 116.50) - /unlink 3454 (time 1783973665.15 total 30.02 last 107.69) total: 4500 /unlink in 38.71 seconds: 116.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 589 (time 1783973688.70 total 10.01 last 58.86) - open/close 1243 (time 1783973698.71 total 20.02 last 65.34) - open/close 1903 (time 1783973708.72 total 30.03 last 65.91) - open/close 2538 (time 1783973718.72 total 40.03 last 63.49) - open/close 3235 (time 1783973728.73 total 50.04 last 69.61) - open/close 3864 (time 1783973738.74 total 60.05 last 62.83) total: 4500 open/close in 69.84 seconds: 64.43 ops/second - /unlink 1159 (time 1783973762.52 total 10.01 last 115.78) - /unlink 2370 (time 1783973772.52 total 20.01 last 121.08) - /unlink 3602 (time 1783973782.52 total 30.01 last 123.17) total: 4500 /unlink in 38.04 seconds: 118.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 591 (time 1783973806.17 total 10.01 last 59.06) - open/close 1167 (time 1783973816.18 total 20.02 last 57.55) - open/close 1808 (time 1783973826.18 total 30.02 last 64.05) - open/close 2389 (time 1783973836.19 total 40.03 last 58.06) - open/close 3023 (time 1783973846.19 total 50.03 last 63.39) - open/close 3662 (time 1783973856.20 total 60.04 last 63.85) - open/close 4352 (time 1783973866.20 total 70.04 last 68.95) total: 4500 open/close in 72.14 seconds: 62.38 ops/second - /unlink 1094 (time 1783973882.51 total 10.00 last 109.36) - /unlink 2264 (time 1783973892.52 total 20.02 last 116.85) - /unlink 3300 (time 1783973902.53 total 30.03 last 103.51) total: 4500 /unlink in 39.55 seconds: 113.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 629 (time 1783973927.20 total 10.01 last 62.83) - open/close 1281 (time 1783973937.20 total 20.01 last 65.19) - open/close 1950 (time 1783973947.20 total 30.02 last 66.85) - open/close 2622 (time 1783973957.22 total 40.04 last 67.08) - open/close 3277 (time 1783973967.23 total 50.05 last 65.45) - open/close 3841 (time 1783973977.25 total 60.07 last 56.27) - open/close 4493 (time 1783973987.26 total 70.08 last 65.16) total: 4500 open/close in 70.15 seconds: 64.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1155 (time 1783974001.54 total 10.00 last 115.49) - /unlink 2280 (time 1783974011.54 total 20.00 last 112.49) - /unlink 3347 (time 1783974021.55 total 30.01 last 106.66) - /unlink 4175 (time 1783974031.56 total 40.01 last 82.73) total: 4500 /unlink in 42.95 seconds: 104.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 620 (time 1783974049.26 total 10.01 last 61.95) - open/close 1252 (time 1783974059.28 total 20.03 last 63.09) - open/close 1875 (time 1783974069.29 total 30.04 last 62.22) - open/close 2420 (time 1783974079.31 total 40.05 last 54.43) - open/close 3074 (time 1783974089.31 total 50.06 last 65.38) - open/close 3721 (time 1783974099.32 total 60.07 last 64.63) - open/close 4397 (time 1783974109.33 total 70.07 last 67.56) total: 4500 open/close in 71.65 seconds: 62.80 ops/second - /unlink 1124 (time 1783974125.94 total 10.01 last 112.33) - /unlink 2247 (time 1783974135.94 total 20.01 last 112.26) - /unlink 3427 (time 1783974145.96 total 30.03 last 117.71) total: 4500 /unlink in 38.93 seconds: 115.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 652 (time 1783974168.96 total 10.01 last 65.17) - open/close 1286 (time 1783974178.96 total 20.01 last 63.36) - open/close 1881 (time 1783974188.99 total 30.04 last 59.34) - open/close 2564 (time 1783974199.00 total 40.05 last 68.21) - open/close 3191 (time 1783974209.01 total 50.06 last 62.66) - open/close 3795 (time 1783974219.01 total 60.06 last 60.37) - open/close 4438 (time 1783974229.02 total 70.07 last 64.28) total: 4500 open/close in 70.96 seconds: 63.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1207 (time 1783974244.58 total 10.01 last 120.54) - /unlink 2329 (time 1783974254.59 total 20.02 last 112.10) - /unlink 3389 (time 1783974264.60 total 30.03 last 105.89) - /unlink 4394 (time 1783974274.60 total 40.03 last 100.49) total: 4500 /unlink in 41.12 seconds: 109.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 701 (time 1783974291.47 total 10.00 last 70.09) - open/close 1353 (time 1783974301.47 total 20.01 last 65.17) - open/close 1960 (time 1783974311.48 total 30.01 last 60.66) - open/close 2620 (time 1783974321.49 total 40.02 last 65.93) - open/close 3177 (time 1783974331.51 total 50.04 last 55.59) - open/close 3670 (time 1783974341.57 total 60.10 last 49.01) - open/close 4259 (time 1783974351.57 total 70.11 last 58.88) total: 4500 open/close in 73.61 seconds: 61.14 ops/second - /unlink 1254 (time 1783974369.45 total 10.00 last 125.38) - /unlink 2432 (time 1783974379.46 total 20.01 last 117.72) - /unlink 3607 (time 1783974389.46 total 30.01 last 117.47) total: 4500 /unlink in 37.69 seconds: 119.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 588 (time 1783974411.82 total 10.01 last 58.77) - open/close 1197 (time 1783974421.82 total 20.01 last 60.89) - open/close 1852 (time 1783974431.83 total 30.01 last 65.47) - open/close 2469 (time 1783974441.84 total 40.02 last 61.62) - open/close 3099 (time 1783974451.85 total 50.04 last 62.92) - open/close 3738 (time 1783974461.86 total 60.04 last 63.88) - open/close 4354 (time 1783974471.88 total 70.07 last 61.43) total: 4500 open/close in 73.05 seconds: 61.60 ops/second - /unlink 1111 (time 1783974489.85 total 10.00 last 111.07) - /unlink 2087 (time 1783974499.86 total 20.01 last 97.50) - /unlink 3271 (time 1783974509.88 total 30.02 last 118.28) - /unlink 4290 (time 1783974519.88 total 40.03 last 101.83) total: 4500 /unlink in 42.06 seconds: 106.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 619 (time 1783974536.91 total 10.00 last 61.88) - open/close 1257 (time 1783974546.92 total 20.02 last 63.71) - open/close 1911 (time 1783974556.94 total 30.03 last 65.32) - open/close 2492 (time 1783974566.94 total 40.03 last 58.09) - open/close 3018 (time 1783974576.95 total 50.04 last 52.56) - open/close 3592 (time 1783974586.95 total 60.04 last 57.40) - open/close 4200 (time 1783974596.95 total 70.04 last 60.78) total: 4500 open/close in 74.70 seconds: 60.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1072 (time 1783974617.69 total 10.00 last 107.19) - /unlink 2273 (time 1783974627.69 total 20.00 last 120.07) - /unlink 3452 (time 1783974637.69 total 30.01 last 117.89) total: 4500 /unlink in 39.49 seconds: 113.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 567 (time 1783974662.16 total 10.00 last 56.69) - open/close 1097 (time 1783974672.17 total 20.01 last 52.94) - open/close 1758 (time 1783974682.18 total 30.02 last 66.05) - open/close 2275 (time 1783974692.18 total 40.02 last 51.70) - open/close 2807 (time 1783974702.18 total 50.03 last 53.16) - open/close 3366 (time 1783974712.20 total 60.04 last 55.83) - open/close 3991 (time 1783974722.20 total 70.05 last 62.47) total: 4500 open/close in 78.26 seconds: 57.50 ops/second - /unlink 1048 (time 1783974746.66 total 10.00 last 104.79) - /unlink 2233 (time 1783974756.66 total 20.01 last 118.44) - /unlink 3495 (time 1783974766.66 total 30.01 last 126.19) total: 4500 /unlink in 38.12 seconds: 118.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 615 (time 1783974789.66 total 10.01 last 61.42) - open/close 1277 (time 1783974799.66 total 20.01 last 66.19) - open/close 1933 (time 1783974809.67 total 30.02 last 65.55) - open/close 2565 (time 1783974819.68 total 40.04 last 63.11) - open/close 3151 (time 1783974829.69 total 50.04 last 58.56) - open/close 3820 (time 1783974839.69 total 60.05 last 66.86) - open/close 4498 (time 1783974849.73 total 70.08 last 67.58) total: 4500 open/close in 70.12 seconds: 64.18 ops/second - /unlink 1183 (time 1783974864.90 total 10.01 last 118.23) - /unlink 2418 (time 1783974874.91 total 20.02 last 123.39) - /unlink 3650 (time 1783974884.92 total 30.02 last 123.08) total: 4500 /unlink in 37.69 seconds: 119.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 566 (time 1783974908.00 total 10.01 last 56.56) - open/close 1102 (time 1783974918.01 total 20.01 last 53.56) - open/close 1715 (time 1783974928.02 total 30.03 last 61.22) - open/close 2397 (time 1783974938.04 total 40.04 last 68.10) - open/close 2966 (time 1783974948.04 total 50.04 last 56.88) - open/close 3590 (time 1783974958.05 total 60.05 last 62.35) - open/close 4199 (time 1783974968.06 total 70.06 last 60.84) total: 4500 open/close in 74.90 seconds: 60.08 ops/second - /unlink 1162 (time 1783974987.54 total 10.00 last 116.16) - /unlink 2192 (time 1783974997.54 total 20.01 last 102.96) - /unlink 3439 (time 1783975007.54 total 30.01 last 124.63) - /unlink 4484 (time 1783975017.55 total 40.02 last 104.43) total: 4500 /unlink in 40.24 seconds: 111.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 608 (time 1783975032.76 total 10.01 last 60.72) - open/close 1213 (time 1783975042.77 total 20.02 last 60.44) - open/close 1817 (time 1783975052.78 total 30.03 last 60.34) - open/close 2302 (time 1783975062.78 total 40.04 last 48.48) - open/close 2794 (time 1783975072.79 total 50.04 last 49.17) - open/close 3413 (time 1783975082.80 total 60.05 last 61.85) - open/close 4109 (time 1783975092.81 total 70.06 last 69.55) total: 4500 open/close in 77.19 seconds: 58.30 ops/second - /unlink 964 (time 1783975117.45 total 10.00 last 96.39) - /unlink 1954 (time 1783975127.46 total 20.01 last 98.89) - /unlink 3093 (time 1783975137.48 total 30.03 last 113.67) - /unlink 4297 (time 1783975147.48 total 40.04 last 120.36) total: 4500 /unlink in 41.84 seconds: 107.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 552 (time 1783975165.04 total 10.01 last 55.17) - open/close 1213 (time 1783975175.05 total 20.01 last 66.06) - open/close 1816 (time 1783975185.06 total 30.02 last 60.25) - open/close 2444 (time 1783975195.07 total 40.04 last 62.70) - open/close 3081 (time 1783975205.08 total 50.04 last 63.65) - open/close 3702 (time 1783975215.08 total 60.05 last 62.09) - open/close 4338 (time 1783975225.09 total 70.05 last 63.58) total: 4500 open/close in 72.51 seconds: 62.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1088 (time 1783975242.89 total 10.00 last 108.76) - /unlink 1980 (time 1783975252.91 total 20.02 last 89.08) - /unlink 3063 (time 1783975262.91 total 30.02 last 108.30) - /unlink 4308 (time 1783975272.91 total 40.02 last 124.49) total: 4500 /unlink in 41.77 seconds: 107.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 589 (time 1783975290.65 total 10.03 last 58.75) - open/close 1144 (time 1783975300.65 total 20.03 last 55.49) - open/close 1683 (time 1783975310.67 total 30.04 last 53.83) - open/close 2241 (time 1783975320.68 total 40.06 last 55.71) - open/close 2821 (time 1783975330.68 total 50.06 last 57.98) - open/close 3371 (time 1783975340.69 total 60.07 last 54.95) - open/close 3961 (time 1783975350.70 total 70.07 last 58.99) total: 4500 open/close in 79.61 seconds: 56.53 ops/second - /unlink 922 (time 1783975375.69 total 10.00 last 92.16) - /unlink 2017 (time 1783975385.69 total 20.01 last 109.48) - /unlink 3048 (time 1783975395.70 total 30.01 last 103.02) - /unlink 4096 (time 1783975405.70 total 40.02 last 104.79) total: 4500 /unlink in 43.56 seconds: 103.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 700 (time 1783975424.84 total 10.00 last 70.00) - open/close 1321 (time 1783975434.84 total 20.00 last 62.10) - open/close 1929 (time 1783975444.85 total 30.01 last 60.75) - open/close 2554 (time 1783975454.86 total 40.02 last 62.44) - open/close 3072 (time 1783975464.86 total 50.03 last 51.75) - open/close 3705 (time 1783975474.87 total 60.04 last 63.24) - open/close 4275 (time 1783975484.88 total 70.04 last 56.99) total: 4500 open/close in 73.26 seconds: 61.42 ops/second - /unlink 1139 (time 1783975503.70 total 10.01 last 113.78) - /unlink 2335 (time 1783975513.71 total 20.01 last 119.58) - /unlink 3470 (time 1783975523.71 total 30.01 last 113.49) total: 4500 /unlink in 38.53 seconds: 116.80 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95294 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local oleg417-server: modules unloaded.