== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:07:21 (1780340841) Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg412-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg412-server oleg412-server: oleg412-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg412-server: detected 4 online CPUs by sysfs oleg412-server: MODOPTS_LIBCFS= oleg412-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg412-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg412-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg412-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 589 (time 1780341066.78 total 10.02 last 58.78) - open/close 1159 (time 1780341076.79 total 20.03 last 56.97) - open/close 1731 (time 1780341086.80 total 30.03 last 57.16) - open/close 2213 (time 1780341096.81 total 40.05 last 48.11) - open/close 2721 (time 1780341106.82 total 50.05 last 50.78) - open/close 3271 (time 1780341116.84 total 60.08 last 54.88) - open/close 3851 (time 1780341126.84 total 70.08 last 57.98) total: 4500 open/close in 78.87 seconds: 57.05 ops/second - /unlink 880 (time 1780341151.16 total 10.00 last 87.99) - /unlink 1778 (time 1780341161.17 total 20.01 last 89.75) - /unlink 2972 (time 1780341171.17 total 30.01 last 119.33) - /unlink 4139 (time 1780341181.17 total 40.01 last 116.69) total: 4500 /unlink in 43.42 seconds: 103.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 670 (time 1780341200.09 total 10.01 last 66.95) - open/close 1290 (time 1780341210.11 total 20.03 last 61.89) - open/close 1863 (time 1780341220.12 total 30.03 last 57.25) - open/close 2518 (time 1780341230.14 total 40.06 last 65.35) - open/close 3195 (time 1780341240.14 total 50.06 last 67.69) - open/close 3884 (time 1780341250.15 total 60.07 last 68.85) total: 4500 open/close in 69.48 seconds: 64.77 ops/second - /unlink 1244 (time 1780341274.62 total 10.00 last 124.39) - /unlink 2441 (time 1780341284.63 total 20.01 last 119.64) - /unlink 3549 (time 1780341294.64 total 30.02 last 110.66) total: 4500 /unlink in 37.69 seconds: 119.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 657 (time 1780341317.66 total 10.00 last 65.67) - open/close 1276 (time 1780341327.67 total 20.01 last 61.87) - open/close 1962 (time 1780341337.67 total 30.01 last 68.56) - open/close 2688 (time 1780341347.68 total 40.02 last 72.59) - open/close 3343 (time 1780341357.69 total 50.03 last 65.43) - open/close 3972 (time 1780341367.71 total 60.05 last 62.77) total: 4500 open/close in 66.86 seconds: 67.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1202 (time 1780341389.28 total 10.00 last 120.18) - /unlink 2451 (time 1780341399.28 total 20.00 last 124.87) - /unlink 3700 (time 1780341409.29 total 30.01 last 124.82) total: 4500 /unlink in 36.58 seconds: 123.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 662 (time 1780341430.95 total 10.01 last 66.11) - open/close 1296 (time 1780341440.95 total 20.02 last 63.35) - open/close 1998 (time 1780341450.97 total 30.03 last 70.12) - open/close 2717 (time 1780341460.97 total 40.04 last 71.86) - open/close 3371 (time 1780341470.97 total 50.04 last 65.40) - open/close 4024 (time 1780341481.02 total 60.08 last 65.01) total: 4500 open/close in 67.80 seconds: 66.38 ops/second - /unlink 1176 (time 1780341504.34 total 10.00 last 117.56) - /unlink 2306 (time 1780341514.35 total 20.01 last 112.93) - /unlink 3486 (time 1780341524.35 total 30.01 last 117.99) total: 4500 /unlink in 38.60 seconds: 116.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 655 (time 1780341547.71 total 10.00 last 65.48) - open/close 1255 (time 1780341557.71 total 20.00 last 60.00) - open/close 1884 (time 1780341567.73 total 30.02 last 62.78) - open/close 2548 (time 1780341577.73 total 40.02 last 66.39) - open/close 3255 (time 1780341587.75 total 50.04 last 70.60) - open/close 3961 (time 1780341597.75 total 60.04 last 70.56) total: 4500 open/close in 67.18 seconds: 66.98 ops/second - /unlink 1214 (time 1780341619.47 total 10.01 last 121.34) - /unlink 2415 (time 1780341629.47 total 20.01 last 120.08) - /unlink 3600 (time 1780341639.47 total 30.01 last 118.45) total: 4500 /unlink in 37.04 seconds: 121.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 716 (time 1780341660.91 total 10.00 last 71.58) - open/close 1485 (time 1780341670.92 total 20.02 last 76.80) - open/close 2260 (time 1780341680.93 total 30.02 last 77.45) - open/close 2919 (time 1780341690.94 total 40.03 last 65.85) - open/close 3586 (time 1780341700.95 total 50.04 last 66.61) - open/close 4288 (time 1780341710.96 total 60.05 last 70.13) total: 4500 open/close in 63.40 seconds: 70.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1364 (time 1780341727.96 total 10.00 last 136.37) - /unlink 2767 (time 1780341737.96 total 20.00 last 140.28) - /unlink 4394 (time 1780341747.96 total 30.00 last 162.68) total: 4500 /unlink in 30.73 seconds: 146.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 857 (time 1780341761.85 total 10.00 last 85.68) - open/close 1683 (time 1780341771.85 total 20.01 last 82.58) - open/close 2543 (time 1780341781.87 total 30.02 last 85.88) - open/close 3428 (time 1780341791.87 total 40.03 last 88.44) - open/close 4317 (time 1780341801.88 total 50.03 last 88.88) total: 4500 open/close in 52.18 seconds: 86.24 ops/second - /unlink 1700 (time 1780341817.23 total 10.00 last 169.92) - /unlink 3352 (time 1780341827.23 total 20.01 last 165.09) total: 4500 /unlink in 26.61 seconds: 169.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 841 (time 1780341846.78 total 10.00 last 84.09) - open/close 1737 (time 1780341856.79 total 20.02 last 89.47) - open/close 2722 (time 1780341866.80 total 30.02 last 98.48) - open/close 3659 (time 1780341876.80 total 40.02 last 93.65) total: 4500 open/close in 49.11 seconds: 91.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1732 (time 1780341899.20 total 10.00 last 173.11) - /unlink 3606 (time 1780341909.20 total 20.01 last 187.38) total: 4500 /unlink in 24.84 seconds: 181.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 909 (time 1780341927.04 total 10.01 last 90.83) - open/close 1956 (time 1780341937.04 total 20.01 last 104.68) - open/close 2964 (time 1780341947.04 total 30.01 last 100.77) - open/close 3957 (time 1780341957.05 total 40.02 last 99.26) total: 4500 open/close in 45.37 seconds: 99.17 ops/second - /unlink 1771 (time 1780341975.24 total 10.01 last 176.97) - /unlink 3626 (time 1780341985.25 total 20.02 last 185.28) total: 4500 /unlink in 24.90 seconds: 180.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1021 (time 1780342002.72 total 10.01 last 102.05) - open/close 1944 (time 1780342012.72 total 20.01 last 92.28) - open/close 2930 (time 1780342022.72 total 30.01 last 98.59) - open/close 3946 (time 1780342032.72 total 40.01 last 101.59) total: 4500 open/close in 45.42 seconds: 99.09 ops/second - /unlink 1848 (time 1780342050.70 total 10.00 last 184.75) - /unlink 3770 (time 1780342060.70 total 20.01 last 192.15) total: 4500 /unlink in 23.69 seconds: 189.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1017 (time 1780342077.29 total 10.00 last 101.69) - open/close 2023 (time 1780342087.30 total 20.01 last 100.49) - open/close 2972 (time 1780342097.30 total 30.02 last 94.86) - open/close 3916 (time 1780342107.31 total 40.02 last 94.37) total: 4500 open/close in 45.95 seconds: 97.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1726 (time 1780342126.28 total 10.00 last 172.58) - /unlink 3607 (time 1780342136.28 total 20.00 last 188.05) total: 4500 /unlink in 24.93 seconds: 180.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1052 (time 1780342154.13 total 10.00 last 105.19) - open/close 2049 (time 1780342164.13 total 20.01 last 99.63) - open/close 3101 (time 1780342174.14 total 30.01 last 105.15) - open/close 4073 (time 1780342184.14 total 40.02 last 97.13) total: 4500 open/close in 44.06 seconds: 102.13 ops/second - /unlink 1727 (time 1780342200.99 total 10.00 last 172.69) - /unlink 3408 (time 1780342210.99 total 20.00 last 168.05) total: 4500 /unlink in 25.53 seconds: 176.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 954 (time 1780342229.38 total 10.01 last 95.35) - open/close 1993 (time 1780342239.38 total 20.01 last 103.86) - open/close 2933 (time 1780342249.38 total 30.01 last 93.97) - open/close 3907 (time 1780342259.39 total 40.01 last 97.39) total: 4500 open/close in 46.51 seconds: 96.76 ops/second - /unlink 1879 (time 1780342278.97 total 10.00 last 187.87) - /unlink 3769 (time 1780342288.98 total 20.01 last 188.91) total: 4500 /unlink in 23.99 seconds: 187.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1039 (time 1780342305.82 total 10.02 last 103.74) - open/close 2007 (time 1780342315.82 total 20.02 last 96.77) - open/close 3058 (time 1780342325.83 total 30.03 last 104.96) - open/close 4088 (time 1780342335.83 total 40.03 last 102.99) total: 4500 open/close in 44.07 seconds: 102.10 ops/second - /unlink 2143 (time 1780342352.46 total 10.00 last 214.25) - /unlink 4302 (time 1780342362.46 total 20.01 last 215.81) total: 4500 /unlink in 21.02 seconds: 214.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 990 (time 1780342376.50 total 10.00 last 98.98) - open/close 1965 (time 1780342386.51 total 20.01 last 97.45) - open/close 2907 (time 1780342396.53 total 30.03 last 94.02) - open/close 3780 (time 1780342406.53 total 40.03 last 87.26) total: 4500 open/close in 48.00 seconds: 93.76 ops/second - /unlink 2000 (time 1780342427.26 total 10.00 last 199.98) - /unlink 4030 (time 1780342437.26 total 20.00 last 202.92) total: 4500 /unlink in 22.67 seconds: 198.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1056 (time 1780342452.70 total 10.01 last 105.53) - open/close 2092 (time 1780342462.71 total 20.01 last 103.53) - open/close 3065 (time 1780342472.71 total 30.02 last 97.22) - open/close 4011 (time 1780342482.73 total 40.04 last 94.42) total: 4500 open/close in 45.42 seconds: 99.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1606 (time 1780342501.18 total 10.00 last 160.53) - /unlink 3414 (time 1780342511.18 total 20.01 last 180.76) total: 4500 /unlink in 27.31 seconds: 164.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 821 (time 1780342531.84 total 10.01 last 82.04) - open/close 1685 (time 1780342541.85 total 20.02 last 86.28) - open/close 2690 (time 1780342551.86 total 30.03 last 100.44) - open/close 3630 (time 1780342561.87 total 40.04 last 93.88) total: 4500 open/close in 49.38 seconds: 91.12 ops/second - /unlink 1852 (time 1780342583.91 total 10.01 last 185.06) - /unlink 3404 (time 1780342593.92 total 20.01 last 155.13) total: 4500 /unlink in 26.86 seconds: 167.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 897 (time 1780342613.69 total 10.00 last 89.67) - open/close 1802 (time 1780342623.69 total 20.00 last 90.49) - open/close 2751 (time 1780342633.70 total 30.01 last 94.81) - open/close 3614 (time 1780342643.71 total 40.02 last 86.24) total: 4500 open/close in 49.57 seconds: 90.79 ops/second - /unlink 1916 (time 1780342665.98 total 10.00 last 191.59) - /unlink 3524 (time 1780342675.98 total 20.00 last 160.73) total: 4500 /unlink in 26.70 seconds: 168.52 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93937 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server unloading modules via unload_modules_local on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local oleg412-server: modules unloaded.