== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:40:39 (1788817239) Stopping clients: oleg239-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg239-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg239-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg239-server oleg239-server: oleg239-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: kmemleak disabled oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg239-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg239-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 663 (time 1788817416.21 total 10.00 last 66.29) - open/close 1413 (time 1788817426.21 total 20.01 last 74.97) - open/close 2212 (time 1788817436.24 total 30.04 last 79.63) - open/close 3153 (time 1788817446.27 total 40.07 last 93.81) - open/close 4030 (time 1788817456.30 total 50.09 last 87.50) total: 4500 open/close in 55.84 seconds: 80.59 ops/second - /unlink 1636 (time 1788817476.57 total 10.01 last 163.49) - /unlink 3035 (time 1788817486.57 total 20.01 last 139.87) - /unlink 4425 (time 1788817496.57 total 30.01 last 138.99) total: 4500 /unlink in 30.68 seconds: 146.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 898 (time 1788817511.39 total 10.00 last 89.77) - open/close 1759 (time 1788817521.39 total 20.01 last 86.08) - open/close 2547 (time 1788817531.39 total 30.01 last 78.79) - open/close 3465 (time 1788817541.39 total 40.01 last 91.79) - open/close 4306 (time 1788817551.41 total 50.03 last 83.92) total: 4500 open/close in 52.88 seconds: 85.10 ops/second - /unlink 1500 (time 1788817568.69 total 10.00 last 149.97) - /unlink 3169 (time 1788817578.70 total 20.01 last 166.82) total: 4500 /unlink in 29.29 seconds: 153.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 777 (time 1788817602.78 total 10.00 last 77.67) - open/close 1493 (time 1788817612.79 total 20.01 last 71.54) - open/close 2240 (time 1788817622.79 total 30.02 last 74.66) - open/close 3070 (time 1788817632.80 total 40.02 last 82.97) - open/close 3890 (time 1788817642.80 total 50.03 last 81.95) total: 4500 open/close in 55.90 seconds: 80.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1813 (time 1788817663.01 total 10.00 last 181.28) - /unlink 3315 (time 1788817673.01 total 20.00 last 150.18) total: 4500 /unlink in 29.52 seconds: 152.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 926 (time 1788817697.36 total 10.00 last 92.59) - open/close 1693 (time 1788817707.36 total 20.00 last 76.68) - open/close 2555 (time 1788817717.37 total 30.00 last 86.19) - open/close 3542 (time 1788817727.37 total 40.01 last 98.62) - open/close 4357 (time 1788817737.39 total 50.03 last 81.39) total: 4500 open/close in 51.75 seconds: 86.96 ops/second - /unlink 1219 (time 1788817754.05 total 10.00 last 121.87) - /unlink 3008 (time 1788817764.05 total 20.01 last 178.80) - /unlink 4422 (time 1788817774.06 total 30.01 last 141.34) total: 4500 /unlink in 30.83 seconds: 145.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 863 (time 1788817789.34 total 10.01 last 86.25) - open/close 1814 (time 1788817799.35 total 20.02 last 95.01) - open/close 2645 (time 1788817809.37 total 30.03 last 82.98) - open/close 3604 (time 1788817819.38 total 40.04 last 95.77) total: 4500 open/close in 49.55 seconds: 90.81 ops/second - /unlink 1492 (time 1788817843.15 total 10.01 last 149.11) - /unlink 2859 (time 1788817853.15 total 20.01 last 136.68) - /unlink 4183 (time 1788817863.17 total 30.02 last 132.21) total: 4500 /unlink in 32.30 seconds: 139.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 602 (time 1788817880.99 total 10.02 last 60.05) - open/close 1212 (time 1788817891.01 total 20.05 last 60.86) - open/close 1755 (time 1788817901.02 total 30.05 last 54.29) - open/close 2219 (time 1788817911.02 total 40.06 last 46.37) - open/close 2860 (time 1788817921.03 total 50.07 last 64.02) - open/close 3674 (time 1788817931.04 total 60.07 last 81.35) - open/close 4482 (time 1788817941.05 total 70.08 last 80.75) total: 4500 open/close in 70.24 seconds: 64.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1086 (time 1788817957.70 total 10.01 last 108.53) - /unlink 2648 (time 1788817967.70 total 20.01 last 156.20) - /unlink 4250 (time 1788817977.70 total 30.01 last 160.18) total: 4500 /unlink in 31.76 seconds: 141.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 802 (time 1788817993.84 total 10.02 last 80.06) - open/close 1537 (time 1788818003.85 total 20.02 last 73.45) - open/close 2342 (time 1788818013.89 total 30.06 last 80.18) - open/close 3229 (time 1788818023.89 total 40.07 last 88.65) - open/close 4024 (time 1788818033.91 total 50.09 last 79.35) total: 4500 open/close in 55.91 seconds: 80.48 ops/second - /unlink 1759 (time 1788818054.17 total 10.01 last 175.71) - /unlink 3188 (time 1788818064.18 total 20.02 last 142.81) total: 4500 /unlink in 27.69 seconds: 162.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 1008 (time 1788818086.80 total 10.01 last 100.70) - open/close 1851 (time 1788818096.82 total 20.03 last 84.14) - open/close 2728 (time 1788818106.84 total 30.05 last 87.54) - open/close 3626 (time 1788818116.84 total 40.05 last 89.79) - open/close 4475 (time 1788818126.85 total 50.06 last 84.83) total: 4500 open/close in 50.44 seconds: 89.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1604 (time 1788818141.79 total 10.00 last 160.38) - /unlink 3109 (time 1788818151.79 total 20.01 last 150.42) - /unlink 4448 (time 1788818161.80 total 30.01 last 133.86) total: 4500 /unlink in 30.58 seconds: 147.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 948 (time 1788818176.96 total 10.00 last 94.77) - open/close 1812 (time 1788818186.97 total 20.01 last 86.33) - open/close 2711 (time 1788818196.97 total 30.01 last 89.89) - open/close 3622 (time 1788818206.98 total 40.02 last 91.00) - open/close 4456 (time 1788818216.98 total 50.03 last 83.37) total: 4500 open/close in 50.50 seconds: 89.11 ops/second - /unlink 1536 (time 1788818231.38 total 10.00 last 153.55) - /unlink 2859 (time 1788818241.38 total 20.00 last 132.29) - /unlink 4266 (time 1788818251.39 total 30.02 last 140.52) total: 4500 /unlink in 32.06 seconds: 140.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 543 (time 1788818269.61 total 10.02 last 54.20) - open/close 1323 (time 1788818279.62 total 20.03 last 77.92) - open/close 2130 (time 1788818289.62 total 30.03 last 80.66) - open/close 2973 (time 1788818299.64 total 40.05 last 84.17) - open/close 3717 (time 1788818309.64 total 50.05 last 74.40) - open/close 4323 (time 1788818319.64 total 60.05 last 60.59) total: 4500 open/close in 61.86 seconds: 72.74 ops/second - /unlink 1133 (time 1788818336.83 total 10.01 last 113.18) - /unlink 2077 (time 1788818346.83 total 20.01 last 94.38) - /unlink 3704 (time 1788818356.83 total 30.02 last 162.64) total: 4500 /unlink in 34.83 seconds: 129.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 659 (time 1788818376.40 total 10.00 last 65.89) - open/close 1244 (time 1788818386.40 total 20.01 last 58.46) - open/close 1947 (time 1788818396.41 total 30.01 last 70.25) - open/close 2557 (time 1788818406.43 total 40.03 last 60.91) - open/close 3400 (time 1788818416.43 total 50.03 last 84.28) - open/close 4321 (time 1788818426.44 total 60.04 last 92.03) total: 4500 open/close in 61.96 seconds: 72.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1648 (time 1788818442.36 total 10.00 last 164.79) - /unlink 3340 (time 1788818452.36 total 20.00 last 169.17) total: 4500 /unlink in 27.10 seconds: 166.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 896 (time 1788818474.65 total 10.00 last 89.57) - open/close 1734 (time 1788818484.65 total 20.00 last 83.80) - open/close 2689 (time 1788818494.65 total 30.01 last 95.47) - open/close 3657 (time 1788818504.65 total 40.01 last 96.80) total: 4500 open/close in 49.01 seconds: 91.82 ops/second - /unlink 1703 (time 1788818528.03 total 10.00 last 170.27) - /unlink 3172 (time 1788818538.04 total 20.01 last 146.83) total: 4500 /unlink in 27.59 seconds: 163.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 766 (time 1788818559.71 total 10.00 last 76.56) - open/close 1563 (time 1788818569.75 total 20.05 last 79.36) - open/close 2141 (time 1788818579.78 total 30.08 last 57.62) - open/close 2874 (time 1788818589.78 total 40.08 last 73.30) - open/close 3607 (time 1788818599.78 total 50.08 last 73.30) - open/close 4114 (time 1788818609.89 total 60.19 last 50.14) total: 4500 open/close in 68.22 seconds: 65.97 ops/second - /unlink 994 (time 1788818636.26 total 10.00 last 99.36) - /unlink 2305 (time 1788818646.26 total 20.01 last 131.06) - /unlink 3989 (time 1788818656.27 total 30.01 last 168.29) total: 4500 /unlink in 32.85 seconds: 136.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 913 (time 1788818672.71 total 10.01 last 91.19) - open/close 1808 (time 1788818682.71 total 20.01 last 89.49) - open/close 2681 (time 1788818692.72 total 30.02 last 87.24) - open/close 3444 (time 1788818702.72 total 40.02 last 76.30) - open/close 4417 (time 1788818712.73 total 50.03 last 97.23) total: 4500 open/close in 50.90 seconds: 88.42 ops/second - /unlink 1826 (time 1788818727.92 total 10.00 last 182.58) - /unlink 3409 (time 1788818737.92 total 20.00 last 158.26) total: 4500 /unlink in 26.79 seconds: 167.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 759 (time 1788818759.95 total 10.01 last 75.85) - open/close 1573 (time 1788818769.97 total 20.02 last 81.29) - open/close 2469 (time 1788818779.97 total 30.03 last 89.54) - open/close 3407 (time 1788818789.98 total 40.03 last 93.75) - open/close 4282 (time 1788818800.00 total 50.05 last 87.34) total: 4500 open/close in 53.11 seconds: 84.73 ops/second - /unlink 1600 (time 1788818817.63 total 10.01 last 159.82) - /unlink 3389 (time 1788818827.63 total 20.01 last 178.87) total: 4500 /unlink in 26.28 seconds: 171.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 859 (time 1788818849.27 total 10.00 last 85.89) - open/close 1746 (time 1788818859.27 total 20.00 last 88.67) - open/close 2631 (time 1788818869.28 total 30.01 last 88.44) - open/close 3660 (time 1788818879.28 total 40.01 last 102.86) total: 4500 open/close in 49.85 seconds: 90.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1475 (time 1788818903.30 total 10.01 last 147.42) - /unlink 2986 (time 1788818913.30 total 20.01 last 151.05) - /unlink 4494 (time 1788818923.31 total 30.02 last 150.69) total: 4500 /unlink in 30.06 seconds: 149.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 750 (time 1788818939.64 total 10.01 last 74.96) - open/close 1458 (time 1788818949.64 total 20.01 last 70.80) - open/close 2379 (time 1788818959.65 total 30.02 last 91.99) - open/close 3015 (time 1788818969.65 total 40.02 last 63.58) - open/close 3726 (time 1788818979.66 total 50.03 last 71.05) total: 4500 open/close in 58.44 seconds: 77.01 ops/second - /unlink 1608 (time 1788819002.23 total 10.01 last 160.67) - /unlink 3244 (time 1788819012.24 total 20.01 last 163.55) total: 4500 /unlink in 27.01 seconds: 166.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 935 (time 1788819032.99 total 10.00 last 93.47) - open/close 1939 (time 1788819043.00 total 20.01 last 100.28) - open/close 2731 (time 1788819053.02 total 30.04 last 79.03) - open/close 3660 (time 1788819063.03 total 40.04 last 92.84) - open/close 4490 (time 1788819073.03 total 50.05 last 82.98) total: 4500 open/close in 50.17 seconds: 89.69 ops/second - /unlink 1631 (time 1788819087.47 total 10.00 last 163.03) - /unlink 2719 (time 1788819097.48 total 20.01 last 108.73) - /unlink 4409 (time 1788819107.48 total 30.01 last 168.97) total: 4500 /unlink in 30.53 seconds: 147.39 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 94961 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local oleg239-server: kmemleak disabled modules unloaded.