== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:39:28 (1789054768) Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg450-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg450-server oleg450-server: oleg450-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: kmemleak disabled oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg450-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg450-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 508 (time 1789055008.94 total 10.01 last 50.72) - open/close 1085 (time 1789055018.97 total 20.04 last 57.54) - open/close 1683 (time 1789055028.99 total 30.06 last 59.67) - open/close 2277 (time 1789055039.00 total 40.07 last 59.34) - open/close 2788 (time 1789055049.02 total 50.09 last 51.00) - open/close 3247 (time 1789055059.02 total 60.10 last 45.88) - open/close 3744 (time 1789055069.03 total 70.10 last 49.67) - open/close 4284 (time 1789055079.03 total 80.11 last 54.00) total: 4500 open/close in 83.80 seconds: 53.70 ops/second - /unlink 1034 (time 1789055099.17 total 10.02 last 103.24) - /unlink 1978 (time 1789055109.17 total 20.02 last 94.39) - /unlink 2891 (time 1789055119.17 total 30.02 last 91.30) - /unlink 4058 (time 1789055129.17 total 40.02 last 116.69) total: 4500 /unlink in 43.72 seconds: 102.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 497 (time 1789055148.47 total 10.01 last 49.66) - open/close 959 (time 1789055158.47 total 20.01 last 46.18) - open/close 1474 (time 1789055168.50 total 30.04 last 51.35) - open/close 1970 (time 1789055178.51 total 40.05 last 49.55) - open/close 2515 (time 1789055188.51 total 50.05 last 54.49) - open/close 3106 (time 1789055198.51 total 60.05 last 59.10) - open/close 3754 (time 1789055208.52 total 70.06 last 64.77) - open/close 4471 (time 1789055218.52 total 80.06 last 71.65) total: 4500 open/close in 80.50 seconds: 55.90 ops/second - /unlink 1136 (time 1789055233.68 total 10.01 last 113.50) - /unlink 2267 (time 1789055243.70 total 20.02 last 112.95) - /unlink 3381 (time 1789055253.70 total 30.03 last 111.33) total: 4500 /unlink in 39.31 seconds: 114.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 538 (time 1789055279.57 total 10.00 last 53.79) - open/close 1175 (time 1789055289.58 total 20.01 last 63.62) - open/close 1753 (time 1789055299.59 total 30.02 last 57.76) - open/close 2334 (time 1789055309.60 total 40.03 last 58.03) - open/close 2973 (time 1789055319.61 total 50.04 last 63.88) - open/close 3599 (time 1789055329.61 total 60.04 last 62.60) - open/close 4267 (time 1789055339.61 total 70.04 last 66.80) total: 4500 open/close in 73.54 seconds: 61.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1258 (time 1789055357.97 total 10.00 last 125.77) - /unlink 2445 (time 1789055367.98 total 20.01 last 118.65) - /unlink 3709 (time 1789055377.98 total 30.01 last 126.35) total: 4500 /unlink in 36.84 seconds: 122.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 615 (time 1789055399.19 total 10.00 last 61.48) - open/close 1225 (time 1789055409.20 total 20.01 last 60.95) - open/close 1805 (time 1789055419.22 total 30.03 last 57.90) - open/close 2340 (time 1789055429.22 total 40.03 last 53.50) - open/close 2966 (time 1789055439.24 total 50.06 last 62.42) - open/close 3608 (time 1789055449.26 total 60.07 last 64.12) - open/close 4173 (time 1789055459.28 total 70.10 last 56.36) total: 4500 open/close in 76.76 seconds: 58.63 ops/second - /unlink 1126 (time 1789055481.85 total 10.00 last 112.59) - /unlink 2183 (time 1789055491.86 total 20.01 last 105.59) - /unlink 3276 (time 1789055501.86 total 30.02 last 109.25) - /unlink 4292 (time 1789055511.86 total 40.02 last 101.60) total: 4500 /unlink in 41.89 seconds: 107.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 436 (time 1789055529.31 total 10.02 last 43.53) - open/close 1017 (time 1789055539.35 total 20.06 last 57.85) - open/close 1515 (time 1789055549.37 total 30.08 last 49.71) - open/close 2126 (time 1789055559.40 total 40.11 last 60.92) - open/close 2784 (time 1789055569.40 total 50.11 last 65.80) - open/close 3456 (time 1789055579.41 total 60.11 last 67.14) - open/close 4135 (time 1789055589.42 total 70.12 last 67.85) total: 4500 open/close in 76.89 seconds: 58.53 ops/second - /unlink 1108 (time 1789055613.08 total 10.00 last 110.79) - /unlink 2220 (time 1789055623.09 total 20.00 last 111.18) - /unlink 3386 (time 1789055633.09 total 30.01 last 116.53) - /unlink 4441 (time 1789055643.12 total 40.04 last 105.18) total: 4500 /unlink in 40.83 seconds: 110.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 669 (time 1789055659.91 total 10.01 last 66.82) - open/close 1269 (time 1789055669.92 total 20.02 last 59.95) - open/close 1886 (time 1789055679.94 total 30.03 last 61.61) - open/close 2543 (time 1789055689.94 total 40.04 last 65.69) - open/close 3199 (time 1789055699.94 total 50.04 last 65.55) - open/close 3799 (time 1789055709.96 total 60.06 last 59.90) - open/close 4365 (time 1789055719.98 total 70.07 last 56.52) total: 4500 open/close in 72.94 seconds: 61.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1148 (time 1789055739.02 total 10.00 last 114.76) - /unlink 2248 (time 1789055749.02 total 20.00 last 109.98) - /unlink 3354 (time 1789055759.03 total 30.01 last 110.58) total: 4500 /unlink in 39.65 seconds: 113.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 594 (time 1789055784.66 total 10.01 last 59.33) - open/close 1100 (time 1789055794.69 total 20.04 last 50.44) - open/close 1636 (time 1789055804.70 total 30.05 last 53.57) - open/close 2099 (time 1789055814.73 total 40.08 last 46.17) - open/close 2705 (time 1789055824.73 total 50.08 last 60.60) - open/close 3277 (time 1789055834.74 total 60.09 last 57.15) - open/close 3843 (time 1789055844.74 total 70.09 last 56.60) - open/close 4452 (time 1789055854.74 total 80.09 last 60.87) total: 4500 open/close in 80.61 seconds: 55.83 ops/second - /unlink 1190 (time 1789055869.53 total 10.01 last 118.89) - /unlink 2321 (time 1789055879.53 total 20.01 last 113.05) - /unlink 3505 (time 1789055889.53 total 30.01 last 118.39) total: 4500 /unlink in 38.12 seconds: 118.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 657 (time 1789055912.58 total 10.00 last 65.69) - open/close 1299 (time 1789055922.61 total 20.04 last 63.97) - open/close 1927 (time 1789055932.67 total 30.10 last 62.42) - open/close 2380 (time 1789055942.70 total 40.12 last 45.18) - open/close 2872 (time 1789055952.71 total 50.14 last 49.14) - open/close 3363 (time 1789055962.72 total 60.15 last 49.05) - open/close 3921 (time 1789055972.73 total 70.16 last 55.75) total: 4500 open/close in 78.76 seconds: 57.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1302 (time 1789055996.14 total 10.01 last 130.01) - /unlink 2457 (time 1789056006.15 total 20.03 last 115.36) - /unlink 3640 (time 1789056016.15 total 30.03 last 118.24) total: 4500 /unlink in 37.80 seconds: 119.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 537 (time 1789056038.80 total 10.01 last 53.67) - open/close 1142 (time 1789056048.80 total 20.01 last 60.48) - open/close 1725 (time 1789056058.81 total 30.01 last 58.27) - open/close 2260 (time 1789056068.82 total 40.03 last 53.44) - open/close 2791 (time 1789056078.83 total 50.04 last 53.02) - open/close 3334 (time 1789056088.85 total 60.06 last 54.22) - open/close 3870 (time 1789056098.85 total 70.06 last 53.57) - open/close 4458 (time 1789056108.87 total 80.08 last 58.72) total: 4500 open/close in 80.93 seconds: 55.61 ops/second - /unlink 1223 (time 1789056125.21 total 10.00 last 122.28) - /unlink 2402 (time 1789056135.21 total 20.01 last 117.84) - /unlink 3519 (time 1789056145.22 total 30.02 last 111.58) total: 4500 /unlink in 38.82 seconds: 115.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 597 (time 1789056169.53 total 10.01 last 59.62) - open/close 1216 (time 1789056179.56 total 20.04 last 61.75) - open/close 1762 (time 1789056189.58 total 30.06 last 54.49) - open/close 2357 (time 1789056199.58 total 40.06 last 59.49) - open/close 2883 (time 1789056209.60 total 50.08 last 52.49) - open/close 3356 (time 1789056219.63 total 60.11 last 47.16) - open/close 4002 (time 1789056229.65 total 70.13 last 64.49) total: 4500 open/close in 79.29 seconds: 56.76 ops/second - /unlink 1113 (time 1789056254.64 total 10.01 last 111.22) - /unlink 1946 (time 1789056264.65 total 20.01 last 83.24) - /unlink 2864 (time 1789056274.65 total 30.01 last 91.80) - /unlink 4008 (time 1789056284.65 total 40.02 last 114.34) total: 4500 /unlink in 44.25 seconds: 101.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 633 (time 1789056303.91 total 10.00 last 63.27) - open/close 1212 (time 1789056313.92 total 20.01 last 57.85) - open/close 1698 (time 1789056323.93 total 30.02 last 48.58) - open/close 2300 (time 1789056333.94 total 40.03 last 60.14) - open/close 2841 (time 1789056343.94 total 50.03 last 54.10) - open/close 3423 (time 1789056353.94 total 60.03 last 58.20) - open/close 3978 (time 1789056363.96 total 70.05 last 55.39) total: 4500 open/close in 79.72 seconds: 56.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1017 (time 1789056389.55 total 10.01 last 101.59) - /unlink 1959 (time 1789056399.55 total 20.01 last 94.19) - /unlink 3109 (time 1789056409.55 total 30.01 last 114.97) - /unlink 4266 (time 1789056419.56 total 40.02 last 115.60) total: 4500 /unlink in 41.81 seconds: 107.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 639 (time 1789056436.49 total 10.00 last 63.89) - open/close 1256 (time 1789056446.49 total 20.01 last 61.66) - open/close 1930 (time 1789056456.51 total 30.03 last 67.27) - open/close 2542 (time 1789056466.52 total 40.03 last 61.16) - open/close 3210 (time 1789056476.52 total 50.04 last 66.78) - open/close 3824 (time 1789056486.54 total 60.05 last 61.32) - open/close 4463 (time 1789056496.55 total 70.07 last 63.79) total: 4500 open/close in 70.78 seconds: 63.57 ops/second - /unlink 1099 (time 1789056512.71 total 10.01 last 109.83) - /unlink 2128 (time 1789056522.71 total 20.01 last 102.87) - /unlink 3330 (time 1789056532.71 total 30.02 last 120.13) total: 4500 /unlink in 39.24 seconds: 114.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 657 (time 1789056557.67 total 10.01 last 65.61) - open/close 1331 (time 1789056567.67 total 20.02 last 67.39) - open/close 1903 (time 1789056577.72 total 30.07 last 56.91) - open/close 2457 (time 1789056587.73 total 40.08 last 55.35) - open/close 2912 (time 1789056597.76 total 50.10 last 45.40) - open/close 3380 (time 1789056607.76 total 60.10 last 46.80) - open/close 3873 (time 1789056617.76 total 70.10 last 49.28) - open/close 4408 (time 1789056627.76 total 80.10 last 53.48) total: 4500 open/close in 81.66 seconds: 55.11 ops/second - /unlink 993 (time 1789056646.00 total 10.00 last 99.29) - /unlink 2061 (time 1789056656.01 total 20.01 last 106.70) - /unlink 2910 (time 1789056666.01 total 30.02 last 84.86) - /unlink 3683 (time 1789056676.03 total 40.03 last 77.20) total: 4500 /unlink in 49.79 seconds: 90.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 576 (time 1789056702.64 total 10.01 last 57.55) - open/close 1172 (time 1789056712.70 total 20.07 last 59.25) - open/close 1800 (time 1789056722.70 total 30.08 last 62.75) - open/close 2392 (time 1789056732.71 total 40.08 last 59.16) - open/close 3023 (time 1789056742.73 total 50.10 last 63.01) - open/close 3619 (time 1789056752.73 total 60.10 last 59.58) - open/close 4327 (time 1789056762.73 total 70.10 last 70.79) total: 4500 open/close in 72.90 seconds: 61.72 ops/second - /unlink 1043 (time 1789056781.22 total 10.00 last 104.29) - /unlink 2348 (time 1789056791.23 total 20.01 last 130.37) - /unlink 3692 (time 1789056801.24 total 30.02 last 134.32) total: 4500 /unlink in 37.00 seconds: 121.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 603 (time 1789056824.19 total 10.00 last 60.29) - open/close 1296 (time 1789056834.20 total 20.01 last 69.27) - open/close 1973 (time 1789056844.22 total 30.03 last 67.55) - open/close 2665 (time 1789056854.22 total 40.03 last 69.19) - open/close 3322 (time 1789056864.23 total 50.04 last 65.65) - open/close 4019 (time 1789056874.24 total 60.04 last 69.65) total: 4500 open/close in 66.02 seconds: 68.16 ops/second - /unlink 1290 (time 1789056895.86 total 10.01 last 128.92) - /unlink 2374 (time 1789056905.87 total 20.01 last 108.32) - /unlink 3503 (time 1789056915.88 total 30.02 last 112.78) total: 4500 /unlink in 38.39 seconds: 117.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 584 (time 1789056939.89 total 10.01 last 58.34) - open/close 1098 (time 1789056949.90 total 20.02 last 51.35) - open/close 1640 (time 1789056959.90 total 30.03 last 54.17) - open/close 2259 (time 1789056969.91 total 40.03 last 61.87) - open/close 2911 (time 1789056979.94 total 50.07 last 64.95) - open/close 3598 (time 1789056989.95 total 60.07 last 68.70) - open/close 4151 (time 1789056999.95 total 70.07 last 55.28) total: 4500 open/close in 75.50 seconds: 59.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1202 (time 1789057020.75 total 10.00 last 120.16) - /unlink 2400 (time 1789057030.75 total 20.00 last 119.79) - /unlink 3620 (time 1789057040.75 total 30.01 last 121.98) total: 4500 /unlink in 37.53 seconds: 119.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 494 (time 1789057063.56 total 10.00 last 49.39) - open/close 1208 (time 1789057073.56 total 20.00 last 71.39) - open/close 1812 (time 1789057083.57 total 30.01 last 60.37) - open/close 2408 (time 1789057093.58 total 40.03 last 59.51) - open/close 3009 (time 1789057103.59 total 50.04 last 60.04) - open/close 3486 (time 1789057113.60 total 60.04 last 47.68) - open/close 4027 (time 1789057123.60 total 70.04 last 54.09) total: 4500 open/close in 78.55 seconds: 57.29 ops/second - /unlink 1092 (time 1789057148.19 total 10.01 last 109.05) - /unlink 2129 (time 1789057158.20 total 20.02 last 103.66) - /unlink 3109 (time 1789057168.20 total 30.02 last 97.96) - /unlink 4179 (time 1789057178.20 total 40.02 last 107.00) total: 4500 /unlink in 43.45 seconds: 103.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 558 (time 1789057198.25 total 10.00 last 55.80) - open/close 1125 (time 1789057208.27 total 20.02 last 56.58) - open/close 1527 (time 1789057218.28 total 30.03 last 40.16) - open/close 2126 (time 1789057228.31 total 40.06 last 59.71) - open/close 2686 (time 1789057238.31 total 50.07 last 55.96) - open/close 3219 (time 1789057248.32 total 60.08 last 53.26) - open/close 3864 (time 1789057258.33 total 70.09 last 64.44) - open/close 4407 (time 1789057268.33 total 80.09 last 54.29) total: 4500 open/close in 81.61 seconds: 55.14 ops/second - /unlink 1036 (time 1789057285.62 total 10.01 last 103.50) - /unlink 2232 (time 1789057295.62 total 20.01 last 119.60) - /unlink 3463 (time 1789057305.63 total 30.02 last 123.01) - /unlink 4497 (time 1789057315.63 total 40.02 last 103.34) total: 4500 /unlink in 40.03 seconds: 112.40 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95352 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server unloading modules via unload_modules_local on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local oleg450-server: kmemleak disabled modules unloaded.