== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:29:59 (1788481799) Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg412-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg412-server oleg412-server: oleg412-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: kmemleak disabled oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: detected 4 online CPUs by sysfs oleg412-server: MODOPTS_LIBCFS= oleg412-server: Force libcfs to create 2 CPU partitions oleg412-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg412-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg412-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg412-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 654 (time 1788482016.29 total 10.01 last 65.35) - open/close 1269 (time 1788482026.30 total 20.02 last 61.41) - open/close 1921 (time 1788482036.32 total 30.04 last 65.11) - open/close 2606 (time 1788482046.33 total 40.05 last 68.44) - open/close 3282 (time 1788482056.33 total 50.05 last 67.59) - open/close 3917 (time 1788482066.34 total 60.06 last 63.40) total: 4500 open/close in 69.95 seconds: 64.33 ops/second - /unlink 1299 (time 1788482090.41 total 10.02 last 129.60) - /unlink 2527 (time 1788482100.42 total 20.03 last 122.77) - /unlink 3716 (time 1788482110.42 total 30.03 last 118.88) total: 4500 /unlink in 36.83 seconds: 122.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 662 (time 1788482131.28 total 10.01 last 66.12) - open/close 1274 (time 1788482141.29 total 20.02 last 61.17) - open/close 1943 (time 1788482151.29 total 30.02 last 66.87) - open/close 2614 (time 1788482161.29 total 40.02 last 67.09) - open/close 3239 (time 1788482171.30 total 50.03 last 62.48) - open/close 3872 (time 1788482181.30 total 60.03 last 63.30) total: 4500 open/close in 69.49 seconds: 64.76 ops/second - /unlink 1196 (time 1788482205.23 total 10.01 last 119.51) - /unlink 2399 (time 1788482215.24 total 20.01 last 120.24) - /unlink 3474 (time 1788482225.24 total 30.02 last 107.46) total: 4500 /unlink in 37.98 seconds: 118.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 670 (time 1788482247.90 total 10.01 last 66.96) - open/close 1239 (time 1788482257.90 total 20.01 last 56.86) - open/close 1908 (time 1788482267.91 total 30.02 last 66.87) - open/close 2470 (time 1788482277.91 total 40.02 last 56.18) - open/close 3017 (time 1788482287.92 total 50.03 last 54.63) - open/close 3618 (time 1788482297.94 total 60.05 last 60.02) - open/close 4312 (time 1788482307.94 total 70.05 last 69.36) total: 4500 open/close in 73.13 seconds: 61.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1022 (time 1788482326.96 total 10.01 last 102.12) - /unlink 2267 (time 1788482336.96 total 20.01 last 124.42) - /unlink 3413 (time 1788482346.97 total 30.02 last 114.54) total: 4500 /unlink in 39.51 seconds: 113.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 662 (time 1788482372.12 total 10.01 last 66.15) - open/close 1308 (time 1788482382.14 total 20.02 last 64.50) - open/close 1967 (time 1788482392.15 total 30.03 last 65.83) - open/close 2665 (time 1788482402.15 total 40.04 last 69.76) - open/close 3323 (time 1788482412.16 total 50.04 last 65.77) - open/close 3971 (time 1788482422.17 total 60.05 last 64.76) total: 4500 open/close in 68.46 seconds: 65.73 ops/second - /unlink 1228 (time 1788482446.00 total 10.00 last 122.77) - /unlink 2389 (time 1788482456.00 total 20.01 last 116.06) - /unlink 3548 (time 1788482466.01 total 30.01 last 115.85) total: 4500 /unlink in 38.10 seconds: 118.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 569 (time 1788482489.62 total 10.01 last 56.82) - open/close 1128 (time 1788482499.64 total 20.03 last 55.84) - open/close 1695 (time 1788482509.64 total 30.03 last 56.66) - open/close 2309 (time 1788482519.65 total 40.04 last 61.35) - open/close 2851 (time 1788482529.66 total 50.05 last 54.16) - open/close 3408 (time 1788482539.67 total 60.06 last 55.61) - open/close 4042 (time 1788482549.69 total 70.07 last 63.33) total: 4500 open/close in 77.41 seconds: 58.13 ops/second - /unlink 1210 (time 1788482572.64 total 10.00 last 120.96) - /unlink 2336 (time 1788482582.65 total 20.01 last 112.49) - /unlink 3615 (time 1788482592.65 total 30.02 last 127.86) total: 4500 /unlink in 38.88 seconds: 115.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 571 (time 1788482616.97 total 10.00 last 57.09) - open/close 1140 (time 1788482626.98 total 20.01 last 56.87) - open/close 1712 (time 1788482636.98 total 30.01 last 57.19) - open/close 2217 (time 1788482647.02 total 40.05 last 50.28) - open/close 2760 (time 1788482657.03 total 50.06 last 54.28) - open/close 3316 (time 1788482667.04 total 60.06 last 55.55) - open/close 3901 (time 1788482677.04 total 70.07 last 58.48) - open/close 4463 (time 1788482687.05 total 80.07 last 56.16) total: 4500 open/close in 80.68 seconds: 55.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1153 (time 1788482703.35 total 10.01 last 115.21) - /unlink 2209 (time 1788482713.36 total 20.02 last 105.47) - /unlink 3355 (time 1788482723.37 total 30.03 last 114.54) total: 4500 /unlink in 39.70 seconds: 113.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 650 (time 1788482747.52 total 10.01 last 64.94) - open/close 1284 (time 1788482757.54 total 20.02 last 63.32) - open/close 1879 (time 1788482767.54 total 30.02 last 59.48) - open/close 2463 (time 1788482777.55 total 40.03 last 58.34) - open/close 3083 (time 1788482787.55 total 50.04 last 61.99) - open/close 3725 (time 1788482797.56 total 60.05 last 64.14) - open/close 4433 (time 1788482807.57 total 70.06 last 70.69) total: 4500 open/close in 70.91 seconds: 63.46 ops/second - /unlink 1247 (time 1788482823.05 total 10.00 last 124.66) - /unlink 2374 (time 1788482833.07 total 20.02 last 112.48) - /unlink 3367 (time 1788482843.07 total 30.02 last 99.28) total: 4500 /unlink in 39.68 seconds: 113.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 621 (time 1788482868.61 total 10.01 last 62.02) - open/close 1180 (time 1788482878.62 total 20.02 last 55.86) - open/close 1833 (time 1788482888.63 total 30.04 last 65.21) - open/close 2511 (time 1788482898.63 total 40.04 last 67.79) - open/close 3226 (time 1788482908.64 total 50.04 last 71.46) - open/close 3844 (time 1788482918.64 total 60.05 last 61.77) total: 4500 open/close in 69.85 seconds: 64.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1268 (time 1788482943.24 total 10.00 last 126.74) - /unlink 2350 (time 1788482953.24 total 20.01 last 108.19) - /unlink 3559 (time 1788482963.25 total 30.02 last 120.75) total: 4500 /unlink in 38.96 seconds: 115.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 660 (time 1788482987.97 total 10.01 last 65.96) - open/close 1339 (time 1788482997.97 total 20.01 last 67.86) - open/close 2015 (time 1788483007.98 total 30.02 last 67.58) - open/close 2689 (time 1788483017.98 total 40.02 last 67.39) - open/close 3299 (time 1788483027.99 total 50.03 last 60.92) - open/close 3877 (time 1788483038.00 total 60.04 last 57.74) - open/close 4472 (time 1788483048.01 total 70.05 last 59.44) total: 4500 open/close in 70.40 seconds: 63.92 ops/second - /unlink 1254 (time 1788483063.48 total 10.01 last 125.32) - /unlink 2587 (time 1788483073.49 total 20.01 last 133.22) - /unlink 3784 (time 1788483083.49 total 30.02 last 119.65) total: 4500 /unlink in 36.31 seconds: 123.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 656 (time 1788483104.31 total 10.01 last 65.52) - open/close 1345 (time 1788483114.32 total 20.02 last 68.86) - open/close 2066 (time 1788483124.32 total 30.02 last 72.08) - open/close 2712 (time 1788483134.34 total 40.04 last 64.45) - open/close 3316 (time 1788483144.35 total 50.05 last 60.35) - open/close 3968 (time 1788483154.36 total 60.06 last 65.16) total: 4500 open/close in 69.87 seconds: 64.41 ops/second - /unlink 1176 (time 1788483179.74 total 10.00 last 117.55) - /unlink 2369 (time 1788483189.74 total 20.01 last 119.25) - /unlink 3684 (time 1788483199.74 total 30.01 last 131.48) total: 4500 /unlink in 36.12 seconds: 124.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 586 (time 1788483221.52 total 10.01 last 58.55) - open/close 1190 (time 1788483231.52 total 20.01 last 60.39) - open/close 1853 (time 1788483241.52 total 30.01 last 66.29) - open/close 2532 (time 1788483251.52 total 40.01 last 67.89) - open/close 3199 (time 1788483261.52 total 50.01 last 66.70) - open/close 3892 (time 1788483271.53 total 60.02 last 69.25) total: 4500 open/close in 67.95 seconds: 66.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1156 (time 1788483293.74 total 10.00 last 115.60) - /unlink 2232 (time 1788483303.77 total 20.03 last 107.31) - /unlink 3194 (time 1788483313.79 total 30.05 last 96.01) - /unlink 4157 (time 1788483323.79 total 40.05 last 96.28) total: 4500 /unlink in 45.09 seconds: 99.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 478 (time 1788483346.40 total 10.00 last 47.78) - open/close 1050 (time 1788483356.43 total 20.03 last 57.03) - open/close 1639 (time 1788483366.46 total 30.06 last 58.72) - open/close 2241 (time 1788483376.47 total 40.07 last 60.14) - open/close 2833 (time 1788483386.47 total 50.08 last 59.18) - open/close 3485 (time 1788483396.50 total 60.10 last 65.02) - open/close 4174 (time 1788483406.51 total 70.11 last 68.85) total: 4500 open/close in 75.12 seconds: 59.91 ops/second - /unlink 1155 (time 1788483426.68 total 10.01 last 115.41) - /unlink 2417 (time 1788483436.68 total 20.01 last 126.20) - /unlink 3628 (time 1788483446.68 total 30.01 last 121.10) total: 4500 /unlink in 37.18 seconds: 121.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 651 (time 1788483468.82 total 10.01 last 65.03) - open/close 1233 (time 1788483478.83 total 20.02 last 58.15) - open/close 1889 (time 1788483488.83 total 30.02 last 65.59) - open/close 2491 (time 1788483498.84 total 40.02 last 60.18) - open/close 3059 (time 1788483508.84 total 50.03 last 56.77) - open/close 3707 (time 1788483518.86 total 60.05 last 64.69) - open/close 4337 (time 1788483528.87 total 70.06 last 62.89) total: 4500 open/close in 73.06 seconds: 61.60 ops/second - /unlink 1183 (time 1788483547.10 total 10.01 last 118.14) - /unlink 2361 (time 1788483557.11 total 20.02 last 117.77) - /unlink 3560 (time 1788483567.11 total 30.02 last 119.86) total: 4500 /unlink in 37.36 seconds: 120.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 604 (time 1788483590.14 total 10.03 last 60.24) - open/close 1199 (time 1788483600.14 total 20.03 last 59.49) - open/close 1868 (time 1788483610.14 total 30.03 last 66.89) - open/close 2513 (time 1788483620.16 total 40.05 last 64.40) - open/close 3125 (time 1788483630.16 total 50.05 last 61.19) - open/close 3755 (time 1788483640.17 total 60.06 last 62.95) - open/close 4365 (time 1788483650.18 total 70.08 last 60.89) total: 4500 open/close in 72.65 seconds: 61.94 ops/second - /unlink 1154 (time 1788483668.08 total 10.00 last 115.36) - /unlink 2386 (time 1788483678.09 total 20.01 last 123.17) - /unlink 3482 (time 1788483688.09 total 30.01 last 109.55) total: 4500 /unlink in 38.06 seconds: 118.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 605 (time 1788483712.04 total 10.01 last 60.45) - open/close 1217 (time 1788483722.05 total 20.02 last 61.14) - open/close 1853 (time 1788483732.06 total 30.03 last 63.53) - open/close 2511 (time 1788483742.07 total 40.04 last 65.76) - open/close 3138 (time 1788483752.07 total 50.04 last 62.69) - open/close 3718 (time 1788483762.07 total 60.04 last 57.99) - open/close 4144 (time 1788483772.08 total 70.04 last 42.58) total: 4500 open/close in 76.91 seconds: 58.51 ops/second - /unlink 862 (time 1788483795.77 total 10.00 last 86.16) - /unlink 1754 (time 1788483805.78 total 20.01 last 89.17) - /unlink 2585 (time 1788483815.78 total 30.01 last 83.10) - /unlink 3298 (time 1788483825.78 total 40.01 last 71.29) - /unlink 3900 (time 1788483835.79 total 50.02 last 60.12) total: 4500 /unlink in 56.32 seconds: 79.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 484 (time 1788483859.41 total 10.00 last 48.38) - open/close 1018 (time 1788483869.42 total 20.01 last 53.35) - open/close 1541 (time 1788483879.44 total 30.03 last 52.20) - open/close 2095 (time 1788483889.45 total 40.04 last 55.34) - open/close 2636 (time 1788483899.45 total 50.05 last 54.08) - open/close 3273 (time 1788483909.46 total 60.05 last 63.64) - open/close 3914 (time 1788483919.48 total 70.08 last 63.96) total: 4500 open/close in 78.72 seconds: 57.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1195 (time 1788483942.86 total 10.00 last 119.45) - /unlink 2335 (time 1788483952.87 total 20.01 last 113.94) - /unlink 3449 (time 1788483962.88 total 30.02 last 111.25) - /unlink 4474 (time 1788483972.88 total 40.02 last 102.48) total: 4500 /unlink in 40.37 seconds: 111.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 497 (time 1788483988.86 total 10.01 last 49.65) - open/close 987 (time 1788483998.87 total 20.02 last 48.94) - open/close 1542 (time 1788484008.88 total 30.04 last 55.41) - open/close 2109 (time 1788484018.90 total 40.05 last 56.64) - open/close 2832 (time 1788484028.92 total 50.07 last 72.14) - open/close 3400 (time 1788484038.92 total 60.08 last 56.77) - open/close 3992 (time 1788484048.93 total 70.09 last 59.15) total: 4500 open/close in 78.78 seconds: 57.12 ops/second - /unlink 1100 (time 1788484072.84 total 10.03 last 109.72) - /unlink 2066 (time 1788484082.85 total 20.04 last 96.48) - /unlink 3255 (time 1788484092.86 total 30.04 last 118.85) - /unlink 4379 (time 1788484102.87 total 40.05 last 112.30) total: 4500 /unlink in 41.36 seconds: 108.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 655 (time 1788484120.05 total 10.01 last 65.40) - open/close 1348 (time 1788484130.06 total 20.03 last 69.19) - open/close 1951 (time 1788484140.09 total 30.05 last 60.15) - open/close 2482 (time 1788484150.11 total 40.07 last 53.00) - open/close 2929 (time 1788484160.13 total 50.10 last 44.58) - open/close 3427 (time 1788484170.14 total 60.10 last 49.78) - open/close 3946 (time 1788484180.14 total 70.11 last 51.89) - open/close 4417 (time 1788484190.16 total 80.13 last 47.01) total: 4500 open/close in 81.69 seconds: 55.08 ops/second - /unlink 1152 (time 1788484207.72 total 10.01 last 115.11) - /unlink 2216 (time 1788484217.72 total 20.01 last 106.38) - /unlink 3448 (time 1788484227.72 total 30.02 last 123.13) total: 4500 /unlink in 37.98 seconds: 118.49 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95378 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server unloading modules via unload_modules_local on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local oleg412-server: kmemleak disabled modules unloaded.