== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:47:35 (1789069655) Stopping clients: oleg641-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg641-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg641-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg641-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg641-server oleg641-server: oleg641-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing load_modules_local oleg641-server: kmemleak disabled oleg641-server: Loading modules from /home/green/git/lustre-release/lustre oleg641-server: detected 4 online CPUs by sysfs oleg641-server: MODOPTS_LIBCFS= oleg641-server: Force libcfs to create 2 CPU partitions oleg641-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg641-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg641-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg641-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg641-client.virtnet: -o user_xattr,flock 192.168.206.141@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 683 (time 1789069870.68 total 10.01 last 68.23) - open/close 1370 (time 1789069880.70 total 20.02 last 68.61) - open/close 2062 (time 1789069890.70 total 30.03 last 69.18) - open/close 2771 (time 1789069900.70 total 40.03 last 70.89) - open/close 3481 (time 1789069910.71 total 50.03 last 70.96) - open/close 4240 (time 1789069920.72 total 60.05 last 75.78) total: 4500 open/close in 63.54 seconds: 70.82 ops/second - /unlink 1386 (time 1789069938.72 total 10.00 last 138.55) - /unlink 2679 (time 1789069948.72 total 20.00 last 129.29) - /unlink 3938 (time 1789069958.72 total 30.01 last 125.83) total: 4500 /unlink in 34.25 seconds: 131.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 668 (time 1789069978.66 total 10.01 last 66.75) - open/close 1330 (time 1789069988.66 total 20.01 last 66.18) - open/close 2036 (time 1789069998.67 total 30.02 last 70.56) - open/close 2757 (time 1789070008.68 total 40.02 last 72.04) - open/close 3524 (time 1789070018.68 total 50.03 last 76.65) - open/close 4256 (time 1789070028.72 total 60.06 last 72.96) total: 4500 open/close in 63.12 seconds: 71.30 ops/second - /unlink 1358 (time 1789070045.82 total 10.00 last 135.75) - /unlink 2804 (time 1789070055.82 total 20.01 last 144.57) - /unlink 4281 (time 1789070065.82 total 30.01 last 147.63) total: 4500 /unlink in 31.60 seconds: 142.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 666 (time 1789070081.74 total 10.01 last 66.56) - open/close 1380 (time 1789070091.75 total 20.01 last 71.35) - open/close 2042 (time 1789070101.76 total 30.03 last 66.11) - open/close 2797 (time 1789070111.77 total 40.04 last 75.42) - open/close 3472 (time 1789070121.77 total 50.04 last 67.47) - open/close 4248 (time 1789070131.80 total 60.07 last 77.38) total: 4500 open/close in 63.31 seconds: 71.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1364 (time 1789070149.82 total 10.00 last 136.34) - /unlink 2588 (time 1789070159.83 total 20.01 last 122.33) - /unlink 3876 (time 1789070169.84 total 30.02 last 128.66) total: 4500 /unlink in 34.25 seconds: 131.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 796 (time 1789070189.37 total 10.00 last 79.60) - open/close 1560 (time 1789070199.38 total 20.01 last 76.34) - open/close 2304 (time 1789070209.38 total 30.01 last 74.39) - open/close 2954 (time 1789070219.39 total 40.01 last 64.97) - open/close 3554 (time 1789070229.39 total 50.02 last 59.98) - open/close 4175 (time 1789070239.41 total 60.03 last 62.02) total: 4500 open/close in 64.91 seconds: 69.32 ops/second - /unlink 1311 (time 1789070259.90 total 10.03 last 130.70) - /unlink 2552 (time 1789070269.90 total 20.04 last 124.03) - /unlink 3725 (time 1789070279.91 total 30.04 last 117.23) total: 4500 /unlink in 36.48 seconds: 123.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 690 (time 1789070302.29 total 10.00 last 68.98) - open/close 1360 (time 1789070312.29 total 20.00 last 67.00) - open/close 1935 (time 1789070322.29 total 30.01 last 57.48) - open/close 2665 (time 1789070332.30 total 40.02 last 72.92) - open/close 3332 (time 1789070342.34 total 50.05 last 66.48) - open/close 3989 (time 1789070352.34 total 60.05 last 65.68) total: 4500 open/close in 67.38 seconds: 66.79 ops/second - /unlink 1413 (time 1789070373.32 total 10.00 last 141.25) - /unlink 2664 (time 1789070383.32 total 20.00 last 125.09) - /unlink 4096 (time 1789070393.32 total 30.01 last 143.17) total: 4500 /unlink in 32.49 seconds: 138.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 721 (time 1789070410.71 total 10.01 last 72.02) - open/close 1458 (time 1789070420.71 total 20.01 last 73.68) - open/close 2262 (time 1789070430.72 total 30.02 last 80.40) - open/close 2991 (time 1789070440.72 total 40.02 last 72.89) - open/close 3756 (time 1789070450.72 total 50.02 last 76.50) total: 4500 open/close in 59.77 seconds: 75.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1412 (time 1789070474.66 total 10.00 last 141.20) - /unlink 2737 (time 1789070484.66 total 20.00 last 132.47) - /unlink 4154 (time 1789070494.67 total 30.00 last 141.68) total: 4500 /unlink in 32.51 seconds: 138.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 709 (time 1789070512.17 total 10.00 last 70.87) - open/close 1421 (time 1789070522.17 total 20.01 last 71.15) - open/close 2181 (time 1789070532.18 total 30.02 last 75.97) - open/close 2979 (time 1789070542.18 total 40.02 last 79.77) - open/close 3734 (time 1789070552.19 total 50.02 last 75.46) - open/close 4372 (time 1789070562.19 total 60.03 last 63.76) total: 4500 open/close in 61.97 seconds: 72.62 ops/second - /unlink 1231 (time 1789070579.82 total 10.00 last 123.08) - /unlink 2374 (time 1789070589.82 total 20.00 last 114.27) - /unlink 3353 (time 1789070599.83 total 30.01 last 97.79) total: 4500 /unlink in 38.35 seconds: 117.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 705 (time 1789070623.33 total 10.01 last 70.46) - open/close 1380 (time 1789070633.35 total 20.02 last 67.38) - open/close 2131 (time 1789070643.35 total 30.03 last 75.07) - open/close 2933 (time 1789070653.37 total 40.04 last 80.09) - open/close 3617 (time 1789070663.37 total 50.04 last 68.39) - open/close 4357 (time 1789070673.39 total 60.07 last 73.84) total: 4500 open/close in 61.88 seconds: 72.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1272 (time 1789070689.42 total 10.01 last 127.10) - /unlink 2589 (time 1789070699.44 total 20.02 last 131.52) - /unlink 3928 (time 1789070709.44 total 30.03 last 133.84) total: 4500 /unlink in 34.39 seconds: 130.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 712 (time 1789070729.27 total 10.01 last 71.16) - open/close 1485 (time 1789070739.28 total 20.02 last 77.23) - open/close 2326 (time 1789070749.28 total 30.02 last 84.09) - open/close 3098 (time 1789070759.28 total 40.02 last 77.15) - open/close 3852 (time 1789070769.30 total 50.04 last 75.28) total: 4500 open/close in 57.89 seconds: 77.73 ops/second - /unlink 1368 (time 1789070791.97 total 10.00 last 136.78) - /unlink 2803 (time 1789070801.97 total 20.00 last 143.49) - /unlink 4113 (time 1789070811.97 total 30.00 last 130.97) total: 4500 /unlink in 32.92 seconds: 136.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 574 (time 1789070831.84 total 10.01 last 57.34) - open/close 1154 (time 1789070841.85 total 20.02 last 57.94) - open/close 1817 (time 1789070851.86 total 30.04 last 66.19) - open/close 2456 (time 1789070861.87 total 40.04 last 63.87) - open/close 2974 (time 1789070871.88 total 50.05 last 51.75) - open/close 3623 (time 1789070881.89 total 60.06 last 64.83) - open/close 4191 (time 1789070891.90 total 70.08 last 56.73) total: 4500 open/close in 75.87 seconds: 59.31 ops/second - /unlink 1265 (time 1789070913.00 total 10.01 last 126.36) - /unlink 2395 (time 1789070923.01 total 20.03 last 112.83) - /unlink 3637 (time 1789070933.03 total 30.05 last 123.96) total: 4500 /unlink in 36.64 seconds: 122.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 572 (time 1789070956.00 total 10.00 last 57.17) - open/close 1177 (time 1789070966.01 total 20.01 last 60.48) - open/close 1904 (time 1789070976.03 total 30.03 last 72.51) - open/close 2571 (time 1789070986.06 total 40.06 last 66.55) - open/close 3131 (time 1789070996.11 total 50.11 last 55.69) - open/close 3795 (time 1789071006.12 total 60.12 last 66.35) - open/close 4411 (time 1789071016.12 total 70.12 last 61.57) total: 4500 open/close in 71.19 seconds: 63.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1271 (time 1789071032.24 total 10.01 last 127.02) - /unlink 2470 (time 1789071042.24 total 20.01 last 119.83) - /unlink 3660 (time 1789071052.25 total 30.02 last 118.92) total: 4500 /unlink in 36.54 seconds: 123.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 597 (time 1789071075.10 total 10.01 last 59.62) - open/close 1252 (time 1789071085.11 total 20.02 last 65.46) - open/close 1771 (time 1789071095.15 total 30.06 last 51.67) - open/close 2315 (time 1789071105.16 total 40.07 last 54.37) - open/close 2965 (time 1789071115.16 total 50.07 last 64.99) - open/close 3650 (time 1789071125.16 total 60.07 last 68.47) - open/close 4194 (time 1789071135.17 total 70.09 last 54.34) total: 4500 open/close in 75.25 seconds: 59.80 ops/second - /unlink 1406 (time 1789071155.34 total 10.01 last 140.43) - /unlink 2835 (time 1789071165.35 total 20.03 last 142.68) - /unlink 4051 (time 1789071175.35 total 30.03 last 121.58) total: 4500 /unlink in 33.15 seconds: 135.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 747 (time 1789071193.84 total 10.01 last 74.65) - open/close 1412 (time 1789071203.85 total 20.01 last 66.46) - open/close 2130 (time 1789071213.86 total 30.03 last 71.70) - open/close 2880 (time 1789071223.87 total 40.04 last 74.94) - open/close 3529 (time 1789071233.88 total 50.04 last 64.85) - open/close 4188 (time 1789071243.88 total 60.05 last 65.88) total: 4500 open/close in 64.98 seconds: 69.26 ops/second - /unlink 1116 (time 1789071263.26 total 10.00 last 111.60) - /unlink 2320 (time 1789071273.26 total 20.01 last 120.34) - /unlink 3358 (time 1789071283.27 total 30.01 last 103.74) - /unlink 4437 (time 1789071293.27 total 40.01 last 107.86) total: 4500 /unlink in 40.89 seconds: 110.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 541 (time 1789071311.66 total 10.02 last 54.02) - open/close 1151 (time 1789071321.67 total 20.03 last 60.94) - open/close 1784 (time 1789071331.68 total 30.03 last 63.27) - open/close 2416 (time 1789071341.68 total 40.03 last 63.20) - open/close 2914 (time 1789071351.68 total 50.03 last 49.78) - open/close 3328 (time 1789071361.69 total 60.04 last 41.38) - open/close 3773 (time 1789071371.69 total 70.04 last 44.50) - open/close 4293 (time 1789071381.70 total 80.05 last 51.96) total: 4500 open/close in 84.03 seconds: 53.55 ops/second - /unlink 1272 (time 1789071400.32 total 10.00 last 127.15) - /unlink 2572 (time 1789071410.33 total 20.01 last 129.91) - /unlink 3791 (time 1789071420.33 total 30.01 last 121.88) total: 4500 /unlink in 35.78 seconds: 125.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 726 (time 1789071441.06 total 10.00 last 72.57) - open/close 1400 (time 1789071451.07 total 20.01 last 67.35) - open/close 2187 (time 1789071461.07 total 30.02 last 78.67) - open/close 2822 (time 1789071471.08 total 40.02 last 63.44) - open/close 3533 (time 1789071481.08 total 50.03 last 71.08) - open/close 4132 (time 1789071491.09 total 60.03 last 59.88) total: 4500 open/close in 67.62 seconds: 66.55 ops/second - /unlink 1443 (time 1789071514.82 total 10.00 last 144.26) - /unlink 2628 (time 1789071524.82 total 20.01 last 118.47) - /unlink 3912 (time 1789071534.82 total 30.01 last 128.35) total: 4500 /unlink in 34.57 seconds: 130.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 638 (time 1789071555.16 total 10.01 last 63.76) - open/close 1353 (time 1789071565.18 total 20.03 last 71.36) - open/close 2020 (time 1789071575.19 total 30.03 last 66.66) - open/close 2725 (time 1789071585.20 total 40.04 last 70.43) - open/close 3424 (time 1789071595.20 total 50.05 last 69.85) - open/close 4087 (time 1789071605.22 total 60.06 last 66.22) total: 4500 open/close in 66.60 seconds: 67.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1253 (time 1789071626.76 total 10.02 last 125.07) - /unlink 2513 (time 1789071636.76 total 20.02 last 125.93) - /unlink 3870 (time 1789071646.76 total 30.02 last 135.69) total: 4500 /unlink in 34.71 seconds: 129.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 739 (time 1789071666.22 total 10.01 last 73.84) - open/close 1424 (time 1789071676.25 total 20.04 last 68.31) - open/close 2069 (time 1789071686.25 total 30.04 last 64.49) - open/close 2688 (time 1789071696.26 total 40.06 last 61.79) - open/close 3145 (time 1789071706.28 total 50.07 last 45.65) - open/close 3574 (time 1789071716.28 total 60.07 last 42.87) - open/close 4082 (time 1789071726.29 total 70.08 last 50.79) total: 4500 open/close in 77.42 seconds: 58.12 ops/second - /unlink 881 (time 1789071747.88 total 10.00 last 88.06) - /unlink 2108 (time 1789071757.89 total 20.02 last 122.54) - /unlink 3311 (time 1789071767.90 total 30.02 last 120.22) total: 4500 /unlink in 40.01 seconds: 112.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 753 (time 1789071792.76 total 10.01 last 75.24) - open/close 1358 (time 1789071802.81 total 20.06 last 60.20) - open/close 2029 (time 1789071812.81 total 30.06 last 67.05) - open/close 2793 (time 1789071822.82 total 40.07 last 76.38) - open/close 3483 (time 1789071832.82 total 50.07 last 68.96) - open/close 4192 (time 1789071842.82 total 60.07 last 70.89) total: 4500 open/close in 64.72 seconds: 69.53 ops/second - /unlink 1380 (time 1789071862.12 total 10.00 last 137.93) - /unlink 2721 (time 1789071872.12 total 20.01 last 134.05) - /unlink 4073 (time 1789071882.12 total 30.01 last 135.15) total: 4500 /unlink in 33.05 seconds: 136.14 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95192 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg641-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server stop mds service on oleg641-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg641-server stop mds service on oleg641-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg641-server unloading modules via unload_modules_local on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing unload_modules_local oleg641-server: kmemleak disabled modules unloaded.