== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:30:57 (1788845457) Stopping clients: oleg630-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg630-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg630-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg630-server oleg630-server: oleg630-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: kmemleak disabled oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg630-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg630-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 794 (time 1788845635.65 total 10.01 last 79.32) - open/close 1580 (time 1788845645.66 total 20.02 last 78.51) - open/close 2258 (time 1788845655.66 total 30.03 last 67.77) - open/close 2840 (time 1788845665.67 total 40.03 last 58.16) - open/close 3564 (time 1788845675.67 total 50.03 last 72.40) - open/close 4308 (time 1788845685.69 total 60.05 last 74.29) total: 4500 open/close in 62.38 seconds: 72.14 ops/second - /unlink 1410 (time 1788845702.31 total 10.00 last 140.93) - /unlink 2813 (time 1788845712.31 total 20.01 last 140.30) - /unlink 4366 (time 1788845722.31 total 30.01 last 155.27) total: 4500 /unlink in 31.10 seconds: 144.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 787 (time 1788845737.52 total 10.01 last 78.65) - open/close 1547 (time 1788845747.53 total 20.01 last 75.96) - open/close 2247 (time 1788845757.54 total 30.03 last 69.88) - open/close 2961 (time 1788845767.55 total 40.04 last 71.34) - open/close 3714 (time 1788845777.56 total 50.04 last 75.24) - open/close 4468 (time 1788845787.56 total 60.05 last 75.35) total: 4500 open/close in 60.50 seconds: 74.38 ops/second - /unlink 1488 (time 1788845801.88 total 10.00 last 148.73) - /unlink 2795 (time 1788845811.88 total 20.00 last 130.70) - /unlink 4085 (time 1788845821.89 total 30.01 last 128.87) total: 4500 /unlink in 33.22 seconds: 135.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 653 (time 1788845839.01 total 10.02 last 65.15) - open/close 1215 (time 1788845849.01 total 20.03 last 56.17) - open/close 1985 (time 1788845859.03 total 30.04 last 76.89) - open/close 2704 (time 1788845869.04 total 40.05 last 71.82) - open/close 3442 (time 1788845879.04 total 50.06 last 73.79) - open/close 4192 (time 1788845889.04 total 60.06 last 75.00) total: 4500 open/close in 64.03 seconds: 70.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1466 (time 1788845906.78 total 10.01 last 146.50) - /unlink 2832 (time 1788845916.78 total 20.01 last 136.56) - /unlink 4161 (time 1788845926.79 total 30.01 last 132.88) total: 4500 /unlink in 32.37 seconds: 139.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 765 (time 1788845943.59 total 10.01 last 76.42) - open/close 1471 (time 1788845953.59 total 20.02 last 70.54) - open/close 2236 (time 1788845963.60 total 30.03 last 76.46) - open/close 2998 (time 1788845973.61 total 40.04 last 76.12) - open/close 3746 (time 1788845983.62 total 50.05 last 74.71) - open/close 4474 (time 1788845993.63 total 60.05 last 72.78) total: 4500 open/close in 60.39 seconds: 74.51 ops/second - /unlink 1593 (time 1788846007.84 total 10.01 last 159.11) - /unlink 2964 (time 1788846017.84 total 20.01 last 137.06) - /unlink 4300 (time 1788846027.85 total 30.02 last 133.50) total: 4500 /unlink in 31.44 seconds: 143.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 718 (time 1788846044.22 total 10.02 last 71.62) - open/close 1384 (time 1788846054.23 total 20.03 last 66.54) - open/close 2165 (time 1788846064.23 total 30.04 last 78.04) - open/close 2856 (time 1788846074.23 total 40.04 last 69.10) - open/close 3633 (time 1788846084.25 total 50.05 last 77.61) - open/close 4404 (time 1788846094.27 total 60.08 last 76.94) total: 4500 open/close in 61.47 seconds: 73.21 ops/second - /unlink 1418 (time 1788846109.71 total 10.01 last 141.67) - /unlink 2812 (time 1788846119.71 total 20.01 last 139.40) - /unlink 4225 (time 1788846129.71 total 30.01 last 141.26) total: 4500 /unlink in 32.43 seconds: 138.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 754 (time 1788846147.34 total 10.01 last 75.36) - open/close 1449 (time 1788846157.34 total 20.01 last 69.46) - open/close 2198 (time 1788846167.36 total 30.03 last 74.76) - open/close 2957 (time 1788846177.37 total 40.03 last 75.88) - open/close 3653 (time 1788846187.37 total 50.04 last 69.55) - open/close 4334 (time 1788846197.38 total 60.05 last 68.03) total: 4500 open/close in 62.43 seconds: 72.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1199 (time 1788846215.14 total 10.00 last 119.88) - /unlink 2619 (time 1788846225.14 total 20.01 last 141.90) - /unlink 4026 (time 1788846235.15 total 30.01 last 140.65) total: 4500 /unlink in 33.42 seconds: 134.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 748 (time 1788846254.09 total 10.00 last 74.78) - open/close 1518 (time 1788846264.10 total 20.01 last 76.97) - open/close 2226 (time 1788846274.11 total 30.02 last 70.70) - open/close 2879 (time 1788846284.12 total 40.03 last 65.23) - open/close 3568 (time 1788846294.13 total 50.05 last 68.81) - open/close 4251 (time 1788846304.13 total 60.05 last 68.29) total: 4500 open/close in 63.84 seconds: 70.49 ops/second - /unlink 1447 (time 1788846322.83 total 10.00 last 144.64) - /unlink 2736 (time 1788846332.83 total 20.01 last 128.84) - /unlink 4108 (time 1788846342.84 total 30.01 last 137.18) total: 4500 /unlink in 32.69 seconds: 137.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 771 (time 1788846360.20 total 10.00 last 77.08) - open/close 1518 (time 1788846370.20 total 20.00 last 74.70) - open/close 2220 (time 1788846380.21 total 30.01 last 70.13) - open/close 3001 (time 1788846390.21 total 40.02 last 78.07) - open/close 3715 (time 1788846400.22 total 50.02 last 71.35) - open/close 4416 (time 1788846410.22 total 60.02 last 70.09) total: 4500 open/close in 61.33 seconds: 73.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1437 (time 1788846425.99 total 10.01 last 143.62) - /unlink 2738 (time 1788846435.99 total 20.01 last 130.03) - /unlink 3986 (time 1788846445.99 total 30.01 last 124.80) total: 4500 /unlink in 33.89 seconds: 132.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 735 (time 1788846464.74 total 10.00 last 73.47) - open/close 1461 (time 1788846474.75 total 20.02 last 72.51) - open/close 2230 (time 1788846484.76 total 30.03 last 76.82) - open/close 2953 (time 1788846494.77 total 40.03 last 72.24) - open/close 3706 (time 1788846504.77 total 50.04 last 75.26) - open/close 4447 (time 1788846514.79 total 60.06 last 73.97) total: 4500 open/close in 60.90 seconds: 73.89 ops/second - /unlink 1327 (time 1788846530.12 total 10.00 last 132.68) - /unlink 2669 (time 1788846540.12 total 20.00 last 134.15) - /unlink 4069 (time 1788846550.13 total 30.01 last 139.93) total: 4500 /unlink in 33.35 seconds: 134.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 674 (time 1788846568.38 total 10.00 last 67.37) - open/close 1378 (time 1788846578.38 total 20.01 last 70.39) - open/close 2139 (time 1788846588.38 total 30.01 last 76.10) - open/close 2852 (time 1788846598.38 total 40.01 last 71.29) - open/close 3568 (time 1788846608.38 total 50.01 last 71.60) - open/close 4322 (time 1788846618.39 total 60.01 last 75.38) total: 4500 open/close in 62.45 seconds: 72.06 ops/second - /unlink 1297 (time 1788846635.54 total 10.00 last 129.67) - /unlink 2532 (time 1788846645.55 total 20.01 last 123.43) - /unlink 3967 (time 1788846655.55 total 30.01 last 143.40) total: 4500 /unlink in 33.89 seconds: 132.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 714 (time 1788846674.76 total 10.01 last 71.35) - open/close 1423 (time 1788846684.78 total 20.02 last 70.82) - open/close 2012 (time 1788846694.79 total 30.03 last 58.83) - open/close 2732 (time 1788846704.79 total 40.04 last 71.95) - open/close 3521 (time 1788846714.80 total 50.05 last 78.83) - open/close 4255 (time 1788846724.81 total 60.05 last 73.36) total: 4500 open/close in 63.39 seconds: 70.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1385 (time 1788846742.08 total 10.00 last 138.44) - /unlink 2725 (time 1788846752.08 total 20.01 last 133.98) - /unlink 4160 (time 1788846762.08 total 30.01 last 143.48) total: 4500 /unlink in 32.73 seconds: 137.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 774 (time 1788846779.47 total 10.01 last 77.32) - open/close 1530 (time 1788846789.48 total 20.02 last 75.54) - open/close 2285 (time 1788846799.48 total 30.02 last 75.48) - open/close 2977 (time 1788846809.48 total 40.02 last 69.18) - open/close 3705 (time 1788846819.49 total 50.03 last 72.73) - open/close 4484 (time 1788846829.50 total 60.04 last 77.84) total: 4500 open/close in 60.31 seconds: 74.62 ops/second - /unlink 1392 (time 1788846843.89 total 10.00 last 139.15) - /unlink 2677 (time 1788846853.89 total 20.01 last 128.48) - /unlink 4133 (time 1788846863.89 total 30.01 last 145.54) total: 4500 /unlink in 32.42 seconds: 138.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 761 (time 1788846880.68 total 10.01 last 76.04) - open/close 1523 (time 1788846890.68 total 20.01 last 76.18) - open/close 2249 (time 1788846900.69 total 30.02 last 72.55) - open/close 3038 (time 1788846910.70 total 40.02 last 78.85) - open/close 3857 (time 1788846920.71 total 50.03 last 81.83) total: 4500 open/close in 58.25 seconds: 77.26 ops/second - /unlink 1427 (time 1788846942.54 total 10.02 last 142.47) - /unlink 2780 (time 1788846952.54 total 20.02 last 135.23) - /unlink 4199 (time 1788846962.55 total 30.03 last 141.83) total: 4500 /unlink in 32.16 seconds: 139.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 767 (time 1788846978.55 total 10.01 last 76.64) - open/close 1475 (time 1788846988.55 total 20.01 last 70.76) - open/close 2193 (time 1788846998.56 total 30.02 last 71.77) - open/close 2877 (time 1788847008.56 total 40.02 last 68.38) - open/close 3584 (time 1788847018.56 total 50.03 last 70.67) - open/close 4288 (time 1788847028.57 total 60.03 last 70.34) total: 4500 open/close in 63.02 seconds: 71.41 ops/second - /unlink 1441 (time 1788847045.93 total 10.00 last 144.09) - /unlink 2788 (time 1788847055.93 total 20.00 last 134.66) - /unlink 4195 (time 1788847065.94 total 30.01 last 140.62) total: 4500 /unlink in 32.43 seconds: 138.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 794 (time 1788847083.27 total 10.00 last 79.37) - open/close 1540 (time 1788847093.28 total 20.01 last 74.55) - open/close 2299 (time 1788847103.29 total 30.03 last 75.78) - open/close 3035 (time 1788847113.29 total 40.03 last 73.59) - open/close 3793 (time 1788847123.30 total 50.03 last 75.75) total: 4500 open/close in 59.54 seconds: 75.58 ops/second - /unlink 1315 (time 1788847147.75 total 10.00 last 131.48) - /unlink 2388 (time 1788847157.75 total 20.01 last 107.26) - /unlink 3577 (time 1788847167.76 total 30.01 last 118.82) total: 4500 /unlink in 38.98 seconds: 115.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 751 (time 1788847191.81 total 10.01 last 75.06) - open/close 1479 (time 1788847201.82 total 20.02 last 72.69) - open/close 2213 (time 1788847211.83 total 30.03 last 73.33) - open/close 2964 (time 1788847221.84 total 40.04 last 75.02) - open/close 3653 (time 1788847231.85 total 50.04 last 68.90) - open/close 4407 (time 1788847241.86 total 60.06 last 75.27) total: 4500 open/close in 61.21 seconds: 73.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1362 (time 1788847256.78 total 10.02 last 135.99) - /unlink 2650 (time 1788847266.79 total 20.02 last 128.73) - /unlink 3892 (time 1788847276.80 total 30.03 last 124.06) total: 4500 /unlink in 35.57 seconds: 126.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 720 (time 1788847297.83 total 10.00 last 71.98) - open/close 1423 (time 1788847307.85 total 20.02 last 70.16) - open/close 2214 (time 1788847317.85 total 30.02 last 79.10) - open/close 2953 (time 1788847327.86 total 40.03 last 73.82) - open/close 3659 (time 1788847337.86 total 50.04 last 70.59) - open/close 4417 (time 1788847347.87 total 60.05 last 75.71) total: 4500 open/close in 61.05 seconds: 73.71 ops/second - /unlink 1312 (time 1788847363.03 total 10.02 last 130.99) - /unlink 2703 (time 1788847373.04 total 20.02 last 139.00) - /unlink 4135 (time 1788847383.04 total 30.02 last 143.19) total: 4500 /unlink in 32.51 seconds: 138.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 771 (time 1788847399.44 total 10.01 last 77.04) - open/close 1473 (time 1788847409.45 total 20.01 last 70.16) - open/close 2187 (time 1788847419.45 total 30.01 last 71.40) - open/close 2883 (time 1788847429.45 total 40.02 last 69.58) - open/close 3523 (time 1788847439.47 total 50.03 last 63.88) - open/close 4148 (time 1788847449.48 total 60.04 last 62.44) total: 4500 open/close in 65.47 seconds: 68.74 ops/second - /unlink 1335 (time 1788847469.29 total 10.00 last 133.45) - /unlink 2749 (time 1788847479.29 total 20.00 last 141.39) - /unlink 4100 (time 1788847489.29 total 30.01 last 135.09) total: 4500 /unlink in 33.33 seconds: 135.00 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95198 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg630-server unloading modules via unload_modules_local on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: kmemleak disabled modules unloaded.