== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:26:48 (1789043208) Stopping clients: oleg206-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg206-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg206-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg206-server oleg206-server: oleg206-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: kmemleak disabled oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg206-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 795 (time 1789043441.16 total 10.00 last 79.49) - open/close 1499 (time 1789043451.16 total 20.00 last 70.39) - open/close 2289 (time 1789043461.16 total 30.01 last 78.98) - open/close 3046 (time 1789043471.16 total 40.01 last 75.69) - open/close 3766 (time 1789043481.17 total 50.01 last 71.97) total: 4500 open/close in 59.33 seconds: 75.84 ops/second - /unlink 1356 (time 1789043504.81 total 10.00 last 135.57) - /unlink 2598 (time 1789043514.81 total 20.00 last 124.20) - /unlink 3904 (time 1789043524.81 total 30.00 last 130.59) total: 4500 /unlink in 35.35 seconds: 127.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 607 (time 1789043545.94 total 10.01 last 60.65) - open/close 1249 (time 1789043555.96 total 20.03 last 64.08) - open/close 1962 (time 1789043565.97 total 30.03 last 71.25) - open/close 2568 (time 1789043575.98 total 40.04 last 60.55) - open/close 3328 (time 1789043586.00 total 50.06 last 75.86) - open/close 4088 (time 1789043596.01 total 60.07 last 75.91) total: 4500 open/close in 65.25 seconds: 68.97 ops/second - /unlink 1199 (time 1789043616.77 total 10.01 last 119.82) - /unlink 2434 (time 1789043626.77 total 20.01 last 123.47) - /unlink 3753 (time 1789043636.78 total 30.02 last 131.83) total: 4500 /unlink in 35.30 seconds: 127.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 728 (time 1789043656.50 total 10.01 last 72.70) - open/close 1399 (time 1789043666.52 total 20.04 last 66.92) - open/close 2090 (time 1789043676.53 total 30.05 last 69.04) - open/close 2533 (time 1789043686.53 total 40.05 last 44.30) - open/close 2963 (time 1789043696.54 total 50.06 last 42.97) - open/close 3728 (time 1789043706.54 total 60.06 last 76.49) - open/close 4423 (time 1789043716.55 total 70.06 last 69.47) total: 4500 open/close in 71.35 seconds: 63.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1087 (time 1789043732.75 total 10.01 last 108.64) - /unlink 2180 (time 1789043742.76 total 20.02 last 109.13) - /unlink 3530 (time 1789043752.76 total 30.02 last 134.96) total: 4500 /unlink in 37.97 seconds: 118.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 752 (time 1789043776.39 total 10.00 last 75.18) - open/close 1470 (time 1789043786.40 total 20.02 last 71.69) - open/close 2316 (time 1789043796.41 total 30.02 last 84.56) - open/close 3032 (time 1789043806.41 total 40.03 last 71.57) - open/close 3703 (time 1789043816.42 total 50.03 last 67.09) - open/close 4441 (time 1789043826.42 total 60.04 last 73.74) total: 4500 open/close in 60.78 seconds: 74.04 ops/second - /unlink 1595 (time 1789043842.45 total 10.00 last 159.43) - /unlink 2848 (time 1789043852.46 total 20.01 last 125.23) - /unlink 4156 (time 1789043862.46 total 30.01 last 130.78) total: 4500 /unlink in 32.41 seconds: 138.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 744 (time 1789043879.92 total 10.00 last 74.37) - open/close 1582 (time 1789043889.93 total 20.01 last 83.73) - open/close 2230 (time 1789043899.93 total 30.01 last 64.80) - open/close 3053 (time 1789043909.93 total 40.01 last 82.28) - open/close 3704 (time 1789043919.94 total 50.03 last 65.03) - open/close 4354 (time 1789043929.94 total 60.03 last 64.97) total: 4500 open/close in 62.08 seconds: 72.49 ops/second - /unlink 1289 (time 1789043947.90 total 10.01 last 128.78) - /unlink 2491 (time 1789043957.90 total 20.01 last 120.18) - /unlink 3814 (time 1789043967.91 total 30.02 last 132.21) total: 4500 /unlink in 35.15 seconds: 128.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 651 (time 1789043988.34 total 10.01 last 65.05) - open/close 1434 (time 1789043998.36 total 20.02 last 78.19) - open/close 2215 (time 1789044008.36 total 30.02 last 78.08) - open/close 3054 (time 1789044018.36 total 40.03 last 83.89) - open/close 3739 (time 1789044028.36 total 50.03 last 68.48) - open/close 4377 (time 1789044038.37 total 60.03 last 63.76) total: 4500 open/close in 62.04 seconds: 72.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1419 (time 1789044055.17 total 10.01 last 141.77) - /unlink 2702 (time 1789044065.17 total 20.01 last 128.28) - /unlink 4044 (time 1789044075.17 total 30.01 last 134.18) total: 4500 /unlink in 33.99 seconds: 132.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 642 (time 1789044095.39 total 10.01 last 64.15) - open/close 1243 (time 1789044105.40 total 20.01 last 60.09) - open/close 1774 (time 1789044115.42 total 30.03 last 52.97) - open/close 2519 (time 1789044125.43 total 40.05 last 74.41) - open/close 3225 (time 1789044135.46 total 50.08 last 70.40) - open/close 3872 (time 1789044145.47 total 60.09 last 64.64) total: 4500 open/close in 67.51 seconds: 66.66 ops/second - /unlink 1372 (time 1789044167.62 total 10.01 last 137.08) - /unlink 2602 (time 1789044177.62 total 20.01 last 122.97) - /unlink 3814 (time 1789044187.62 total 30.01 last 121.16) total: 4500 /unlink in 35.83 seconds: 125.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 487 (time 1789044211.13 total 10.01 last 48.67) - open/close 1002 (time 1789044221.14 total 20.02 last 51.42) - open/close 1438 (time 1789044231.15 total 30.03 last 43.56) - open/close 1986 (time 1789044241.16 total 40.04 last 54.78) - open/close 2729 (time 1789044251.16 total 50.04 last 74.25) - open/close 3378 (time 1789044261.18 total 60.06 last 64.81) - open/close 4013 (time 1789044271.20 total 70.08 last 63.34) total: 4500 open/close in 78.16 seconds: 57.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1149 (time 1789044294.14 total 10.02 last 114.71) - /unlink 2273 (time 1789044304.15 total 20.03 last 112.30) - /unlink 3584 (time 1789044314.15 total 30.03 last 131.05) total: 4500 /unlink in 38.52 seconds: 116.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 693 (time 1789044338.26 total 10.00 last 69.28) - open/close 1404 (time 1789044348.30 total 20.04 last 70.82) - open/close 2082 (time 1789044358.31 total 30.05 last 67.72) - open/close 2683 (time 1789044368.31 total 40.06 last 60.07) - open/close 3257 (time 1789044378.33 total 50.08 last 57.28) - open/close 3917 (time 1789044388.36 total 60.10 last 65.84) - open/close 4368 (time 1789044398.37 total 70.11 last 45.05) total: 4500 open/close in 72.44 seconds: 62.12 ops/second - /unlink 1077 (time 1789044415.78 total 10.00 last 107.70) - /unlink 2128 (time 1789044425.79 total 20.01 last 105.03) - /unlink 3665 (time 1789044435.79 total 30.01 last 153.66) total: 4500 /unlink in 38.05 seconds: 118.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 396 (time 1789044459.99 total 10.02 last 39.54) - open/close 930 (time 1789044470.01 total 20.04 last 53.26) - open/close 1720 (time 1789044480.02 total 30.05 last 78.93) - open/close 2454 (time 1789044490.03 total 40.06 last 73.35) - open/close 3125 (time 1789044500.05 total 50.07 last 67.00) - open/close 3838 (time 1789044510.06 total 60.08 last 71.23) - open/close 4335 (time 1789044520.06 total 70.09 last 49.68) total: 4500 open/close in 72.92 seconds: 61.71 ops/second - /unlink 1682 (time 1789044540.16 total 10.00 last 168.15) - /unlink 3263 (time 1789044550.18 total 20.02 last 157.84) total: 4500 /unlink in 29.62 seconds: 151.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 749 (time 1789044575.54 total 10.00 last 74.87) - open/close 1608 (time 1789044585.55 total 20.01 last 85.81) - open/close 2266 (time 1789044595.57 total 30.03 last 65.69) - open/close 2949 (time 1789044605.58 total 40.04 last 68.23) - open/close 3730 (time 1789044615.61 total 50.07 last 77.89) - open/close 4481 (time 1789044625.62 total 60.08 last 75.00) total: 4500 open/close in 60.49 seconds: 74.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1302 (time 1789044642.39 total 10.00 last 130.15) - /unlink 2674 (time 1789044652.39 total 20.01 last 137.19) - /unlink 4049 (time 1789044662.40 total 30.01 last 137.44) total: 4500 /unlink in 33.73 seconds: 133.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 664 (time 1789044682.12 total 10.01 last 66.35) - open/close 1212 (time 1789044692.13 total 20.02 last 54.71) - open/close 1730 (time 1789044702.14 total 30.03 last 51.78) - open/close 2525 (time 1789044712.14 total 40.03 last 79.49) - open/close 3240 (time 1789044722.16 total 50.05 last 71.35) - open/close 4016 (time 1789044732.17 total 60.06 last 77.54) total: 4500 open/close in 66.34 seconds: 67.84 ops/second - /unlink 1645 (time 1789044752.73 total 10.01 last 164.41) - /unlink 3161 (time 1789044762.73 total 20.01 last 151.52) total: 4500 /unlink in 28.65 seconds: 157.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 663 (time 1789044786.01 total 10.01 last 66.26) - open/close 1344 (time 1789044796.03 total 20.02 last 67.99) - open/close 1998 (time 1789044806.03 total 30.03 last 65.36) - open/close 2762 (time 1789044816.04 total 40.04 last 76.33) - open/close 3551 (time 1789044826.05 total 50.04 last 78.87) - open/close 4317 (time 1789044836.05 total 60.05 last 76.53) total: 4500 open/close in 62.11 seconds: 72.45 ops/second - /unlink 1496 (time 1789044852.98 total 10.00 last 149.53) - /unlink 3014 (time 1789044862.98 total 20.01 last 151.77) - /unlink 4420 (time 1789044872.98 total 30.01 last 140.53) total: 4500 /unlink in 30.84 seconds: 145.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 700 (time 1789044888.41 total 10.01 last 69.96) - open/close 1512 (time 1789044898.42 total 20.01 last 81.16) - open/close 2278 (time 1789044908.42 total 30.01 last 76.59) - open/close 3078 (time 1789044918.42 total 40.02 last 79.96) - open/close 3941 (time 1789044928.43 total 50.03 last 86.24) total: 4500 open/close in 57.42 seconds: 78.37 ops/second - /unlink 1526 (time 1789044950.67 total 10.00 last 152.56) - /unlink 3035 (time 1789044960.67 total 20.00 last 150.89) - /unlink 4469 (time 1789044970.67 total 30.00 last 143.38) total: 4500 /unlink in 30.43 seconds: 147.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 812 (time 1789044985.64 total 10.00 last 81.19) - open/close 1624 (time 1789044995.64 total 20.00 last 81.18) - open/close 2472 (time 1789045005.65 total 30.01 last 84.73) - open/close 3238 (time 1789045015.66 total 40.02 last 76.55) - open/close 3975 (time 1789045025.67 total 50.03 last 73.64) total: 4500 open/close in 56.36 seconds: 79.85 ops/second - /unlink 1386 (time 1789045047.05 total 10.00 last 138.56) - /unlink 2923 (time 1789045057.08 total 20.03 last 153.37) - /unlink 4363 (time 1789045067.08 total 30.03 last 143.96) total: 4500 /unlink in 31.01 seconds: 145.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 692 (time 1789045082.31 total 10.00 last 69.19) - open/close 1386 (time 1789045092.31 total 20.00 last 69.39) - open/close 2180 (time 1789045102.32 total 30.02 last 79.30) - open/close 2984 (time 1789045112.33 total 40.03 last 80.31) - open/close 3711 (time 1789045122.33 total 50.03 last 72.69) - open/close 4400 (time 1789045132.34 total 60.04 last 68.84) total: 4500 open/close in 61.18 seconds: 73.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1389 (time 1789045148.04 total 10.00 last 138.87) - /unlink 2747 (time 1789045158.05 total 20.00 last 135.77) - /unlink 4205 (time 1789045168.05 total 30.01 last 145.77) total: 4500 /unlink in 32.81 seconds: 137.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 672 (time 1789045186.32 total 10.01 last 67.15) - open/close 1315 (time 1789045196.33 total 20.01 last 64.29) - open/close 2015 (time 1789045206.33 total 30.02 last 69.94) - open/close 2777 (time 1789045216.34 total 40.02 last 76.17) - open/close 3519 (time 1789045226.35 total 50.03 last 74.11) - open/close 4254 (time 1789045236.36 total 60.04 last 73.44) total: 4500 open/close in 63.55 seconds: 70.81 ops/second - /unlink 1345 (time 1789045254.48 total 10.00 last 134.48) - /unlink 2592 (time 1789045264.48 total 20.00 last 124.67) - /unlink 4159 (time 1789045274.49 total 30.01 last 156.62) total: 4500 /unlink in 32.28 seconds: 139.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 714 (time 1789045291.36 total 10.01 last 71.31) - open/close 1488 (time 1789045301.37 total 20.03 last 77.30) - open/close 2079 (time 1789045311.39 total 30.04 last 59.03) - open/close 2750 (time 1789045321.39 total 40.04 last 67.07) - open/close 3556 (time 1789045331.41 total 50.06 last 80.45) - open/close 4183 (time 1789045341.42 total 60.07 last 62.64) total: 4500 open/close in 64.35 seconds: 69.93 ops/second - /unlink 1348 (time 1789045360.34 total 10.01 last 134.66) - /unlink 2700 (time 1789045370.35 total 20.02 last 135.13) - /unlink 4133 (time 1789045380.35 total 30.02 last 143.30) total: 4500 /unlink in 32.95 seconds: 136.58 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95243 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: kmemleak disabled modules unloaded.