== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:06:58 (1789448818) Stopping clients: oleg636-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg636-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg636-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg636-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg636-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg636-server oleg636-server: oleg636-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg636-server' oleg636-server: oleg636-server.virtnet: executing load_modules_local oleg636-server: kmemleak disabled oleg636-server: Loading modules from /home/green/git/lustre-release/lustre oleg636-server: detected 4 online CPUs by sysfs oleg636-server: MODOPTS_LIBCFS= oleg636-server: Force libcfs to create 2 CPU partitions oleg636-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg636-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg636-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg636-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 651 (time 1789449022.40 total 10.00 last 65.07) - open/close 1409 (time 1789449032.41 total 20.02 last 75.70) - open/close 2022 (time 1789449042.41 total 30.02 last 61.26) - open/close 2633 (time 1789449052.42 total 40.03 last 61.08) - open/close 3434 (time 1789449062.42 total 50.03 last 80.06) - open/close 4185 (time 1789449072.44 total 60.05 last 74.96) total: 4500 open/close in 64.78 seconds: 69.47 ops/second - /unlink 1444 (time 1789449091.36 total 10.00 last 144.34) - /unlink 2766 (time 1789449101.36 total 20.01 last 132.16) - /unlink 4113 (time 1789449111.36 total 30.01 last 134.69) total: 4500 /unlink in 33.21 seconds: 135.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 652 (time 1789449130.52 total 10.01 last 65.14) - open/close 1336 (time 1789449140.52 total 20.01 last 68.39) - open/close 2125 (time 1789449150.53 total 30.02 last 78.79) - open/close 2831 (time 1789449160.54 total 40.03 last 70.59) - open/close 3578 (time 1789449170.54 total 50.03 last 74.68) - open/close 4259 (time 1789449180.54 total 60.03 last 68.07) total: 4500 open/close in 63.56 seconds: 70.80 ops/second - /unlink 1335 (time 1789449199.50 total 10.00 last 133.43) - /unlink 2594 (time 1789449209.51 total 20.01 last 125.82) - /unlink 3954 (time 1789449219.51 total 30.02 last 135.94) total: 4500 /unlink in 34.49 seconds: 130.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 736 (time 1789449238.47 total 10.00 last 73.57) - open/close 1445 (time 1789449248.49 total 20.02 last 70.80) - open/close 2231 (time 1789449258.49 total 30.02 last 78.60) - open/close 2952 (time 1789449268.50 total 40.03 last 72.02) - open/close 3735 (time 1789449278.51 total 50.04 last 78.23) - open/close 4431 (time 1789449288.52 total 60.05 last 69.53) total: 4500 open/close in 61.05 seconds: 73.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1436 (time 1789449303.90 total 10.01 last 143.52) - /unlink 2731 (time 1789449313.90 total 20.01 last 129.45) - /unlink 4097 (time 1789449323.91 total 30.02 last 136.48) total: 4500 /unlink in 33.16 seconds: 135.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 698 (time 1789449341.91 total 10.01 last 69.70) - open/close 1430 (time 1789449351.92 total 20.03 last 73.11) - open/close 2074 (time 1789449361.94 total 30.04 last 64.28) - open/close 2777 (time 1789449371.94 total 40.05 last 70.28) - open/close 3486 (time 1789449381.95 total 50.05 last 70.86) - open/close 4244 (time 1789449391.96 total 60.06 last 75.74) total: 4500 open/close in 63.84 seconds: 70.49 ops/second - /unlink 1318 (time 1789449410.27 total 10.00 last 131.77) - /unlink 2495 (time 1789449420.27 total 20.01 last 117.64) - /unlink 3525 (time 1789449430.28 total 30.01 last 102.97) - /unlink 4483 (time 1789449440.28 total 40.01 last 95.79) total: 4500 /unlink in 40.12 seconds: 112.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 591 (time 1789449455.99 total 10.01 last 59.06) - open/close 1179 (time 1789449466.00 total 20.02 last 58.75) - open/close 1846 (time 1789449476.01 total 30.02 last 66.65) - open/close 2595 (time 1789449486.03 total 40.04 last 74.74) - open/close 3337 (time 1789449496.05 total 50.07 last 74.04) - open/close 4104 (time 1789449506.06 total 60.07 last 76.65) total: 4500 open/close in 65.43 seconds: 68.78 ops/second - /unlink 1329 (time 1789449525.61 total 10.00 last 132.88) - /unlink 2599 (time 1789449535.62 total 20.01 last 126.87) - /unlink 3963 (time 1789449545.62 total 30.01 last 136.36) total: 4500 /unlink in 33.82 seconds: 133.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 743 (time 1789449564.46 total 10.00 last 74.29) - open/close 1393 (time 1789449574.47 total 20.01 last 64.94) - open/close 2088 (time 1789449584.47 total 30.02 last 69.45) - open/close 2791 (time 1789449594.48 total 40.02 last 70.28) - open/close 3516 (time 1789449604.48 total 50.03 last 72.46) - open/close 4290 (time 1789449614.49 total 60.03 last 77.35) total: 4500 open/close in 62.98 seconds: 71.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1207 (time 1789449632.08 total 10.00 last 120.69) - /unlink 2515 (time 1789449642.09 total 20.01 last 130.67) - /unlink 3812 (time 1789449652.09 total 30.01 last 129.67) total: 4500 /unlink in 35.83 seconds: 125.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 583 (time 1789449672.32 total 10.03 last 58.10) - open/close 1204 (time 1789449682.34 total 20.05 last 61.99) - open/close 1893 (time 1789449692.35 total 30.06 last 68.85) - open/close 2580 (time 1789449702.37 total 40.09 last 68.50) - open/close 3316 (time 1789449712.38 total 50.09 last 73.57) - open/close 4084 (time 1789449722.39 total 60.10 last 76.75) total: 4500 open/close in 65.97 seconds: 68.21 ops/second - /unlink 1310 (time 1789449742.59 total 10.01 last 130.93) - /unlink 2512 (time 1789449752.59 total 20.01 last 120.19) - /unlink 3858 (time 1789449762.59 total 30.01 last 134.56) total: 4500 /unlink in 34.74 seconds: 129.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 731 (time 1789449781.80 total 10.01 last 73.06) - open/close 1492 (time 1789449791.82 total 20.03 last 75.94) - open/close 2234 (time 1789449801.82 total 30.03 last 74.20) - open/close 3009 (time 1789449811.84 total 40.04 last 77.39) - open/close 3629 (time 1789449821.85 total 50.05 last 61.93) - open/close 4245 (time 1789449831.85 total 60.06 last 61.57) total: 4500 open/close in 63.92 seconds: 70.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1377 (time 1789449850.25 total 10.00 last 137.70) - /unlink 2749 (time 1789449860.26 total 20.00 last 137.16) - /unlink 4125 (time 1789449870.26 total 30.01 last 137.54) total: 4500 /unlink in 33.06 seconds: 136.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 738 (time 1789449887.56 total 10.00 last 73.79) - open/close 1481 (time 1789449897.57 total 20.01 last 74.24) - open/close 2270 (time 1789449907.57 total 30.01 last 78.88) - open/close 3039 (time 1789449917.59 total 40.03 last 76.78) - open/close 3849 (time 1789449927.59 total 50.03 last 81.00) total: 4500 open/close in 58.15 seconds: 77.39 ops/second - /unlink 1473 (time 1789449949.88 total 10.00 last 147.26) - /unlink 2937 (time 1789449959.89 total 20.01 last 146.23) - /unlink 4336 (time 1789449969.90 total 30.02 last 139.81) total: 4500 /unlink in 31.23 seconds: 144.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 764 (time 1789449985.14 total 10.00 last 76.40) - open/close 1533 (time 1789449995.15 total 20.01 last 76.86) - open/close 2259 (time 1789450005.16 total 30.02 last 72.53) - open/close 2985 (time 1789450015.16 total 40.02 last 72.57) - open/close 3697 (time 1789450025.17 total 50.02 last 71.18) - open/close 4444 (time 1789450035.18 total 60.03 last 74.62) total: 4500 open/close in 60.68 seconds: 74.16 ops/second - /unlink 1390 (time 1789450049.45 total 10.00 last 138.94) - /unlink 2719 (time 1789450059.46 total 20.01 last 132.83) - /unlink 4107 (time 1789450069.46 total 30.01 last 138.74) total: 4500 /unlink in 33.07 seconds: 136.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 744 (time 1789450087.03 total 10.02 last 74.28) - open/close 1476 (time 1789450097.05 total 20.04 last 73.02) - open/close 2190 (time 1789450107.06 total 30.05 last 71.33) - open/close 2839 (time 1789450117.07 total 40.06 last 64.84) - open/close 3547 (time 1789450127.10 total 50.09 last 70.61) - open/close 4304 (time 1789450137.10 total 60.09 last 75.69) total: 4500 open/close in 62.60 seconds: 71.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1505 (time 1789450153.39 total 10.01 last 150.42) - /unlink 2763 (time 1789450163.39 total 20.01 last 125.76) - /unlink 3941 (time 1789450173.39 total 30.01 last 117.78) total: 4500 /unlink in 34.62 seconds: 129.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 764 (time 1789450193.59 total 10.00 last 76.37) - open/close 1495 (time 1789450203.59 total 20.01 last 73.07) - open/close 2265 (time 1789450213.60 total 30.02 last 76.93) - open/close 3021 (time 1789450223.61 total 40.03 last 75.52) - open/close 3729 (time 1789450233.61 total 50.03 last 70.80) - open/close 4452 (time 1789450243.61 total 60.03 last 72.30) total: 4500 open/close in 60.70 seconds: 74.13 ops/second - /unlink 1415 (time 1789450258.16 total 10.00 last 141.43) - /unlink 2767 (time 1789450268.16 total 20.01 last 135.12) - /unlink 4158 (time 1789450278.16 total 30.01 last 139.10) total: 4500 /unlink in 32.66 seconds: 137.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 747 (time 1789450294.58 total 10.01 last 74.60) - open/close 1524 (time 1789450304.58 total 20.02 last 77.65) - open/close 2384 (time 1789450314.59 total 30.03 last 85.93) - open/close 3126 (time 1789450324.60 total 40.04 last 74.13) - open/close 3912 (time 1789450334.60 total 50.04 last 78.58) total: 4500 open/close in 57.78 seconds: 77.88 ops/second - /unlink 1300 (time 1789450356.26 total 10.00 last 129.94) - /unlink 2545 (time 1789450366.26 total 20.01 last 124.48) - /unlink 3804 (time 1789450376.27 total 30.01 last 125.81) total: 4500 /unlink in 34.98 seconds: 128.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 745 (time 1789450396.16 total 10.02 last 74.37) - open/close 1461 (time 1789450406.16 total 20.02 last 71.56) - open/close 2244 (time 1789450416.17 total 30.03 last 78.26) - open/close 2990 (time 1789450426.17 total 40.03 last 74.58) - open/close 3722 (time 1789450436.17 total 50.03 last 73.19) - open/close 4432 (time 1789450446.19 total 60.04 last 70.92) total: 4500 open/close in 60.87 seconds: 73.93 ops/second - /unlink 1447 (time 1789450460.81 total 10.00 last 144.69) - /unlink 2733 (time 1789450470.81 total 20.00 last 128.59) - /unlink 4149 (time 1789450480.82 total 30.01 last 141.55) total: 4500 /unlink in 33.10 seconds: 135.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 768 (time 1789450497.96 total 10.01 last 76.72) - open/close 1461 (time 1789450507.96 total 20.01 last 69.30) - open/close 2144 (time 1789450517.97 total 30.02 last 68.24) - open/close 2888 (time 1789450527.97 total 40.02 last 74.40) - open/close 3611 (time 1789450537.97 total 50.02 last 72.27) - open/close 4343 (time 1789450547.98 total 60.03 last 73.17) total: 4500 open/close in 61.89 seconds: 72.71 ops/second - /unlink 1402 (time 1789450564.71 total 10.00 last 140.19) - /unlink 2815 (time 1789450574.71 total 20.00 last 141.28) - /unlink 4210 (time 1789450584.71 total 30.01 last 139.46) total: 4500 /unlink in 32.14 seconds: 140.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 773 (time 1789450601.19 total 10.00 last 77.30) - open/close 1501 (time 1789450611.20 total 20.01 last 72.73) - open/close 2310 (time 1789450621.21 total 30.02 last 80.84) - open/close 3091 (time 1789450631.22 total 40.03 last 78.02) - open/close 3851 (time 1789450641.22 total 50.03 last 76.00) total: 4500 open/close in 58.31 seconds: 77.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1440 (time 1789450663.38 total 10.01 last 143.87) - /unlink 2773 (time 1789450673.38 total 20.01 last 133.29) - /unlink 4098 (time 1789450683.38 total 30.01 last 132.49) total: 4500 /unlink in 33.21 seconds: 135.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 750 (time 1789450702.12 total 10.01 last 74.94) - open/close 1294 (time 1789450712.13 total 20.02 last 54.33) - open/close 1905 (time 1789450722.15 total 30.04 last 61.02) - open/close 2541 (time 1789450732.17 total 40.06 last 63.46) - open/close 3226 (time 1789450742.17 total 50.06 last 68.49) - open/close 3945 (time 1789450752.19 total 60.08 last 71.76) total: 4500 open/close in 68.21 seconds: 65.97 ops/second - /unlink 1308 (time 1789450774.75 total 10.00 last 130.79) - /unlink 2596 (time 1789450784.75 total 20.00 last 128.75) - /unlink 4010 (time 1789450794.76 total 30.01 last 141.36) total: 4500 /unlink in 33.89 seconds: 132.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 712 (time 1789450813.44 total 10.02 last 71.03) - open/close 1418 (time 1789450823.44 total 20.03 last 70.58) - open/close 2112 (time 1789450833.45 total 30.03 last 69.39) - open/close 2840 (time 1789450843.45 total 40.04 last 72.74) - open/close 3572 (time 1789450853.46 total 50.04 last 73.16) - open/close 4286 (time 1789450863.47 total 60.06 last 71.30) total: 4500 open/close in 62.90 seconds: 71.54 ops/second - /unlink 1356 (time 1789450880.79 total 10.01 last 135.51) - /unlink 2784 (time 1789450890.79 total 20.01 last 142.79) - /unlink 4161 (time 1789450900.79 total 30.01 last 137.67) total: 4500 /unlink in 32.36 seconds: 139.08 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95334 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg636-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg636-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server stop mds service on oleg636-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg636-server stop mds service on oleg636-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg636-server unloading modules via unload_modules_local on: 'oleg636-server' oleg636-server: oleg636-server.virtnet: executing unload_modules_local oleg636-server: kmemleak disabled modules unloaded.