== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:23:06 (1788441786) Stopping clients: oleg608-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg608-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg608-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg608-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg608-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg608-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg608-server oleg608-server: oleg608-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg608-server' oleg608-server: oleg608-server.virtnet: executing load_modules_local oleg608-server: kmemleak disabled oleg608-server: Loading modules from /home/green/git/lustre-release/lustre oleg608-server: detected 4 online CPUs by sysfs oleg608-server: MODOPTS_LIBCFS= oleg608-server: Force libcfs to create 2 CPU partitions oleg608-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg608-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg608-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg608-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg608-server: oleg608-server.virtnet: executing set_default_debug -1 all pdsh@oleg608-client: oleg608-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg608-client.virtnet: -o user_xattr,flock 192.168.206.108@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 732 (time 1788441972.65 total 10.00 last 73.18) - open/close 1659 (time 1788441982.66 total 20.01 last 92.61) - open/close 2570 (time 1788441992.66 total 30.01 last 91.09) - open/close 3410 (time 1788442002.67 total 40.02 last 83.91) - open/close 4294 (time 1788442012.68 total 50.03 last 88.35) total: 4500 open/close in 52.31 seconds: 86.02 ops/second - /unlink 1661 (time 1788442028.76 total 10.00 last 166.03) - /unlink 3234 (time 1788442038.76 total 20.01 last 157.26) total: 4500 /unlink in 27.69 seconds: 162.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 890 (time 1788442060.70 total 10.01 last 88.94) - open/close 1636 (time 1788442070.70 total 20.01 last 74.54) - open/close 2547 (time 1788442080.71 total 30.02 last 91.09) - open/close 3380 (time 1788442090.71 total 40.02 last 83.28) - open/close 4248 (time 1788442100.71 total 50.02 last 86.74) total: 4500 open/close in 52.75 seconds: 85.31 ops/second - /unlink 1820 (time 1788442117.41 total 10.00 last 181.96) - /unlink 3351 (time 1788442127.41 total 20.00 last 153.08) total: 4500 /unlink in 27.43 seconds: 164.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 772 (time 1788442149.75 total 10.01 last 77.15) - open/close 1561 (time 1788442159.76 total 20.01 last 78.85) - open/close 2412 (time 1788442169.77 total 30.02 last 85.04) - open/close 3344 (time 1788442179.78 total 40.03 last 93.13) - open/close 4135 (time 1788442189.79 total 50.04 last 79.02) total: 4500 open/close in 54.46 seconds: 82.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1713 (time 1788442208.14 total 10.00 last 171.24) - /unlink 3299 (time 1788442218.14 total 20.00 last 158.60) total: 4500 /unlink in 27.45 seconds: 163.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 874 (time 1788442239.39 total 10.00 last 87.39) - open/close 1754 (time 1788442249.41 total 20.02 last 87.79) - open/close 2568 (time 1788442259.41 total 30.03 last 81.36) - open/close 3455 (time 1788442269.43 total 40.04 last 88.58) - open/close 4339 (time 1788442279.43 total 50.05 last 88.36) total: 4500 open/close in 51.88 seconds: 86.74 ops/second - /unlink 1680 (time 1788442294.62 total 10.00 last 167.97) - /unlink 3298 (time 1788442304.63 total 20.01 last 161.66) total: 4500 /unlink in 28.38 seconds: 158.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 840 (time 1788442327.35 total 10.00 last 83.99) - open/close 1754 (time 1788442337.35 total 20.00 last 91.37) - open/close 2607 (time 1788442347.35 total 30.01 last 85.28) - open/close 3354 (time 1788442357.36 total 40.02 last 74.63) - open/close 4173 (time 1788442367.37 total 50.02 last 81.86) total: 4500 open/close in 53.97 seconds: 83.38 ops/second - /unlink 1704 (time 1788442385.79 total 10.00 last 170.34) - /unlink 3264 (time 1788442395.79 total 20.00 last 155.99) total: 4500 /unlink in 28.40 seconds: 158.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 791 (time 1788442419.16 total 10.02 last 78.94) - open/close 1671 (time 1788442429.17 total 20.03 last 87.95) - open/close 2492 (time 1788442439.18 total 30.03 last 82.04) - open/close 3355 (time 1788442449.18 total 40.04 last 86.24) - open/close 4233 (time 1788442459.19 total 50.05 last 87.74) total: 4500 open/close in 53.00 seconds: 84.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1664 (time 1788442476.17 total 10.00 last 166.37) - /unlink 3263 (time 1788442486.17 total 20.00 last 159.86) total: 4500 /unlink in 27.63 seconds: 162.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 889 (time 1788442508.37 total 10.01 last 88.83) - open/close 1819 (time 1788442518.37 total 20.02 last 92.93) - open/close 2782 (time 1788442528.37 total 30.02 last 96.29) - open/close 3644 (time 1788442538.38 total 40.02 last 86.15) total: 4500 open/close in 49.60 seconds: 90.72 ops/second - /unlink 1705 (time 1788442562.52 total 10.01 last 170.41) - /unlink 3269 (time 1788442572.52 total 20.01 last 156.35) total: 4500 /unlink in 27.29 seconds: 164.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 905 (time 1788442594.27 total 10.01 last 90.43) - open/close 1848 (time 1788442604.27 total 20.01 last 94.27) - open/close 2746 (time 1788442614.28 total 30.02 last 89.74) - open/close 3633 (time 1788442624.28 total 40.02 last 88.65) total: 4500 open/close in 49.61 seconds: 90.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1507 (time 1788442648.33 total 10.00 last 150.65) - /unlink 2901 (time 1788442658.34 total 20.01 last 139.33) - /unlink 4344 (time 1788442668.35 total 30.02 last 144.08) total: 4500 /unlink in 31.25 seconds: 143.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 875 (time 1788442684.03 total 10.00 last 87.50) - open/close 1648 (time 1788442694.04 total 20.01 last 77.24) - open/close 2341 (time 1788442704.04 total 30.01 last 69.27) - open/close 3158 (time 1788442714.05 total 40.02 last 81.66) - open/close 3942 (time 1788442724.06 total 50.03 last 78.32) total: 4500 open/close in 56.92 seconds: 79.06 ops/second - /unlink 1513 (time 1788442745.47 total 10.00 last 151.27) - /unlink 3043 (time 1788442755.48 total 20.01 last 152.94) total: 4500 /unlink in 28.49 seconds: 157.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 845 (time 1788442778.49 total 10.01 last 84.40) - open/close 1670 (time 1788442788.49 total 20.01 last 82.48) - open/close 2486 (time 1788442798.50 total 30.02 last 81.52) - open/close 3327 (time 1788442808.51 total 40.03 last 84.07) - open/close 4111 (time 1788442818.55 total 50.07 last 78.10) total: 4500 open/close in 54.62 seconds: 82.39 ops/second - /unlink 1598 (time 1788442837.55 total 10.01 last 159.67) - /unlink 3276 (time 1788442847.55 total 20.01 last 167.77) total: 4500 /unlink in 27.76 seconds: 162.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 875 (time 1788442870.09 total 10.02 last 87.36) - open/close 1737 (time 1788442880.10 total 20.02 last 86.13) - open/close 2568 (time 1788442890.11 total 30.03 last 83.04) - open/close 3427 (time 1788442900.11 total 40.04 last 85.86) - open/close 4282 (time 1788442910.12 total 50.04 last 85.44) total: 4500 open/close in 52.47 seconds: 85.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1693 (time 1788442926.56 total 10.00 last 169.27) - /unlink 3339 (time 1788442936.56 total 20.01 last 164.48) total: 4500 /unlink in 27.86 seconds: 161.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 832 (time 1788442959.22 total 10.00 last 83.18) - open/close 1669 (time 1788442969.23 total 20.02 last 83.59) - open/close 2581 (time 1788442979.24 total 30.02 last 91.11) - open/close 3400 (time 1788442989.24 total 40.03 last 81.89) - open/close 4265 (time 1788442999.25 total 50.04 last 86.42) total: 4500 open/close in 52.51 seconds: 85.70 ops/second - /unlink 1764 (time 1788443015.07 total 10.01 last 176.23) - /unlink 3402 (time 1788443025.07 total 20.01 last 163.80) total: 4500 /unlink in 27.15 seconds: 165.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 817 (time 1788443046.79 total 10.00 last 81.67) - open/close 1783 (time 1788443056.80 total 20.01 last 96.49) - open/close 2580 (time 1788443066.81 total 30.02 last 79.68) - open/close 3482 (time 1788443076.81 total 40.02 last 90.17) - open/close 4315 (time 1788443086.81 total 50.02 last 83.27) total: 4500 open/close in 52.26 seconds: 86.10 ops/second - /unlink 1394 (time 1788443103.72 total 10.00 last 139.36) - /unlink 2833 (time 1788443113.73 total 20.01 last 143.82) - /unlink 4295 (time 1788443123.73 total 30.02 last 146.10) total: 4500 /unlink in 31.80 seconds: 141.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 785 (time 1788443140.74 total 10.01 last 78.45) - open/close 1473 (time 1788443150.75 total 20.02 last 68.73) - open/close 2370 (time 1788443160.75 total 30.02 last 89.67) - open/close 3241 (time 1788443170.76 total 40.03 last 87.05) - open/close 4108 (time 1788443180.77 total 50.04 last 86.62) total: 4500 open/close in 54.62 seconds: 82.39 ops/second - /unlink 1720 (time 1788443199.52 total 10.00 last 171.99) - /unlink 3350 (time 1788443209.53 total 20.01 last 162.92) total: 4500 /unlink in 26.96 seconds: 166.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 914 (time 1788443230.10 total 10.00 last 91.36) - open/close 1862 (time 1788443240.11 total 20.01 last 94.74) - open/close 2773 (time 1788443250.12 total 30.02 last 91.01) - open/close 3683 (time 1788443260.13 total 40.03 last 90.92) total: 4500 open/close in 49.09 seconds: 91.67 ops/second - /unlink 1719 (time 1788443283.37 total 10.01 last 171.76) - /unlink 3454 (time 1788443293.38 total 20.02 last 173.26) total: 4500 /unlink in 26.91 seconds: 167.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 911 (time 1788443314.53 total 10.01 last 91.05) - open/close 1769 (time 1788443324.54 total 20.01 last 85.74) - open/close 2714 (time 1788443334.55 total 30.02 last 94.43) - open/close 3604 (time 1788443344.55 total 40.03 last 88.95) - open/close 4402 (time 1788443354.56 total 50.03 last 79.76) total: 4500 open/close in 51.34 seconds: 87.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1610 (time 1788443370.01 total 10.00 last 160.94) - /unlink 3322 (time 1788443380.01 total 20.01 last 171.16) total: 4500 /unlink in 27.10 seconds: 166.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 969 (time 1788443401.99 total 10.01 last 96.79) - open/close 1649 (time 1788443412.00 total 20.02 last 67.93) - open/close 2515 (time 1788443422.00 total 30.03 last 86.54) - open/close 3402 (time 1788443432.01 total 40.03 last 88.68) - open/close 4240 (time 1788443442.01 total 50.03 last 83.79) total: 4500 open/close in 52.83 seconds: 85.18 ops/second - /unlink 1592 (time 1788443459.39 total 10.00 last 159.20) - /unlink 3225 (time 1788443469.40 total 20.01 last 163.19) total: 4500 /unlink in 28.01 seconds: 160.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 910 (time 1788443491.81 total 10.01 last 90.92) - open/close 1805 (time 1788443501.81 total 20.01 last 89.46) - open/close 2677 (time 1788443511.81 total 30.01 last 87.19) - open/close 3582 (time 1788443521.82 total 40.02 last 90.47) - open/close 4444 (time 1788443531.82 total 50.02 last 86.15) total: 4500 open/close in 50.61 seconds: 88.92 ops/second - /unlink 1779 (time 1788443546.66 total 10.00 last 177.87) - /unlink 3576 (time 1788443556.67 total 20.00 last 179.66) total: 4500 /unlink in 26.94 seconds: 167.06 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11351: 95168 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg608-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg608-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg608-server stop mds service on oleg608-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg608-server stop mds service on oleg608-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg608-server unloading modules via unload_modules_local on: 'oleg608-server' oleg608-server: oleg608-server.virtnet: executing unload_modules_local oleg608-server: kmemleak disabled modules unloaded.