== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:26:18 (1788441978) Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg456-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg456-server oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: kmemleak disabled oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: MODOPTS_LIBCFS= oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg456-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg456-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg456-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 605 (time 1788442204.38 total 10.01 last 60.45) - open/close 1190 (time 1788442214.39 total 20.01 last 58.48) - open/close 1838 (time 1788442224.39 total 30.02 last 64.77) - open/close 2491 (time 1788442234.40 total 40.02 last 65.27) - open/close 3177 (time 1788442244.41 total 50.03 last 68.52) - open/close 3806 (time 1788442254.41 total 60.04 last 62.87) - open/close 4455 (time 1788442264.41 total 70.04 last 64.89) total: 4500 open/close in 70.95 seconds: 63.43 ops/second - /unlink 1212 (time 1788442280.57 total 10.01 last 121.07) - /unlink 2315 (time 1788442290.57 total 20.01 last 110.26) - /unlink 3376 (time 1788442300.58 total 30.02 last 106.07) total: 4500 /unlink in 39.60 seconds: 113.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 620 (time 1788442325.47 total 10.00 last 61.99) - open/close 1311 (time 1788442335.47 total 20.01 last 69.06) - open/close 1878 (time 1788442345.49 total 30.02 last 56.61) - open/close 2422 (time 1788442355.50 total 40.03 last 54.36) - open/close 2960 (time 1788442365.50 total 50.04 last 53.76) - open/close 3556 (time 1788442375.50 total 60.04 last 59.60) - open/close 4173 (time 1788442385.51 total 70.05 last 61.64) total: 4500 open/close in 75.42 seconds: 59.67 ops/second - /unlink 1194 (time 1788442405.37 total 10.01 last 119.29) - /unlink 2334 (time 1788442415.38 total 20.01 last 113.95) - /unlink 3521 (time 1788442425.39 total 30.03 last 118.57) total: 4500 /unlink in 38.50 seconds: 116.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 667 (time 1788442449.75 total 10.00 last 66.68) - open/close 1267 (time 1788442459.76 total 20.01 last 59.97) - open/close 1884 (time 1788442469.78 total 30.02 last 61.60) - open/close 2517 (time 1788442479.78 total 40.03 last 63.24) - open/close 3175 (time 1788442489.78 total 50.03 last 65.79) - open/close 3822 (time 1788442499.81 total 60.06 last 64.51) - open/close 4496 (time 1788442509.83 total 70.07 last 67.31) total: 4500 open/close in 70.11 seconds: 64.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1181 (time 1788442524.56 total 10.00 last 118.05) - /unlink 2440 (time 1788442534.57 total 20.01 last 125.86) - /unlink 3743 (time 1788442544.57 total 30.01 last 130.29) total: 4500 /unlink in 37.13 seconds: 121.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 627 (time 1788442566.60 total 10.02 last 62.61) - open/close 1222 (time 1788442576.62 total 20.03 last 59.38) - open/close 1905 (time 1788442586.63 total 30.04 last 68.24) - open/close 2569 (time 1788442596.63 total 40.05 last 66.39) - open/close 3228 (time 1788442606.64 total 50.05 last 65.86) - open/close 3890 (time 1788442616.64 total 60.05 last 66.20) total: 4500 open/close in 68.88 seconds: 65.33 ops/second - /unlink 1169 (time 1788442639.83 total 10.01 last 116.76) - /unlink 2303 (time 1788442649.84 total 20.02 last 113.37) - /unlink 3517 (time 1788442659.84 total 30.02 last 121.31) total: 4500 /unlink in 37.74 seconds: 119.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 672 (time 1788442683.53 total 10.01 last 67.15) - open/close 1290 (time 1788442693.57 total 20.04 last 61.59) - open/close 1927 (time 1788442703.57 total 30.05 last 63.65) - open/close 2523 (time 1788442713.57 total 40.05 last 59.60) - open/close 3106 (time 1788442723.58 total 50.05 last 58.29) - open/close 3758 (time 1788442733.59 total 60.06 last 65.11) - open/close 4453 (time 1788442743.60 total 70.07 last 69.43) total: 4500 open/close in 70.89 seconds: 63.48 ops/second - /unlink 1109 (time 1788442759.48 total 10.01 last 110.74) - /unlink 2220 (time 1788442769.49 total 20.02 last 111.01) - /unlink 3375 (time 1788442779.49 total 30.03 last 115.44) total: 4500 /unlink in 39.81 seconds: 113.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 625 (time 1788442804.91 total 10.01 last 62.41) - open/close 1186 (time 1788442814.93 total 20.03 last 56.02) - open/close 1789 (time 1788442824.94 total 30.04 last 60.23) - open/close 2420 (time 1788442834.94 total 40.04 last 63.08) - open/close 3084 (time 1788442844.95 total 50.06 last 66.31) - open/close 3734 (time 1788442854.96 total 60.06 last 64.98) - open/close 4419 (time 1788442864.96 total 70.07 last 68.45) total: 4500 open/close in 71.16 seconds: 63.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1204 (time 1788442881.00 total 10.00 last 120.39) - /unlink 2324 (time 1788442891.01 total 20.01 last 111.88) - /unlink 3576 (time 1788442901.01 total 30.02 last 125.13) total: 4500 /unlink in 38.93 seconds: 115.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 575 (time 1788442925.18 total 10.00 last 57.48) - open/close 1162 (time 1788442935.18 total 20.01 last 58.68) - open/close 1712 (time 1788442945.19 total 30.01 last 54.97) - open/close 2360 (time 1788442955.19 total 40.02 last 64.77) - open/close 2984 (time 1788442965.21 total 50.04 last 62.28) - open/close 3584 (time 1788442975.22 total 60.04 last 59.96) - open/close 4243 (time 1788442985.22 total 70.05 last 65.87) total: 4500 open/close in 74.35 seconds: 60.52 ops/second - /unlink 1114 (time 1788443004.43 total 10.00 last 111.36) - /unlink 2262 (time 1788443014.43 total 20.00 last 114.79) - /unlink 3465 (time 1788443024.44 total 30.02 last 120.16) total: 4500 /unlink in 38.90 seconds: 115.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 651 (time 1788443049.08 total 10.01 last 65.01) - open/close 1328 (time 1788443059.09 total 20.02 last 67.67) - open/close 2002 (time 1788443069.09 total 30.02 last 67.38) - open/close 2569 (time 1788443079.10 total 40.03 last 56.65) - open/close 3151 (time 1788443089.12 total 50.05 last 58.11) - open/close 3737 (time 1788443099.14 total 60.06 last 58.49) - open/close 4392 (time 1788443109.15 total 70.07 last 65.44) total: 4500 open/close in 71.85 seconds: 62.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 993 (time 1788443125.84 total 10.00 last 99.29) - /unlink 1977 (time 1788443135.84 total 20.00 last 98.38) - /unlink 3066 (time 1788443145.86 total 30.02 last 108.76) - /unlink 4132 (time 1788443155.87 total 40.02 last 106.51) total: 4500 /unlink in 43.16 seconds: 104.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 496 (time 1788443175.76 total 10.02 last 49.52) - open/close 1090 (time 1788443185.77 total 20.03 last 59.34) - open/close 1721 (time 1788443195.78 total 30.03 last 63.05) - open/close 2275 (time 1788443205.78 total 40.03 last 55.40) - open/close 2823 (time 1788443215.79 total 50.05 last 54.72) - open/close 3434 (time 1788443225.79 total 60.05 last 61.10) - open/close 4079 (time 1788443235.80 total 70.06 last 64.44) total: 4500 open/close in 76.80 seconds: 58.59 ops/second - /unlink 1139 (time 1788443256.67 total 10.00 last 113.89) - /unlink 2049 (time 1788443266.67 total 20.00 last 90.97) - /unlink 3110 (time 1788443276.67 total 30.01 last 106.08) - /unlink 4224 (time 1788443286.68 total 40.01 last 111.39) total: 4500 /unlink in 42.64 seconds: 105.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 558 (time 1788443305.65 total 10.00 last 55.78) - open/close 1086 (time 1788443315.66 total 20.01 last 52.74) - open/close 1595 (time 1788443325.68 total 30.03 last 50.81) - open/close 2038 (time 1788443335.68 total 40.04 last 44.28) - open/close 2602 (time 1788443345.69 total 50.04 last 56.36) - open/close 3237 (time 1788443355.71 total 60.07 last 63.35) - open/close 3799 (time 1788443365.72 total 70.08 last 56.14) - open/close 4431 (time 1788443375.73 total 80.09 last 63.15) total: 4500 open/close in 81.04 seconds: 55.53 ops/second - /unlink 1158 (time 1788443392.03 total 10.01 last 115.74) - /unlink 2309 (time 1788443402.03 total 20.01 last 115.05) - /unlink 3473 (time 1788443412.04 total 30.02 last 116.32) total: 4500 /unlink in 38.66 seconds: 116.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 608 (time 1788443437.19 total 10.00 last 60.77) - open/close 1135 (time 1788443447.20 total 20.01 last 52.68) - open/close 1730 (time 1788443457.20 total 30.02 last 59.45) - open/close 2433 (time 1788443467.21 total 40.02 last 70.28) - open/close 3086 (time 1788443477.21 total 50.03 last 65.27) - open/close 3729 (time 1788443487.22 total 60.04 last 64.22) - open/close 4367 (time 1788443497.24 total 70.05 last 63.73) total: 4500 open/close in 72.10 seconds: 62.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1139 (time 1788443514.66 total 10.01 last 113.78) - /unlink 2393 (time 1788443524.66 total 20.01 last 125.37) - /unlink 3616 (time 1788443534.66 total 30.02 last 122.25) total: 4500 /unlink in 37.96 seconds: 118.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 747 (time 1788443558.29 total 10.00 last 74.66) - open/close 1395 (time 1788443568.29 total 20.01 last 64.79) - open/close 2038 (time 1788443578.31 total 30.03 last 64.17) - open/close 2770 (time 1788443588.33 total 40.05 last 73.03) - open/close 3429 (time 1788443598.35 total 50.07 last 65.81) - open/close 4076 (time 1788443608.35 total 60.07 last 64.66) total: 4500 open/close in 66.10 seconds: 68.08 ops/second - /unlink 1133 (time 1788443629.42 total 10.00 last 113.29) - /unlink 2257 (time 1788443639.42 total 20.00 last 112.39) - /unlink 3474 (time 1788443649.43 total 30.01 last 121.64) total: 4500 /unlink in 38.31 seconds: 117.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 637 (time 1788443673.37 total 10.01 last 63.65) - open/close 1301 (time 1788443683.38 total 20.02 last 66.31) - open/close 1922 (time 1788443693.38 total 30.02 last 62.07) - open/close 2484 (time 1788443703.39 total 40.03 last 56.14) - open/close 3110 (time 1788443713.43 total 50.07 last 62.39) - open/close 3683 (time 1788443723.43 total 60.07 last 57.30) - open/close 4308 (time 1788443733.43 total 70.08 last 62.46) total: 4500 open/close in 73.27 seconds: 61.42 ops/second - /unlink 1165 (time 1788443751.60 total 10.02 last 116.32) - /unlink 2294 (time 1788443761.60 total 20.02 last 112.85) - /unlink 3467 (time 1788443771.61 total 30.03 last 117.23) total: 4500 /unlink in 39.18 seconds: 114.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 660 (time 1788443795.48 total 10.02 last 65.85) - open/close 1345 (time 1788443805.49 total 20.03 last 68.47) - open/close 2034 (time 1788443815.51 total 30.05 last 68.76) - open/close 2696 (time 1788443825.52 total 40.06 last 66.10) - open/close 3389 (time 1788443835.52 total 50.06 last 69.30) - open/close 4020 (time 1788443845.52 total 60.07 last 63.09) total: 4500 open/close in 67.09 seconds: 67.08 ops/second - /unlink 1094 (time 1788443867.55 total 10.01 last 109.33) - /unlink 2217 (time 1788443877.56 total 20.01 last 112.21) - /unlink 3358 (time 1788443887.56 total 30.01 last 114.10) - /unlink 4454 (time 1788443897.56 total 40.02 last 109.57) total: 4500 /unlink in 40.55 seconds: 110.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 475 (time 1788443914.31 total 10.01 last 47.48) - open/close 1054 (time 1788443924.32 total 20.01 last 57.85) - open/close 1711 (time 1788443934.32 total 30.01 last 65.70) - open/close 2350 (time 1788443944.32 total 40.02 last 63.89) - open/close 2992 (time 1788443954.34 total 50.03 last 64.11) - open/close 3664 (time 1788443964.34 total 60.04 last 67.16) - open/close 4272 (time 1788443974.35 total 70.04 last 60.75) total: 4500 open/close in 73.60 seconds: 61.14 ops/second - /unlink 1145 (time 1788443992.81 total 10.00 last 114.50) - /unlink 2269 (time 1788444002.82 total 20.01 last 112.34) - /unlink 3470 (time 1788444012.82 total 30.01 last 120.02) total: 4500 /unlink in 38.91 seconds: 115.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 749 (time 1788444036.72 total 10.01 last 74.86) - open/close 1349 (time 1788444046.75 total 20.03 last 59.82) - open/close 1944 (time 1788444056.76 total 30.05 last 59.42) - open/close 2629 (time 1788444066.77 total 40.05 last 68.47) - open/close 3105 (time 1788444076.79 total 50.07 last 47.50) - open/close 3547 (time 1788444086.80 total 60.09 last 44.13) - open/close 4172 (time 1788444096.81 total 70.09 last 62.48) total: 4500 open/close in 75.30 seconds: 59.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1052 (time 1788444117.85 total 10.01 last 105.13) - /unlink 2367 (time 1788444127.85 total 20.01 last 131.45) - /unlink 3589 (time 1788444137.85 total 30.01 last 122.16) total: 4500 /unlink in 36.48 seconds: 123.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 655 (time 1788444158.93 total 10.01 last 65.43) - open/close 1330 (time 1788444168.94 total 20.02 last 67.42) - open/close 1897 (time 1788444178.95 total 30.04 last 56.63) - open/close 2431 (time 1788444188.95 total 40.04 last 53.40) - open/close 2996 (time 1788444198.97 total 50.05 last 56.41) - open/close 3492 (time 1788444208.98 total 60.07 last 49.54) - open/close 4121 (time 1788444218.98 total 70.07 last 62.89) total: 4500 open/close in 75.58 seconds: 59.54 ops/second - /unlink 1264 (time 1788444239.35 total 10.01 last 126.30) - /unlink 2445 (time 1788444249.36 total 20.01 last 118.03) - /unlink 3560 (time 1788444259.38 total 30.03 last 111.30) total: 4500 /unlink in 37.82 seconds: 119.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 671 (time 1788444283.85 total 10.02 last 66.99) - open/close 1308 (time 1788444293.85 total 20.02 last 63.69) - open/close 1964 (time 1788444303.85 total 30.02 last 65.58) - open/close 2553 (time 1788444313.85 total 40.02 last 58.89) - open/close 3156 (time 1788444323.85 total 50.02 last 60.29) - open/close 3733 (time 1788444333.85 total 60.03 last 57.68) - open/close 4443 (time 1788444343.86 total 70.03 last 70.95) total: 4500 open/close in 71.04 seconds: 63.35 ops/second - /unlink 1063 (time 1788444360.26 total 10.00 last 106.27) - /unlink 1803 (time 1788444370.26 total 20.00 last 73.99) - /unlink 2871 (time 1788444380.26 total 30.01 last 106.78) - /unlink 3907 (time 1788444390.27 total 40.01 last 103.55) total: 4500 /unlink in 44.95 seconds: 100.12 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95321 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg456-server unloading modules via unload_modules_local on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local oleg456-server: kmemleak disabled modules unloaded.