== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:47:19 (1790743639) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg331-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg331-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 648 (time 1790743865.90 total 10.00 last 64.78) - open/close 1423 (time 1790743875.92 total 20.02 last 77.35) - open/close 2102 (time 1790743885.93 total 30.03 last 67.83) - open/close 2731 (time 1790743895.94 total 40.04 last 62.86) - open/close 3401 (time 1790743905.94 total 50.04 last 66.97) - open/close 3983 (time 1790743915.95 total 60.04 last 58.18) total: 4500 open/close in 69.40 seconds: 64.84 ops/second - /unlink 1146 (time 1790743941.14 total 10.01 last 114.48) - /unlink 2316 (time 1790743951.14 total 20.01 last 117.00) - /unlink 3603 (time 1790743961.14 total 30.01 last 128.68) total: 4500 /unlink in 36.63 seconds: 122.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 737 (time 1790743982.97 total 10.00 last 73.68) - open/close 1358 (time 1790743992.98 total 20.01 last 62.07) - open/close 2121 (time 1790744002.98 total 30.01 last 76.26) - open/close 2796 (time 1790744012.99 total 40.02 last 67.44) - open/close 3508 (time 1790744022.99 total 50.02 last 71.20) - open/close 4105 (time 1790744033.00 total 60.03 last 59.64) total: 4500 open/close in 66.84 seconds: 67.33 ops/second - /unlink 1206 (time 1790744054.72 total 10.02 last 120.42) - /unlink 2389 (time 1790744064.73 total 20.02 last 118.21) - /unlink 3715 (time 1790744074.74 total 30.03 last 132.46) total: 4500 /unlink in 37.57 seconds: 119.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 585 (time 1790744099.14 total 10.02 last 58.39) - open/close 1101 (time 1790744109.15 total 20.02 last 51.58) - open/close 1708 (time 1790744119.15 total 30.03 last 60.67) - open/close 2366 (time 1790744129.15 total 40.03 last 65.78) - open/close 2975 (time 1790744139.18 total 50.05 last 60.76) - open/close 3563 (time 1790744149.19 total 60.06 last 58.73) - open/close 4155 (time 1790744159.20 total 70.07 last 59.16) total: 4500 open/close in 75.65 seconds: 59.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1328 (time 1790744179.60 total 10.00 last 132.77) - /unlink 2625 (time 1790744189.60 total 20.00 last 129.70) - /unlink 3964 (time 1790744199.61 total 30.01 last 133.82) total: 4500 /unlink in 34.75 seconds: 129.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 708 (time 1790744219.49 total 10.00 last 70.79) - open/close 1356 (time 1790744229.50 total 20.01 last 64.72) - open/close 2007 (time 1790744239.50 total 30.02 last 65.07) - open/close 2682 (time 1790744249.51 total 40.02 last 67.47) - open/close 3369 (time 1790744259.52 total 50.04 last 68.62) - open/close 4031 (time 1790744269.52 total 60.04 last 66.19) total: 4500 open/close in 67.04 seconds: 67.13 ops/second - /unlink 1179 (time 1790744291.72 total 10.00 last 117.84) - /unlink 2353 (time 1790744301.74 total 20.02 last 117.26) - /unlink 3607 (time 1790744311.74 total 30.02 last 125.36) total: 4500 /unlink in 39.18 seconds: 114.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 679 (time 1790744336.00 total 10.01 last 67.81) - open/close 1402 (time 1790744346.01 total 20.02 last 72.27) - open/close 2090 (time 1790744356.03 total 30.04 last 68.66) - open/close 2768 (time 1790744366.03 total 40.04 last 67.79) - open/close 3435 (time 1790744376.03 total 50.04 last 66.67) - open/close 4098 (time 1790744386.06 total 60.07 last 66.11) total: 4500 open/close in 67.37 seconds: 66.80 ops/second - /unlink 1165 (time 1790744410.14 total 10.03 last 116.15) - /unlink 2334 (time 1790744420.16 total 20.04 last 116.77) - /unlink 3514 (time 1790744430.16 total 30.04 last 117.96) total: 4500 /unlink in 37.70 seconds: 119.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 624 (time 1790744452.76 total 10.03 last 62.24) - open/close 1223 (time 1790744462.77 total 20.03 last 59.87) - open/close 1794 (time 1790744472.77 total 30.03 last 57.10) - open/close 2458 (time 1790744482.78 total 40.04 last 66.34) - open/close 3084 (time 1790744492.79 total 50.05 last 62.55) - open/close 3678 (time 1790744502.79 total 60.06 last 59.36) - open/close 4332 (time 1790744512.81 total 70.07 last 65.28) total: 4500 open/close in 72.58 seconds: 62.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1191 (time 1790744530.29 total 10.00 last 119.05) - /unlink 2469 (time 1790744540.29 total 20.00 last 127.80) - /unlink 3758 (time 1790744550.30 total 30.01 last 128.80) total: 4500 /unlink in 36.12 seconds: 124.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 671 (time 1790744570.42 total 10.01 last 67.05) - open/close 1281 (time 1790744580.43 total 20.01 last 60.95) - open/close 1955 (time 1790744590.44 total 30.03 last 67.32) - open/close 2597 (time 1790744600.45 total 40.03 last 64.16) - open/close 3250 (time 1790744610.48 total 50.06 last 65.13) - open/close 3993 (time 1790744620.48 total 60.07 last 74.26) total: 4500 open/close in 67.14 seconds: 67.02 ops/second - /unlink 1288 (time 1790744642.15 total 10.00 last 128.76) - /unlink 2501 (time 1790744652.15 total 20.01 last 121.28) - /unlink 3894 (time 1790744662.16 total 30.01 last 139.27) total: 4500 /unlink in 34.64 seconds: 129.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 726 (time 1790744681.34 total 10.00 last 72.57) - open/close 1446 (time 1790744691.37 total 20.03 last 71.81) - open/close 2173 (time 1790744701.38 total 30.05 last 72.56) - open/close 2873 (time 1790744711.39 total 40.05 last 69.99) - open/close 3578 (time 1790744721.39 total 50.05 last 70.49) - open/close 4248 (time 1790744731.40 total 60.07 last 66.90) total: 4500 open/close in 63.56 seconds: 70.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1325 (time 1790744750.37 total 10.01 last 132.41) - /unlink 2644 (time 1790744760.37 total 20.02 last 131.78) - /unlink 3891 (time 1790744770.38 total 30.02 last 124.66) total: 4500 /unlink in 34.85 seconds: 129.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 611 (time 1790744790.58 total 10.02 last 60.97) - open/close 1185 (time 1790744800.59 total 20.03 last 57.33) - open/close 1779 (time 1790744810.59 total 30.04 last 59.36) - open/close 2264 (time 1790744820.60 total 40.05 last 48.46) - open/close 2786 (time 1790744830.63 total 50.07 last 52.06) - open/close 3380 (time 1790744840.64 total 60.09 last 59.31) - open/close 4065 (time 1790744850.65 total 70.10 last 68.44) total: 4500 open/close in 76.90 seconds: 58.52 ops/second - /unlink 1267 (time 1790744872.08 total 10.00 last 126.66) - /unlink 2448 (time 1790744882.08 total 20.01 last 118.08) - /unlink 3617 (time 1790744892.09 total 30.01 last 116.83) total: 4500 /unlink in 37.55 seconds: 119.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 714 (time 1790744915.00 total 10.01 last 71.31) - open/close 1297 (time 1790744925.00 total 20.02 last 58.28) - open/close 2026 (time 1790744935.01 total 30.03 last 72.83) - open/close 2719 (time 1790744945.02 total 40.04 last 69.21) - open/close 3371 (time 1790744955.03 total 50.05 last 65.15) - open/close 4053 (time 1790744965.05 total 60.06 last 68.08) total: 4500 open/close in 66.82 seconds: 67.34 ops/second - /unlink 1155 (time 1790744986.93 total 10.00 last 115.49) - /unlink 2267 (time 1790744996.94 total 20.02 last 111.02) - /unlink 3366 (time 1790745006.95 total 30.02 last 109.83) total: 4500 /unlink in 38.90 seconds: 115.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 683 (time 1790745030.33 total 10.01 last 68.23) - open/close 1397 (time 1790745040.34 total 20.01 last 71.39) - open/close 2051 (time 1790745050.34 total 30.01 last 65.40) - open/close 2779 (time 1790745060.34 total 40.02 last 72.78) - open/close 3411 (time 1790745070.34 total 50.02 last 63.18) - open/close 4100 (time 1790745080.35 total 60.02 last 68.88) total: 4500 open/close in 66.28 seconds: 67.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1132 (time 1790745101.56 total 10.01 last 113.11) - /unlink 2445 (time 1790745111.56 total 20.01 last 131.27) - /unlink 3612 (time 1790745121.56 total 30.01 last 116.65) total: 4500 /unlink in 36.74 seconds: 122.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 628 (time 1790745143.78 total 10.01 last 62.72) - open/close 1326 (time 1790745153.79 total 20.02 last 69.75) - open/close 1931 (time 1790745163.79 total 30.03 last 60.45) - open/close 2535 (time 1790745173.79 total 40.03 last 60.40) - open/close 3152 (time 1790745183.80 total 50.03 last 61.66) - open/close 3804 (time 1790745193.80 total 60.04 last 65.19) - open/close 4497 (time 1790745203.83 total 70.06 last 69.13) total: 4500 open/close in 70.08 seconds: 64.21 ops/second - /unlink 1286 (time 1790745218.08 total 10.00 last 128.58) - /unlink 2548 (time 1790745228.10 total 20.02 last 125.97) - /unlink 3841 (time 1790745238.11 total 30.02 last 129.25) total: 4500 /unlink in 35.27 seconds: 127.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 707 (time 1790745258.03 total 10.01 last 70.64) - open/close 1407 (time 1790745268.04 total 20.02 last 69.91) - open/close 2097 (time 1790745278.05 total 30.03 last 68.96) - open/close 2728 (time 1790745288.05 total 40.03 last 63.07) - open/close 3371 (time 1790745298.07 total 50.04 last 64.22) - open/close 4094 (time 1790745308.08 total 60.05 last 72.23) total: 4500 open/close in 66.20 seconds: 67.98 ops/second - /unlink 1300 (time 1790745329.62 total 10.01 last 129.92) - /unlink 2534 (time 1790745339.63 total 20.01 last 123.35) - /unlink 3767 (time 1790745349.64 total 30.02 last 123.17) total: 4500 /unlink in 36.54 seconds: 123.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 622 (time 1790745371.10 total 10.02 last 62.06) - open/close 1249 (time 1790745381.12 total 20.04 last 62.60) - open/close 1885 (time 1790745391.13 total 30.05 last 63.51) - open/close 2571 (time 1790745401.14 total 40.06 last 68.52) - open/close 3312 (time 1790745411.15 total 50.07 last 74.08) - open/close 4036 (time 1790745421.16 total 60.09 last 72.26) total: 4500 open/close in 66.62 seconds: 67.55 ops/second - /unlink 1343 (time 1790745441.96 total 10.01 last 134.23) - /unlink 2490 (time 1790745451.97 total 20.01 last 114.64) - /unlink 3746 (time 1790745461.97 total 30.01 last 125.60) total: 4500 /unlink in 36.55 seconds: 123.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 704 (time 1790745484.04 total 10.02 last 70.26) - open/close 1416 (time 1790745494.06 total 20.04 last 71.04) - open/close 1979 (time 1790745504.07 total 30.05 last 56.23) - open/close 2674 (time 1790745514.08 total 40.06 last 69.45) - open/close 3370 (time 1790745524.09 total 50.07 last 69.54) - open/close 3965 (time 1790745534.10 total 60.08 last 59.42) total: 4500 open/close in 67.77 seconds: 66.40 ops/second - /unlink 1285 (time 1790745556.37 total 10.01 last 128.40) - /unlink 2614 (time 1790745566.38 total 20.01 last 132.83) - /unlink 3914 (time 1790745576.39 total 30.02 last 129.85) total: 4500 /unlink in 34.63 seconds: 129.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 736 (time 1790745596.53 total 10.01 last 73.54) - open/close 1340 (time 1790745606.53 total 20.01 last 60.39) - open/close 2007 (time 1790745616.53 total 30.01 last 66.70) - open/close 2669 (time 1790745626.55 total 40.03 last 66.08) - open/close 3404 (time 1790745636.56 total 50.04 last 73.44) - open/close 4056 (time 1790745646.57 total 60.05 last 65.14) total: 4500 open/close in 66.70 seconds: 67.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1264 (time 1790745668.18 total 10.00 last 126.35) - /unlink 2522 (time 1790745678.18 total 20.01 last 125.77) - /unlink 3672 (time 1790745688.19 total 30.01 last 114.97) total: 4500 /unlink in 36.58 seconds: 123.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 623 (time 1790745709.56 total 10.01 last 62.24) - open/close 1275 (time 1790745719.57 total 20.02 last 65.15) - open/close 1967 (time 1790745729.57 total 30.03 last 69.15) - open/close 2655 (time 1790745739.58 total 40.03 last 68.74) - open/close 3317 (time 1790745749.59 total 50.04 last 66.16) - open/close 3981 (time 1790745759.60 total 60.05 last 66.35) total: 4500 open/close in 67.98 seconds: 66.20 ops/second - /unlink 1180 (time 1790745783.04 total 10.00 last 117.98) - /unlink 2477 (time 1790745793.04 total 20.00 last 129.68) - /unlink 3647 (time 1790745803.04 total 30.00 last 116.99) total: 4500 /unlink in 36.80 seconds: 122.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 676 (time 1790745825.11 total 10.01 last 67.55) - open/close 1382 (time 1790745835.12 total 20.01 last 70.56) - open/close 2076 (time 1790745845.12 total 30.02 last 69.37) - open/close 2790 (time 1790745855.13 total 40.02 last 71.37) - open/close 3490 (time 1790745865.13 total 50.03 last 69.96) - open/close 4227 (time 1790745875.14 total 60.04 last 73.62) total: 4500 open/close in 63.74 seconds: 70.60 ops/second - /unlink 1287 (time 1790745893.06 total 10.00 last 128.66) - /unlink 2606 (time 1790745903.06 total 20.01 last 131.86) - /unlink 3875 (time 1790745913.07 total 30.01 last 126.83) total: 4500 /unlink in 34.87 seconds: 129.04 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11358: 95550 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded.