== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:20:37 (1783966837) Stopping clients: oleg157-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg157-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg157-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg157-server oleg157-server: oleg157-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg157-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg157-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 631 (time 1783967052.52 total 10.00 last 63.08) - open/close 1388 (time 1783967062.52 total 20.01 last 75.67) - open/close 2121 (time 1783967072.53 total 30.02 last 73.24) - open/close 2825 (time 1783967082.53 total 40.02 last 70.38) - open/close 3519 (time 1783967092.57 total 50.06 last 69.14) - open/close 4216 (time 1783967102.58 total 60.07 last 69.63) total: 4500 open/close in 63.87 seconds: 70.46 ops/second - /unlink 1319 (time 1783967121.11 total 10.01 last 131.78) - /unlink 2648 (time 1783967131.12 total 20.02 last 132.81) - /unlink 3856 (time 1783967141.12 total 30.02 last 120.74) total: 4500 /unlink in 34.73 seconds: 129.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 663 (time 1783967160.07 total 10.01 last 66.22) - open/close 1381 (time 1783967170.07 total 20.02 last 71.77) - open/close 2065 (time 1783967180.08 total 30.02 last 68.35) - open/close 2761 (time 1783967190.09 total 40.03 last 69.56) - open/close 3409 (time 1783967200.11 total 50.05 last 64.67) - open/close 4083 (time 1783967210.11 total 60.06 last 67.36) total: 4500 open/close in 65.44 seconds: 68.76 ops/second - /unlink 1180 (time 1783967229.90 total 10.01 last 117.94) - /unlink 2340 (time 1783967239.90 total 20.01 last 115.94) - /unlink 3526 (time 1783967249.90 total 30.01 last 118.60) total: 4500 /unlink in 37.00 seconds: 121.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 811 (time 1783967271.63 total 10.02 last 80.93) - open/close 1593 (time 1783967281.63 total 20.02 last 78.19) - open/close 2468 (time 1783967291.64 total 30.03 last 87.43) - open/close 3242 (time 1783967301.66 total 40.04 last 77.30) - open/close 3787 (time 1783967311.67 total 50.06 last 54.40) - open/close 4107 (time 1783967321.69 total 60.08 last 31.95) total: 4500 open/close in 66.38 seconds: 67.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1207 (time 1783967342.62 total 10.01 last 120.64) - /unlink 2471 (time 1783967352.63 total 20.01 last 126.31) - /unlink 3780 (time 1783967362.65 total 30.03 last 130.62) total: 4500 /unlink in 36.14 seconds: 124.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 619 (time 1783967385.60 total 10.00 last 61.87) - open/close 988 (time 1783967395.62 total 20.03 last 36.82) - open/close 1627 (time 1783967405.63 total 30.03 last 63.87) - open/close 2227 (time 1783967415.64 total 40.05 last 59.90) - open/close 2897 (time 1783967425.65 total 50.05 last 66.95) - open/close 3537 (time 1783967435.65 total 60.05 last 64.00) - open/close 4303 (time 1783967445.65 total 70.06 last 76.58) total: 4500 open/close in 72.57 seconds: 62.01 ops/second - /unlink 1171 (time 1783967462.95 total 10.01 last 117.01) - /unlink 2295 (time 1783967472.95 total 20.01 last 112.39) - /unlink 3577 (time 1783967482.95 total 30.01 last 128.19) total: 4500 /unlink in 37.13 seconds: 121.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 721 (time 1783967505.40 total 10.00 last 72.10) - open/close 1524 (time 1783967515.40 total 20.00 last 80.29) - open/close 2294 (time 1783967525.41 total 30.01 last 76.94) - open/close 3011 (time 1783967535.42 total 40.02 last 71.63) - open/close 3716 (time 1783967545.42 total 50.02 last 70.48) total: 4500 open/close in 59.92 seconds: 75.10 ops/second - /unlink 1306 (time 1783967570.17 total 10.01 last 130.52) - /unlink 2437 (time 1783967580.18 total 20.01 last 113.03) - /unlink 3611 (time 1783967590.18 total 30.02 last 117.36) total: 4500 /unlink in 37.87 seconds: 118.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 747 (time 1783967613.23 total 10.01 last 74.63) - open/close 1555 (time 1783967623.24 total 20.02 last 80.71) - open/close 2219 (time 1783967633.25 total 30.03 last 66.35) - open/close 2907 (time 1783967643.25 total 40.03 last 68.79) - open/close 3668 (time 1783967653.25 total 50.03 last 76.06) - open/close 4391 (time 1783967663.25 total 60.04 last 72.28) total: 4500 open/close in 61.48 seconds: 73.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1242 (time 1783967680.69 total 10.00 last 124.16) - /unlink 2468 (time 1783967690.69 total 20.01 last 122.54) - /unlink 3649 (time 1783967700.70 total 30.01 last 118.06) total: 4500 /unlink in 37.38 seconds: 120.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 638 (time 1783967723.70 total 10.01 last 63.74) - open/close 1192 (time 1783967733.70 total 20.02 last 55.35) - open/close 1970 (time 1783967743.71 total 30.03 last 77.73) - open/close 2574 (time 1783967753.72 total 40.03 last 60.37) - open/close 3181 (time 1783967763.73 total 50.04 last 60.64) - open/close 3802 (time 1783967773.73 total 60.05 last 62.08) - open/close 4412 (time 1783967783.74 total 70.06 last 60.94) total: 4500 open/close in 71.34 seconds: 63.08 ops/second - /unlink 1210 (time 1783967800.57 total 10.00 last 120.96) - /unlink 2397 (time 1783967810.57 total 20.00 last 118.69) - /unlink 3775 (time 1783967820.58 total 30.01 last 137.74) total: 4500 /unlink in 35.93 seconds: 125.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 628 (time 1783967842.27 total 10.01 last 62.75) - open/close 1367 (time 1783967852.29 total 20.02 last 73.79) - open/close 2052 (time 1783967862.30 total 30.04 last 68.40) - open/close 2844 (time 1783967872.30 total 40.04 last 79.19) - open/close 3514 (time 1783967882.31 total 50.04 last 66.98) - open/close 4206 (time 1783967892.33 total 60.06 last 69.07) total: 4500 open/close in 64.02 seconds: 70.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1371 (time 1783967911.23 total 10.00 last 137.08) - /unlink 2770 (time 1783967921.24 total 20.01 last 139.78) - /unlink 4139 (time 1783967931.24 total 30.01 last 136.89) total: 4500 /unlink in 33.19 seconds: 135.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 648 (time 1783967950.46 total 10.01 last 64.74) - open/close 1328 (time 1783967960.46 total 20.01 last 67.98) - open/close 2023 (time 1783967970.47 total 30.03 last 69.41) - open/close 2626 (time 1783967980.48 total 40.03 last 60.27) - open/close 3337 (time 1783967990.48 total 50.03 last 71.08) - open/close 3947 (time 1783968000.49 total 60.05 last 60.92) total: 4500 open/close in 68.13 seconds: 66.05 ops/second - /unlink 1149 (time 1783968023.46 total 10.01 last 114.84) - /unlink 2564 (time 1783968033.46 total 20.01 last 141.44) - /unlink 3984 (time 1783968043.47 total 30.01 last 141.95) total: 4500 /unlink in 33.57 seconds: 134.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 672 (time 1783968061.92 total 10.01 last 67.15) - open/close 1334 (time 1783968071.93 total 20.02 last 66.12) - open/close 2067 (time 1783968081.94 total 30.02 last 73.29) - open/close 2779 (time 1783968091.95 total 40.03 last 71.11) - open/close 3463 (time 1783968101.96 total 50.04 last 68.34) - open/close 4234 (time 1783968111.97 total 60.06 last 76.96) total: 4500 open/close in 64.25 seconds: 70.04 ops/second - /unlink 1281 (time 1783968131.10 total 10.01 last 128.01) - /unlink 2549 (time 1783968141.10 total 20.01 last 126.80) - /unlink 3743 (time 1783968151.10 total 30.01 last 119.39) total: 4500 /unlink in 35.46 seconds: 126.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 653 (time 1783968171.24 total 10.00 last 65.30) - open/close 1320 (time 1783968181.24 total 20.00 last 66.69) - open/close 2077 (time 1783968191.24 total 30.00 last 75.69) - open/close 2892 (time 1783968201.24 total 40.01 last 81.48) - open/close 3678 (time 1783968211.25 total 50.01 last 78.56) - open/close 4418 (time 1783968221.26 total 60.02 last 73.90) total: 4500 open/close in 60.94 seconds: 73.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1411 (time 1783968237.56 total 10.00 last 141.03) - /unlink 2521 (time 1783968247.56 total 20.01 last 110.96) - /unlink 3852 (time 1783968257.57 total 30.01 last 133.02) total: 4500 /unlink in 34.80 seconds: 129.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 850 (time 1783968277.35 total 10.00 last 84.97) - open/close 1599 (time 1783968287.35 total 20.00 last 74.90) - open/close 2339 (time 1783968297.35 total 30.01 last 73.94) - open/close 3024 (time 1783968307.40 total 40.05 last 68.21) - open/close 3670 (time 1783968317.40 total 50.06 last 64.58) total: 4500 open/close in 59.77 seconds: 75.29 ops/second - /unlink 1272 (time 1783968342.29 total 10.00 last 127.15) - /unlink 2624 (time 1783968352.30 total 20.01 last 135.15) - /unlink 4042 (time 1783968362.30 total 30.01 last 141.73) total: 4500 /unlink in 33.73 seconds: 133.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 730 (time 1783968381.29 total 10.00 last 72.99) - open/close 1366 (time 1783968391.29 total 20.00 last 63.60) - open/close 2173 (time 1783968401.29 total 30.00 last 80.69) - open/close 2835 (time 1783968411.29 total 40.01 last 66.18) - open/close 3585 (time 1783968421.30 total 50.01 last 74.95) - open/close 4268 (time 1783968431.30 total 60.01 last 68.29) total: 4500 open/close in 62.92 seconds: 71.52 ops/second - /unlink 1309 (time 1783968449.58 total 10.00 last 130.87) - /unlink 2640 (time 1783968459.59 total 20.01 last 133.01) - /unlink 3994 (time 1783968469.59 total 30.01 last 135.39) total: 4500 /unlink in 33.87 seconds: 132.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 658 (time 1783968488.90 total 10.03 last 65.60) - open/close 1348 (time 1783968498.91 total 20.04 last 68.96) - open/close 2133 (time 1783968508.92 total 30.05 last 78.42) - open/close 2904 (time 1783968518.92 total 40.05 last 77.09) - open/close 3503 (time 1783968528.93 total 50.06 last 59.81) - open/close 4237 (time 1783968538.94 total 60.07 last 73.34) total: 4500 open/close in 63.09 seconds: 71.32 ops/second - /unlink 1209 (time 1783968556.48 total 10.00 last 120.85) - /unlink 2542 (time 1783968566.49 total 20.01 last 133.22) - /unlink 3831 (time 1783968576.50 total 30.02 last 128.77) total: 4500 /unlink in 34.50 seconds: 130.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 654 (time 1783968595.96 total 10.01 last 65.35) - open/close 1310 (time 1783968605.96 total 20.01 last 65.59) - open/close 2098 (time 1783968615.97 total 30.01 last 78.78) - open/close 2835 (time 1783968625.97 total 40.02 last 73.66) - open/close 3462 (time 1783968635.98 total 50.03 last 62.65) - open/close 4048 (time 1783968645.99 total 60.04 last 58.54) total: 4500 open/close in 66.43 seconds: 67.75 ops/second - /unlink 1252 (time 1783968667.95 total 10.00 last 125.19) - /unlink 2624 (time 1783968677.95 total 20.00 last 137.15) - /unlink 4097 (time 1783968687.96 total 30.02 last 147.11) total: 4500 /unlink in 33.08 seconds: 136.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 846 (time 1783968706.54 total 10.00 last 84.57) - open/close 1593 (time 1783968716.55 total 20.01 last 74.66) - open/close 2363 (time 1783968726.58 total 30.04 last 76.78) - open/close 3098 (time 1783968736.58 total 40.04 last 73.48) - open/close 3813 (time 1783968746.59 total 50.05 last 71.44) total: 4500 open/close in 59.89 seconds: 75.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1019 (time 1783968772.50 total 10.01 last 101.82) - /unlink 1972 (time 1783968782.50 total 20.01 last 95.26) - /unlink 3054 (time 1783968792.53 total 30.04 last 107.92) - /unlink 4308 (time 1783968802.55 total 40.06 last 125.14) total: 4500 /unlink in 41.75 seconds: 107.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 757 (time 1783968820.20 total 10.00 last 75.66) - open/close 1505 (time 1783968830.20 total 20.01 last 74.78) - open/close 2096 (time 1783968840.23 total 30.03 last 58.96) - open/close 2709 (time 1783968850.24 total 40.04 last 61.23) - open/close 3287 (time 1783968860.24 total 50.05 last 57.77) - open/close 3938 (time 1783968870.26 total 60.06 last 65.03) total: 4500 open/close in 69.01 seconds: 65.21 ops/second - /unlink 1140 (time 1783968893.59 total 10.01 last 113.90) - /unlink 2120 (time 1783968903.59 total 20.01 last 97.95) - /unlink 3583 (time 1783968913.60 total 30.02 last 146.26) total: 4500 /unlink in 37.64 seconds: 119.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 639 (time 1783968936.13 total 10.00 last 63.88) - open/close 1241 (time 1783968946.14 total 20.01 last 60.18) - open/close 1849 (time 1783968956.14 total 30.01 last 60.80) - open/close 2436 (time 1783968966.14 total 40.01 last 58.68) - open/close 2949 (time 1783968976.14 total 50.01 last 51.29) - open/close 3560 (time 1783968986.14 total 60.01 last 61.10) - open/close 4245 (time 1783968996.15 total 70.02 last 68.47) total: 4500 open/close in 73.11 seconds: 61.55 ops/second - /unlink 1270 (time 1783969014.14 total 10.03 last 126.64) - /unlink 2439 (time 1783969024.14 total 20.03 last 116.87) - /unlink 3435 (time 1783969034.15 total 30.05 last 99.45) total: 4500 /unlink in 38.73 seconds: 116.20 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95322 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg157-server unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: modules unloaded.