== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:15:40 (1781000140) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg335-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg335-server oleg335-server: oleg335-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg335-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg335-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg335-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 667 (time 1781000354.57 total 10.01 last 66.60) - open/close 1379 (time 1781000364.59 total 20.03 last 71.09) - open/close 1979 (time 1781000374.59 total 30.04 last 59.95) - open/close 2727 (time 1781000384.60 total 40.05 last 74.73) - open/close 3497 (time 1781000394.62 total 50.06 last 76.90) - open/close 4314 (time 1781000404.62 total 60.06 last 81.69) total: 4500 open/close in 62.99 seconds: 71.44 ops/second - /unlink 1481 (time 1781000422.34 total 10.00 last 148.10) - /unlink 2829 (time 1781000432.34 total 20.00 last 134.77) - /unlink 4283 (time 1781000442.35 total 30.01 last 145.25) total: 4500 /unlink in 31.62 seconds: 142.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 709 (time 1781000458.22 total 10.00 last 70.87) - open/close 1284 (time 1781000468.23 total 20.02 last 57.44) - open/close 1907 (time 1781000478.24 total 30.03 last 62.23) - open/close 2675 (time 1781000488.24 total 40.03 last 76.79) - open/close 3406 (time 1781000498.25 total 50.04 last 73.03) - open/close 4105 (time 1781000508.26 total 60.05 last 69.83) total: 4500 open/close in 64.89 seconds: 69.34 ops/second - /unlink 1163 (time 1781000528.65 total 10.02 last 116.07) - /unlink 2489 (time 1781000538.65 total 20.02 last 132.56) - /unlink 3971 (time 1781000548.66 total 30.03 last 148.05) total: 4500 /unlink in 33.85 seconds: 132.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 842 (time 1781000567.50 total 10.00 last 84.16) - open/close 1631 (time 1781000577.52 total 20.02 last 78.79) - open/close 2459 (time 1781000587.53 total 30.03 last 82.71) - open/close 3264 (time 1781000597.53 total 40.03 last 80.49) - open/close 4082 (time 1781000607.53 total 50.03 last 81.79) total: 4500 open/close in 55.22 seconds: 81.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1277 (time 1781000627.02 total 10.00 last 127.67) - /unlink 2584 (time 1781000637.03 total 20.02 last 130.52) - /unlink 4016 (time 1781000647.03 total 30.02 last 143.19) total: 4500 /unlink in 32.97 seconds: 136.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 843 (time 1781000664.44 total 10.00 last 84.29) - open/close 1673 (time 1781000674.46 total 20.02 last 82.88) - open/close 2478 (time 1781000684.46 total 30.02 last 80.50) - open/close 3305 (time 1781000694.46 total 40.02 last 82.65) - open/close 4087 (time 1781000704.47 total 50.03 last 78.15) total: 4500 open/close in 55.26 seconds: 81.44 ops/second - /unlink 1303 (time 1781000723.47 total 10.03 last 129.96) - /unlink 2665 (time 1781000733.48 total 20.04 last 136.06) - /unlink 3876 (time 1781000743.49 total 30.04 last 121.01) total: 4500 /unlink in 34.04 seconds: 132.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 767 (time 1781000762.62 total 10.00 last 76.68) - open/close 1509 (time 1781000772.63 total 20.00 last 74.19) - open/close 2214 (time 1781000782.64 total 30.01 last 70.43) - open/close 2976 (time 1781000792.64 total 40.02 last 76.18) - open/close 3756 (time 1781000802.65 total 50.02 last 77.95) total: 4500 open/close in 59.63 seconds: 75.47 ops/second - /unlink 1521 (time 1781000827.14 total 10.01 last 151.98) - /unlink 2982 (time 1781000837.14 total 20.01 last 146.08) - /unlink 4448 (time 1781000847.14 total 30.01 last 146.59) total: 4500 /unlink in 30.47 seconds: 147.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 822 (time 1781000861.93 total 10.00 last 82.20) - open/close 1569 (time 1781000871.94 total 20.01 last 74.60) - open/close 2407 (time 1781000881.95 total 30.02 last 83.73) - open/close 3235 (time 1781000891.95 total 40.03 last 82.74) - open/close 4055 (time 1781000901.97 total 50.04 last 81.90) total: 4500 open/close in 55.15 seconds: 81.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1502 (time 1781000920.94 total 10.00 last 150.19) - /unlink 2909 (time 1781000930.94 total 20.00 last 140.64) - /unlink 4245 (time 1781000940.94 total 30.01 last 133.58) total: 4500 /unlink in 31.82 seconds: 141.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 813 (time 1781000957.39 total 10.00 last 81.28) - open/close 1583 (time 1781000967.39 total 20.00 last 76.99) - open/close 2397 (time 1781000977.41 total 30.02 last 81.24) - open/close 3167 (time 1781000987.42 total 40.03 last 76.94) - open/close 3869 (time 1781000997.43 total 50.04 last 70.15) total: 4500 open/close in 59.26 seconds: 75.94 ops/second - /unlink 1280 (time 1781001022.93 total 10.01 last 127.93) - /unlink 2524 (time 1781001032.93 total 20.01 last 124.37) - /unlink 3770 (time 1781001042.93 total 30.01 last 124.57) total: 4500 /unlink in 35.75 seconds: 125.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 825 (time 1781001062.27 total 10.00 last 82.48) - open/close 1626 (time 1781001072.28 total 20.01 last 80.04) - open/close 2413 (time 1781001082.28 total 30.02 last 78.66) - open/close 3236 (time 1781001092.29 total 40.03 last 82.22) - open/close 3935 (time 1781001102.30 total 50.03 last 69.89) total: 4500 open/close in 56.91 seconds: 79.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1353 (time 1781001123.57 total 10.01 last 135.21) - /unlink 2732 (time 1781001133.57 total 20.01 last 137.88) - /unlink 3907 (time 1781001143.57 total 30.01 last 117.49) total: 4500 /unlink in 34.37 seconds: 130.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 740 (time 1781001162.73 total 10.00 last 73.99) - open/close 1569 (time 1781001172.73 total 20.00 last 82.89) - open/close 2350 (time 1781001182.73 total 30.01 last 78.05) - open/close 3202 (time 1781001192.74 total 40.01 last 85.15) - open/close 4005 (time 1781001202.76 total 50.03 last 80.16) total: 4500 open/close in 56.63 seconds: 79.46 ops/second - /unlink 1521 (time 1781001224.39 total 10.00 last 152.07) - /unlink 2857 (time 1781001234.39 total 20.01 last 133.55) - /unlink 4275 (time 1781001244.40 total 30.01 last 141.71) total: 4500 /unlink in 31.62 seconds: 142.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 821 (time 1781001260.59 total 10.01 last 82.00) - open/close 1681 (time 1781001270.61 total 20.03 last 85.85) - open/close 2507 (time 1781001280.61 total 30.04 last 82.55) - open/close 3334 (time 1781001290.62 total 40.05 last 82.62) - open/close 4146 (time 1781001300.63 total 50.06 last 81.12) total: 4500 open/close in 54.38 seconds: 82.76 ops/second - /unlink 1507 (time 1781001319.27 total 10.02 last 150.33) - /unlink 2687 (time 1781001329.28 total 20.04 last 117.86) - /unlink 3879 (time 1781001339.28 total 30.04 last 119.19) total: 4500 /unlink in 35.50 seconds: 126.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 655 (time 1781001360.15 total 10.02 last 65.38) - open/close 1395 (time 1781001370.16 total 20.03 last 73.92) - open/close 2049 (time 1781001380.17 total 30.04 last 65.34) - open/close 2774 (time 1781001390.19 total 40.05 last 72.40) - open/close 3492 (time 1781001400.20 total 50.07 last 71.69) - open/close 4166 (time 1781001410.20 total 60.07 last 67.39) total: 4500 open/close in 64.25 seconds: 70.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1609 (time 1781001429.10 total 10.00 last 160.83) - /unlink 3145 (time 1781001439.10 total 20.01 last 153.56) total: 4500 /unlink in 29.68 seconds: 151.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 700 (time 1781001463.70 total 10.01 last 69.95) - open/close 1543 (time 1781001473.70 total 20.01 last 84.28) - open/close 2303 (time 1781001483.70 total 30.01 last 75.98) - open/close 3150 (time 1781001493.71 total 40.02 last 84.67) - open/close 3972 (time 1781001503.72 total 50.02 last 82.13) total: 4500 open/close in 56.54 seconds: 79.59 ops/second - /unlink 1519 (time 1781001524.33 total 10.00 last 151.85) - /unlink 2974 (time 1781001534.33 total 20.00 last 145.49) - /unlink 4456 (time 1781001544.33 total 30.01 last 148.17) total: 4500 /unlink in 30.28 seconds: 148.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 799 (time 1781001558.74 total 10.00 last 79.89) - open/close 1669 (time 1781001568.75 total 20.02 last 86.86) - open/close 2232 (time 1781001578.76 total 30.03 last 56.25) - open/close 3109 (time 1781001588.76 total 40.03 last 87.70) - open/close 3975 (time 1781001598.76 total 50.03 last 86.59) total: 4500 open/close in 56.38 seconds: 79.81 ops/second - /unlink 1477 (time 1781001619.17 total 10.00 last 147.67) - /unlink 2874 (time 1781001629.19 total 20.01 last 139.54) - /unlink 4173 (time 1781001639.19 total 30.01 last 129.89) total: 4500 /unlink in 33.10 seconds: 135.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 739 (time 1781001657.75 total 10.00 last 73.87) - open/close 1492 (time 1781001667.75 total 20.01 last 75.25) - open/close 2092 (time 1781001677.78 total 30.03 last 59.86) - open/close 2735 (time 1781001687.79 total 40.05 last 64.19) - open/close 3470 (time 1781001697.80 total 50.05 last 73.47) - open/close 4202 (time 1781001707.81 total 60.06 last 73.14) total: 4500 open/close in 64.57 seconds: 69.69 ops/second - /unlink 1094 (time 1781001727.47 total 10.00 last 109.38) - /unlink 2231 (time 1781001737.48 total 20.01 last 113.58) - /unlink 3258 (time 1781001747.48 total 30.02 last 102.66) - /unlink 4106 (time 1781001757.49 total 40.03 last 84.71) total: 4500 /unlink in 43.63 seconds: 103.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 583 (time 1781001776.86 total 10.02 last 58.21) - open/close 1208 (time 1781001786.91 total 20.08 last 62.13) - open/close 1807 (time 1781001796.94 total 30.10 last 59.73) - open/close 2397 (time 1781001806.96 total 40.12 last 58.93) - open/close 3010 (time 1781001816.97 total 50.13 last 61.24) - open/close 3665 (time 1781001826.97 total 60.13 last 65.45) - open/close 4399 (time 1781001836.99 total 70.15 last 73.26) total: 4500 open/close in 71.46 seconds: 62.97 ops/second - /unlink 1363 (time 1781001853.35 total 10.01 last 136.18) - /unlink 2601 (time 1781001863.35 total 20.01 last 123.80) - /unlink 3906 (time 1781001873.36 total 30.01 last 130.44) total: 4500 /unlink in 35.65 seconds: 126.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 768 (time 1781001893.44 total 10.00 last 76.80) - open/close 1479 (time 1781001903.45 total 20.01 last 71.03) - open/close 2227 (time 1781001913.47 total 30.03 last 74.66) - open/close 2994 (time 1781001923.47 total 40.03 last 76.65) - open/close 3691 (time 1781001933.48 total 50.04 last 69.68) - open/close 4415 (time 1781001943.49 total 60.05 last 72.32) total: 4500 open/close in 61.51 seconds: 73.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1224 (time 1781001959.65 total 10.00 last 122.38) - /unlink 2365 (time 1781001969.65 total 20.00 last 114.07) - /unlink 3339 (time 1781001979.65 total 30.01 last 97.38) - /unlink 4469 (time 1781001989.66 total 40.01 last 112.93) total: 4500 /unlink in 40.38 seconds: 111.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 736 (time 1781002004.73 total 10.02 last 73.46) - open/close 1276 (time 1781002014.74 total 20.03 last 53.96) - open/close 1924 (time 1781002024.74 total 30.03 last 64.80) - open/close 2632 (time 1781002034.74 total 40.03 last 70.79) - open/close 3417 (time 1781002044.74 total 50.03 last 78.46) - open/close 4230 (time 1781002054.75 total 60.04 last 81.23) total: 4500 open/close in 63.39 seconds: 70.99 ops/second - /unlink 1384 (time 1781002072.03 total 10.00 last 138.37) - /unlink 2555 (time 1781002082.03 total 20.00 last 117.07) - /unlink 3843 (time 1781002092.03 total 30.01 last 128.76) total: 4500 /unlink in 35.50 seconds: 126.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 663 (time 1781002112.35 total 10.01 last 66.22) - open/close 1309 (time 1781002122.35 total 20.01 last 64.58) - open/close 2045 (time 1781002132.36 total 30.03 last 73.52) - open/close 2859 (time 1781002142.38 total 40.04 last 81.26) - open/close 3571 (time 1781002152.40 total 50.06 last 71.05) - open/close 4246 (time 1781002162.40 total 60.07 last 67.48) total: 4500 open/close in 63.58 seconds: 70.77 ops/second - /unlink 1325 (time 1781002181.28 total 10.00 last 132.47) - /unlink 2557 (time 1781002191.29 total 20.01 last 123.15) - /unlink 3820 (time 1781002201.29 total 30.01 last 126.29) total: 4500 /unlink in 34.84 seconds: 129.15 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11203: 93949 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local oleg335-server: modules unloaded.