== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:15:50 (1783966550) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 818 (time 1783966774.82 total 10.02 last 81.67) - open/close 1531 (time 1783966784.84 total 20.04 last 71.12) - open/close 2262 (time 1783966794.85 total 30.04 last 73.08) - open/close 2918 (time 1783966804.85 total 40.05 last 65.57) - open/close 3672 (time 1783966814.86 total 50.05 last 75.36) - open/close 4390 (time 1783966824.86 total 60.05 last 71.80) total: 4500 open/close in 61.52 seconds: 73.15 ops/second - /unlink 1325 (time 1783966840.80 total 10.00 last 132.43) - /unlink 2645 (time 1783966850.80 total 20.01 last 131.97) - /unlink 3939 (time 1783966860.83 total 30.03 last 129.05) total: 4500 /unlink in 34.56 seconds: 130.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 723 (time 1783966880.24 total 10.00 last 72.27) - open/close 1472 (time 1783966890.25 total 20.01 last 74.85) - open/close 2231 (time 1783966900.25 total 30.02 last 75.84) - open/close 2981 (time 1783966910.26 total 40.03 last 74.95) - open/close 3761 (time 1783966920.26 total 50.03 last 77.99) - open/close 4475 (time 1783966930.27 total 60.03 last 71.37) total: 4500 open/close in 60.43 seconds: 74.46 ops/second - /unlink 1390 (time 1783966945.66 total 10.01 last 138.88) - /unlink 2952 (time 1783966955.66 total 20.01 last 156.17) total: 4500 /unlink in 29.94 seconds: 150.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 676 (time 1783966980.48 total 10.00 last 67.59) - open/close 1454 (time 1783966990.48 total 20.00 last 77.79) - open/close 2177 (time 1783967000.49 total 30.01 last 72.27) - open/close 2930 (time 1783967010.49 total 40.01 last 75.26) - open/close 3685 (time 1783967020.51 total 50.03 last 75.37) - open/close 4414 (time 1783967030.51 total 60.03 last 72.86) total: 4500 open/close in 61.30 seconds: 73.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1129 (time 1783967046.75 total 10.01 last 112.81) - /unlink 2393 (time 1783967056.76 total 20.02 last 126.30) - /unlink 3647 (time 1783967066.76 total 30.02 last 125.36) total: 4500 /unlink in 38.32 seconds: 117.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 615 (time 1783967091.64 total 10.02 last 61.37) - open/close 1318 (time 1783967101.65 total 20.03 last 70.23) - open/close 2116 (time 1783967111.65 total 30.04 last 79.75) - open/close 2871 (time 1783967121.66 total 40.04 last 75.48) - open/close 3637 (time 1783967131.68 total 50.06 last 76.44) - open/close 4354 (time 1783967141.68 total 60.06 last 71.68) total: 4500 open/close in 61.70 seconds: 72.93 ops/second - /unlink 1246 (time 1783967157.63 total 10.02 last 124.35) - /unlink 2487 (time 1783967167.64 total 20.02 last 124.05) - /unlink 3675 (time 1783967177.64 total 30.03 last 118.77) total: 4500 /unlink in 36.41 seconds: 123.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 654 (time 1783967199.28 total 10.01 last 65.31) - open/close 1316 (time 1783967209.28 total 20.01 last 66.20) - open/close 2017 (time 1783967219.29 total 30.02 last 70.04) - open/close 2694 (time 1783967229.29 total 40.03 last 67.68) - open/close 3394 (time 1783967239.30 total 50.04 last 69.94) - open/close 4047 (time 1783967249.31 total 60.04 last 65.26) total: 4500 open/close in 66.47 seconds: 67.70 ops/second - /unlink 1374 (time 1783967271.43 total 10.01 last 137.33) - /unlink 2655 (time 1783967281.44 total 20.01 last 128.05) - /unlink 4118 (time 1783967291.44 total 30.01 last 146.26) total: 4500 /unlink in 33.93 seconds: 132.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 655 (time 1783967310.61 total 10.01 last 65.40) - open/close 1384 (time 1783967320.61 total 20.02 last 72.90) - open/close 2015 (time 1783967330.63 total 30.03 last 62.99) - open/close 2748 (time 1783967340.64 total 40.04 last 73.27) - open/close 3393 (time 1783967350.64 total 50.04 last 64.50) - open/close 4161 (time 1783967360.64 total 60.04 last 76.80) total: 4500 open/close in 64.33 seconds: 69.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1496 (time 1783967379.69 total 10.01 last 149.45) - /unlink 2856 (time 1783967389.69 total 20.01 last 135.97) - /unlink 4331 (time 1783967399.69 total 30.01 last 147.48) total: 4500 /unlink in 31.13 seconds: 144.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 703 (time 1783967415.23 total 10.01 last 70.24) - open/close 1511 (time 1783967425.25 total 20.03 last 80.60) - open/close 2271 (time 1783967435.25 total 30.03 last 76.00) - open/close 3138 (time 1783967445.26 total 40.04 last 86.67) - open/close 3959 (time 1783967455.26 total 50.04 last 82.04) total: 4500 open/close in 57.00 seconds: 78.95 ops/second - /unlink 1207 (time 1783967476.04 total 10.00 last 120.66) - /unlink 2601 (time 1783967486.05 total 20.01 last 139.36) - /unlink 3989 (time 1783967496.06 total 30.02 last 138.68) total: 4500 /unlink in 33.89 seconds: 132.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 680 (time 1783967514.90 total 10.01 last 67.96) - open/close 1402 (time 1783967524.91 total 20.01 last 72.14) - open/close 2285 (time 1783967534.91 total 30.02 last 88.26) - open/close 3025 (time 1783967544.92 total 40.03 last 73.90) - open/close 3716 (time 1783967554.95 total 50.06 last 68.93) - open/close 4442 (time 1783967564.95 total 60.06 last 72.57) total: 4500 open/close in 61.03 seconds: 73.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1200 (time 1783967580.43 total 10.00 last 119.99) - /unlink 2649 (time 1783967590.43 total 20.01 last 144.82) - /unlink 4020 (time 1783967600.44 total 30.02 last 136.97) total: 4500 /unlink in 34.15 seconds: 131.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 670 (time 1783967619.04 total 10.00 last 66.99) - open/close 1309 (time 1783967629.05 total 20.01 last 63.86) - open/close 2001 (time 1783967639.05 total 30.01 last 69.19) - open/close 2755 (time 1783967649.06 total 40.02 last 75.31) - open/close 3510 (time 1783967659.07 total 50.03 last 75.42) - open/close 4269 (time 1783967669.08 total 60.03 last 75.89) total: 4500 open/close in 63.25 seconds: 71.15 ops/second - /unlink 1303 (time 1783967687.08 total 10.00 last 130.24) - /unlink 2504 (time 1783967697.08 total 20.01 last 120.08) - /unlink 3836 (time 1783967707.08 total 30.01 last 133.15) total: 4500 /unlink in 34.69 seconds: 129.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 761 (time 1783967726.78 total 10.01 last 76.04) - open/close 1492 (time 1783967736.78 total 20.01 last 73.08) - open/close 2188 (time 1783967746.80 total 30.03 last 69.49) - open/close 2821 (time 1783967756.81 total 40.04 last 63.23) - open/close 3477 (time 1783967766.83 total 50.05 last 65.48) - open/close 4165 (time 1783967776.84 total 60.07 last 68.73) total: 4500 open/close in 64.54 seconds: 69.72 ops/second - /unlink 1208 (time 1783967795.74 total 10.01 last 120.71) - /unlink 2446 (time 1783967805.74 total 20.01 last 123.79) - /unlink 3766 (time 1783967815.75 total 30.01 last 131.96) total: 4500 /unlink in 36.08 seconds: 124.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 699 (time 1783967836.59 total 10.00 last 69.87) - open/close 1430 (time 1783967846.59 total 20.01 last 73.04) - open/close 2228 (time 1783967856.61 total 30.02 last 79.71) - open/close 2987 (time 1783967866.61 total 40.03 last 75.89) - open/close 3840 (time 1783967876.62 total 50.04 last 85.18) total: 4500 open/close in 58.30 seconds: 77.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1545 (time 1783967898.76 total 10.00 last 154.44) - /unlink 2952 (time 1783967908.76 total 20.01 last 140.62) - /unlink 4214 (time 1783967918.77 total 30.01 last 126.13) total: 4500 /unlink in 31.92 seconds: 140.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 707 (time 1783967935.28 total 10.01 last 70.66) - open/close 1343 (time 1783967945.29 total 20.01 last 63.55) - open/close 2208 (time 1783967955.29 total 30.02 last 86.48) - open/close 3097 (time 1783967965.30 total 40.02 last 88.89) - open/close 3729 (time 1783967975.30 total 50.02 last 63.17) - open/close 4373 (time 1783967985.31 total 60.03 last 64.36) total: 4500 open/close in 62.17 seconds: 72.38 ops/second - /unlink 1448 (time 1783968001.37 total 10.01 last 144.73) - /unlink 2834 (time 1783968011.37 total 20.01 last 138.50) - /unlink 4278 (time 1783968021.38 total 30.02 last 144.34) total: 4500 /unlink in 31.66 seconds: 142.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 866 (time 1783968036.79 total 10.00 last 86.60) - open/close 1748 (time 1783968046.81 total 20.02 last 88.04) - open/close 2421 (time 1783968056.81 total 30.02 last 67.28) - open/close 3065 (time 1783968066.81 total 40.03 last 64.38) - open/close 3876 (time 1783968076.82 total 50.03 last 81.03) total: 4500 open/close in 58.63 seconds: 76.76 ops/second - /unlink 1240 (time 1783968100.27 total 10.00 last 123.95) - /unlink 2395 (time 1783968110.27 total 20.01 last 115.47) - /unlink 3677 (time 1783968120.28 total 30.01 last 128.13) total: 4500 /unlink in 35.70 seconds: 126.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 714 (time 1783968141.37 total 10.00 last 71.39) - open/close 1431 (time 1783968151.37 total 20.00 last 71.69) - open/close 2084 (time 1783968161.41 total 30.04 last 65.03) - open/close 2772 (time 1783968171.43 total 40.06 last 68.70) - open/close 3423 (time 1783968181.43 total 50.06 last 65.09) - open/close 4164 (time 1783968191.43 total 60.06 last 74.09) total: 4500 open/close in 64.72 seconds: 69.53 ops/second - /unlink 1165 (time 1783968211.56 total 10.01 last 116.44) - /unlink 2232 (time 1783968221.57 total 20.01 last 106.66) - /unlink 3621 (time 1783968231.57 total 30.01 last 138.83) total: 4500 /unlink in 38.07 seconds: 118.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 626 (time 1783968255.39 total 10.02 last 62.48) - open/close 1242 (time 1783968265.39 total 20.02 last 61.57) - open/close 1902 (time 1783968275.42 total 30.05 last 65.84) - open/close 2646 (time 1783968285.43 total 40.06 last 74.35) - open/close 3377 (time 1783968295.43 total 50.06 last 73.08) - open/close 4200 (time 1783968305.43 total 60.06 last 82.26) total: 4500 open/close in 64.27 seconds: 70.02 ops/second - /unlink 1371 (time 1783968325.20 total 10.02 last 136.87) - /unlink 2482 (time 1783968335.20 total 20.02 last 111.08) - /unlink 3866 (time 1783968345.21 total 30.02 last 138.35) total: 4500 /unlink in 34.28 seconds: 131.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 439 (time 1783968365.98 total 10.01 last 43.85) - open/close 877 (time 1783968375.99 total 20.02 last 43.75) - open/close 1346 (time 1783968386.00 total 30.03 last 46.85) - open/close 1872 (time 1783968396.01 total 40.04 last 52.56) - open/close 2534 (time 1783968406.02 total 50.05 last 66.14) - open/close 3310 (time 1783968416.03 total 60.06 last 77.54) - open/close 4110 (time 1783968426.03 total 70.06 last 79.96) total: 4500 open/close in 75.18 seconds: 59.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1271 (time 1783968445.79 total 10.01 last 127.03) - /unlink 2592 (time 1783968455.80 total 20.01 last 132.01) - /unlink 3846 (time 1783968465.80 total 30.01 last 125.37) total: 4500 /unlink in 35.83 seconds: 125.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 757 (time 1783968486.81 total 10.01 last 75.65) - open/close 1393 (time 1783968496.85 total 20.04 last 63.37) - open/close 2048 (time 1783968506.86 total 30.05 last 65.45) - open/close 2730 (time 1783968516.86 total 40.06 last 68.17) - open/close 3491 (time 1783968526.86 total 50.06 last 76.08) - open/close 4136 (time 1783968536.88 total 60.07 last 64.42) total: 4500 open/close in 65.51 seconds: 68.69 ops/second - /unlink 1354 (time 1783968557.80 total 10.00 last 135.38) - /unlink 2670 (time 1783968567.80 total 20.01 last 131.54) - /unlink 4106 (time 1783968577.80 total 30.01 last 143.58) total: 4500 /unlink in 33.78 seconds: 133.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 699 (time 1783968597.23 total 10.00 last 69.87) - open/close 1406 (time 1783968607.23 total 20.01 last 70.67) - open/close 1974 (time 1783968617.23 total 30.01 last 56.78) - open/close 2653 (time 1783968627.24 total 40.01 last 67.90) - open/close 3275 (time 1783968637.26 total 50.04 last 62.05) - open/close 3762 (time 1783968647.27 total 60.05 last 48.65) - open/close 4281 (time 1783968657.28 total 70.05 last 51.87) total: 4500 open/close in 74.88 seconds: 60.10 ops/second - /unlink 890 (time 1783968678.04 total 10.02 last 88.82) - /unlink 1651 (time 1783968688.04 total 20.02 last 76.08) - /unlink 2686 (time 1783968698.05 total 30.03 last 103.45) - /unlink 3864 (time 1783968708.05 total 40.03 last 117.79) total: 4500 /unlink in 45.06 seconds: 99.87 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95341 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: modules unloaded.