== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:40:05 (1780454405) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg260-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg260-server oleg260-server: oleg260-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg260-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg260-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 725 (time 1780454610.71 total 10.02 last 72.38) - open/close 1535 (time 1780454620.71 total 20.02 last 80.99) - open/close 2362 (time 1780454630.72 total 30.03 last 82.63) - open/close 3208 (time 1780454640.72 total 40.03 last 84.53) - open/close 4053 (time 1780454650.74 total 50.05 last 84.36) total: 4500 open/close in 55.24 seconds: 81.47 ops/second - /unlink 1616 (time 1780454669.82 total 10.01 last 161.49) - /unlink 3306 (time 1780454679.83 total 20.01 last 168.93) total: 4500 /unlink in 26.75 seconds: 168.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 875 (time 1780454700.64 total 10.00 last 87.49) - open/close 1645 (time 1780454710.64 total 20.01 last 76.97) - open/close 2416 (time 1780454720.65 total 30.01 last 77.05) - open/close 3183 (time 1780454730.65 total 40.02 last 76.66) - open/close 3953 (time 1780454740.66 total 50.02 last 76.98) total: 4500 open/close in 55.90 seconds: 80.50 ops/second - /unlink 1517 (time 1780454761.31 total 10.00 last 151.69) - /unlink 3053 (time 1780454771.32 total 20.01 last 153.46) - /unlink 4490 (time 1780454781.32 total 30.01 last 143.65) total: 4500 /unlink in 30.13 seconds: 149.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 680 (time 1780454798.29 total 10.00 last 67.98) - open/close 1434 (time 1780454808.31 total 20.02 last 75.27) - open/close 2297 (time 1780454818.31 total 30.02 last 86.28) - open/close 3007 (time 1780454828.33 total 40.04 last 70.89) - open/close 3824 (time 1780454838.34 total 50.06 last 81.55) total: 4500 open/close in 57.99 seconds: 77.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1401 (time 1780454861.00 total 10.00 last 140.06) - /unlink 3129 (time 1780454871.00 total 20.01 last 172.72) total: 4500 /unlink in 29.47 seconds: 152.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 881 (time 1780454895.45 total 10.00 last 88.10) - open/close 1582 (time 1780454905.46 total 20.01 last 70.04) - open/close 2408 (time 1780454915.46 total 30.01 last 82.59) - open/close 3255 (time 1780454925.47 total 40.02 last 84.64) - open/close 4150 (time 1780454935.47 total 50.03 last 89.43) total: 4500 open/close in 53.94 seconds: 83.42 ops/second - /unlink 1472 (time 1780454954.08 total 10.01 last 147.05) - /unlink 2877 (time 1780454964.09 total 20.01 last 140.44) - /unlink 4345 (time 1780454974.09 total 30.02 last 146.77) total: 4500 /unlink in 31.04 seconds: 144.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 766 (time 1780454989.61 total 10.00 last 76.58) - open/close 1615 (time 1780454999.61 total 20.00 last 84.88) - open/close 2449 (time 1780455009.62 total 30.01 last 83.33) - open/close 3217 (time 1780455019.63 total 40.02 last 76.73) - open/close 4039 (time 1780455029.63 total 50.02 last 82.20) total: 4500 open/close in 55.83 seconds: 80.60 ops/second - /unlink 1834 (time 1780455049.56 total 10.00 last 183.34) - /unlink 3503 (time 1780455059.56 total 20.01 last 166.85) total: 4500 /unlink in 27.14 seconds: 165.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 848 (time 1780455081.64 total 10.00 last 84.78) - open/close 1640 (time 1780455091.65 total 20.01 last 79.13) - open/close 2375 (time 1780455101.65 total 30.02 last 73.45) - open/close 3157 (time 1780455111.66 total 40.03 last 78.14) - open/close 3949 (time 1780455121.66 total 50.03 last 79.17) total: 4500 open/close in 58.70 seconds: 76.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1242 (time 1780455145.61 total 10.00 last 124.15) - /unlink 2572 (time 1780455155.62 total 20.01 last 132.86) - /unlink 4090 (time 1780455165.63 total 30.02 last 151.73) total: 4500 /unlink in 33.04 seconds: 136.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 657 (time 1780455183.58 total 10.01 last 65.65) - open/close 1510 (time 1780455193.59 total 20.02 last 85.19) - open/close 2470 (time 1780455203.59 total 30.02 last 95.96) - open/close 3230 (time 1780455213.60 total 40.03 last 75.97) - open/close 4017 (time 1780455223.60 total 50.04 last 78.64) total: 4500 open/close in 55.90 seconds: 80.50 ops/second - /unlink 1611 (time 1780455244.14 total 10.00 last 161.04) - /unlink 3118 (time 1780455254.14 total 20.01 last 150.64) total: 4500 /unlink in 28.74 seconds: 156.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 888 (time 1780455277.99 total 10.02 last 88.66) - open/close 1658 (time 1780455287.99 total 20.02 last 76.98) - open/close 2438 (time 1780455298.00 total 30.02 last 77.98) - open/close 3263 (time 1780455308.00 total 40.02 last 82.48) - open/close 4082 (time 1780455318.02 total 50.04 last 81.73) total: 4500 open/close in 55.63 seconds: 80.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1259 (time 1780455338.43 total 10.00 last 125.88) - /unlink 2617 (time 1780455348.44 total 20.01 last 135.72) - /unlink 4041 (time 1780455358.44 total 30.01 last 142.40) total: 4500 /unlink in 34.26 seconds: 131.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 855 (time 1780455378.83 total 10.01 last 85.45) - open/close 1730 (time 1780455388.84 total 20.02 last 87.41) - open/close 2594 (time 1780455398.85 total 30.02 last 86.37) - open/close 3487 (time 1780455408.85 total 40.02 last 89.27) - open/close 4237 (time 1780455418.86 total 50.03 last 74.95) total: 4500 open/close in 53.31 seconds: 84.41 ops/second - /unlink 1371 (time 1780455436.49 total 10.00 last 137.07) - /unlink 2769 (time 1780455446.49 total 20.00 last 139.78) - /unlink 4294 (time 1780455456.49 total 30.01 last 152.45) total: 4500 /unlink in 31.44 seconds: 143.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 732 (time 1780455473.17 total 10.00 last 73.20) - open/close 1504 (time 1780455483.18 total 20.01 last 77.14) - open/close 2088 (time 1780455493.20 total 30.03 last 58.25) - open/close 2668 (time 1780455503.21 total 40.04 last 57.98) - open/close 3424 (time 1780455513.21 total 50.04 last 75.55) - open/close 4084 (time 1780455523.22 total 60.05 last 65.97) total: 4500 open/close in 65.93 seconds: 68.25 ops/second - /unlink 1419 (time 1780455543.39 total 10.00 last 141.85) - /unlink 2704 (time 1780455553.39 total 20.00 last 128.49) - /unlink 3905 (time 1780455563.39 total 30.00 last 120.09) total: 4500 /unlink in 33.88 seconds: 132.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 733 (time 1780455582.79 total 10.00 last 73.29) - open/close 1514 (time 1780455592.80 total 20.01 last 78.00) - open/close 2215 (time 1780455602.82 total 30.04 last 69.95) - open/close 2931 (time 1780455612.82 total 40.04 last 71.59) - open/close 3568 (time 1780455622.83 total 50.04 last 63.67) - open/close 4339 (time 1780455632.84 total 60.05 last 77.01) total: 4500 open/close in 62.37 seconds: 72.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1736 (time 1780455648.31 total 10.00 last 173.52) - /unlink 3073 (time 1780455658.32 total 20.01 last 133.61) - /unlink 4483 (time 1780455668.32 total 30.01 last 140.98) total: 4500 /unlink in 30.08 seconds: 149.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 681 (time 1780455683.33 total 10.03 last 67.92) - open/close 1366 (time 1780455693.34 total 20.03 last 68.49) - open/close 2206 (time 1780455703.35 total 30.04 last 83.92) - open/close 2931 (time 1780455713.36 total 40.06 last 72.37) - open/close 3758 (time 1780455723.38 total 50.07 last 82.59) total: 4500 open/close in 58.82 seconds: 76.51 ops/second - /unlink 1457 (time 1780455746.91 total 10.00 last 145.66) - /unlink 2906 (time 1780455756.91 total 20.00 last 144.87) - /unlink 4348 (time 1780455766.91 total 30.01 last 144.19) total: 4500 /unlink in 31.19 seconds: 144.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 776 (time 1780455781.92 total 10.01 last 77.54) - open/close 1443 (time 1780455791.93 total 20.02 last 66.61) - open/close 2222 (time 1780455801.94 total 30.02 last 77.87) - open/close 3134 (time 1780455811.94 total 40.03 last 91.15) - open/close 3821 (time 1780455821.94 total 50.03 last 68.70) total: 4500 open/close in 58.00 seconds: 77.59 ops/second - /unlink 1512 (time 1780455845.27 total 10.00 last 151.13) - /unlink 2846 (time 1780455855.28 total 20.02 last 133.21) - /unlink 4149 (time 1780455865.28 total 30.02 last 130.28) total: 4500 /unlink in 32.17 seconds: 139.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 640 (time 1780455883.33 total 10.01 last 63.92) - open/close 1354 (time 1780455893.33 total 20.01 last 71.40) - open/close 2085 (time 1780455903.33 total 30.01 last 73.09) - open/close 2869 (time 1780455913.34 total 40.02 last 78.37) - open/close 3495 (time 1780455923.34 total 50.02 last 62.58) - open/close 4128 (time 1780455933.36 total 60.04 last 63.21) total: 4500 open/close in 66.12 seconds: 68.06 ops/second - /unlink 1354 (time 1780455954.98 total 10.01 last 135.31) - /unlink 2977 (time 1780455964.98 total 20.01 last 162.28) total: 4500 /unlink in 29.91 seconds: 150.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 775 (time 1780455989.92 total 10.01 last 77.43) - open/close 1398 (time 1780455999.92 total 20.01 last 62.30) - open/close 2053 (time 1780456009.94 total 30.03 last 65.36) - open/close 2826 (time 1780456019.94 total 40.04 last 77.26) - open/close 3411 (time 1780456029.99 total 50.08 last 58.25) - open/close 4156 (time 1780456040.00 total 60.09 last 74.42) total: 4500 open/close in 64.97 seconds: 69.27 ops/second - /unlink 1530 (time 1780456059.54 total 10.00 last 152.99) - /unlink 2946 (time 1780456069.54 total 20.00 last 141.56) total: 4500 /unlink in 29.71 seconds: 151.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 840 (time 1780456094.06 total 10.00 last 83.99) - open/close 1578 (time 1780456104.07 total 20.01 last 73.74) - open/close 2436 (time 1780456114.08 total 30.02 last 85.71) - open/close 3200 (time 1780456124.12 total 40.06 last 76.11) - open/close 4102 (time 1780456134.12 total 50.06 last 90.19) total: 4500 open/close in 55.96 seconds: 80.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1440 (time 1780456155.80 total 10.00 last 143.97) - /unlink 2799 (time 1780456165.81 total 20.01 last 135.78) - /unlink 4228 (time 1780456175.82 total 30.02 last 142.78) total: 4500 /unlink in 32.03 seconds: 140.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 730 (time 1780456193.59 total 10.01 last 72.93) - open/close 1444 (time 1780456203.60 total 20.02 last 71.34) - open/close 2277 (time 1780456213.61 total 30.03 last 83.17) - open/close 3088 (time 1780456223.63 total 40.05 last 80.98) - open/close 3866 (time 1780456233.63 total 50.05 last 77.78) total: 4500 open/close in 58.92 seconds: 76.38 ops/second - /unlink 1262 (time 1780456258.01 total 10.02 last 125.96) - /unlink 2627 (time 1780456268.01 total 20.02 last 136.43) - /unlink 3808 (time 1780456278.01 total 30.03 last 118.09) total: 4500 /unlink in 35.72 seconds: 125.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 759 (time 1780456298.85 total 10.01 last 75.82) - open/close 1508 (time 1780456308.87 total 20.02 last 74.82) - open/close 2276 (time 1780456318.87 total 30.03 last 76.76) - open/close 3043 (time 1780456328.88 total 40.04 last 76.64) - open/close 3733 (time 1780456338.90 total 50.06 last 68.84) - open/close 4399 (time 1780456348.91 total 60.07 last 66.55) total: 4500 open/close in 61.61 seconds: 73.05 ops/second - /unlink 1368 (time 1780456366.22 total 10.00 last 136.77) - /unlink 2694 (time 1780456376.22 total 20.01 last 132.56) - /unlink 4043 (time 1780456386.23 total 30.01 last 134.84) total: 4500 /unlink in 33.41 seconds: 134.68 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94069 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: modules unloaded.