== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:25:06 (1789140306) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg221-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg221-server oleg221-server: oleg221-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg221-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg221-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 641 (time 1789140533.45 total 10.00 last 64.10) - open/close 1361 (time 1789140543.47 total 20.02 last 71.83) - open/close 2088 (time 1789140553.48 total 30.02 last 72.70) - open/close 2736 (time 1789140563.49 total 40.04 last 64.70) - open/close 3330 (time 1789140573.50 total 50.05 last 59.35) - open/close 4011 (time 1789140583.51 total 60.06 last 68.00) total: 4500 open/close in 68.27 seconds: 65.91 ops/second - /unlink 1511 (time 1789140607.27 total 10.00 last 151.07) - /unlink 2840 (time 1789140617.27 total 20.00 last 132.88) - /unlink 4206 (time 1789140627.27 total 30.01 last 136.56) total: 4500 /unlink in 32.62 seconds: 137.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 765 (time 1789140646.05 total 10.01 last 76.39) - open/close 1388 (time 1789140656.05 total 20.02 last 62.29) - open/close 2012 (time 1789140666.06 total 30.03 last 62.31) - open/close 2743 (time 1789140676.07 total 40.04 last 73.04) - open/close 3437 (time 1789140686.10 total 50.06 last 69.24) - open/close 4195 (time 1789140696.10 total 60.07 last 75.74) total: 4500 open/close in 64.02 seconds: 70.29 ops/second - /unlink 1340 (time 1789140714.75 total 10.00 last 133.94) - /unlink 2568 (time 1789140724.75 total 20.01 last 122.78) - /unlink 4000 (time 1789140734.75 total 30.01 last 143.13) total: 4500 /unlink in 33.78 seconds: 133.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 516 (time 1789140753.63 total 10.01 last 51.57) - open/close 1074 (time 1789140763.64 total 20.02 last 55.72) - open/close 1605 (time 1789140773.66 total 30.04 last 53.01) - open/close 2225 (time 1789140783.66 total 40.04 last 61.99) - open/close 2862 (time 1789140793.66 total 50.04 last 63.66) - open/close 3321 (time 1789140803.67 total 60.05 last 45.89) - open/close 3949 (time 1789140813.68 total 70.06 last 62.73) total: 4500 open/close in 79.31 seconds: 56.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1089 (time 1789140838.42 total 10.00 last 108.88) - /unlink 2404 (time 1789140848.43 total 20.00 last 131.46) - /unlink 3884 (time 1789140858.43 total 30.01 last 147.96) total: 4500 /unlink in 33.93 seconds: 132.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 606 (time 1789140878.24 total 10.01 last 60.57) - open/close 1200 (time 1789140888.25 total 20.01 last 59.38) - open/close 1821 (time 1789140898.25 total 30.02 last 62.05) - open/close 2410 (time 1789140908.27 total 40.03 last 58.82) - open/close 3132 (time 1789140918.27 total 50.03 last 72.18) - open/close 3949 (time 1789140928.28 total 60.05 last 81.58) total: 4500 open/close in 68.43 seconds: 65.76 ops/second - /unlink 1468 (time 1789140951.75 total 10.01 last 146.70) - /unlink 2915 (time 1789140961.76 total 20.02 last 144.57) - /unlink 4321 (time 1789140971.76 total 30.02 last 140.53) total: 4500 /unlink in 31.60 seconds: 142.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 625 (time 1789140987.81 total 10.01 last 62.47) - open/close 1334 (time 1789140997.82 total 20.01 last 70.85) - open/close 2007 (time 1789141007.82 total 30.02 last 67.26) - open/close 2673 (time 1789141017.86 total 40.05 last 66.38) - open/close 3270 (time 1789141027.86 total 50.06 last 59.68) - open/close 3910 (time 1789141037.88 total 60.07 last 63.89) total: 4500 open/close in 67.93 seconds: 66.25 ops/second - /unlink 1521 (time 1789141060.58 total 10.00 last 152.09) - /unlink 2909 (time 1789141070.59 total 20.00 last 138.77) - /unlink 4292 (time 1789141080.59 total 30.00 last 138.30) total: 4500 /unlink in 31.53 seconds: 142.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 791 (time 1789141097.48 total 10.01 last 79.01) - open/close 1346 (time 1789141107.49 total 20.02 last 55.47) - open/close 2052 (time 1789141117.50 total 30.03 last 70.53) - open/close 2696 (time 1789141127.51 total 40.04 last 64.34) - open/close 3357 (time 1789141137.52 total 50.05 last 66.04) - open/close 4079 (time 1789141147.52 total 60.05 last 72.18) total: 4500 open/close in 65.65 seconds: 68.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1481 (time 1789141167.90 total 10.00 last 148.06) - /unlink 2470 (time 1789141177.91 total 20.01 last 98.81) - /unlink 3356 (time 1789141187.92 total 30.01 last 88.58) - /unlink 4479 (time 1789141197.92 total 40.02 last 112.29) total: 4500 /unlink in 40.13 seconds: 112.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 679 (time 1789141214.49 total 10.01 last 67.84) - open/close 1472 (time 1789141224.50 total 20.02 last 79.25) - open/close 2346 (time 1789141234.52 total 30.04 last 87.23) - open/close 3106 (time 1789141244.52 total 40.04 last 75.95) - open/close 3901 (time 1789141254.53 total 50.05 last 79.46) total: 4500 open/close in 57.26 seconds: 78.59 ops/second - /unlink 1245 (time 1789141277.13 total 10.01 last 124.41) - /unlink 2610 (time 1789141287.13 total 20.01 last 136.44) - /unlink 3833 (time 1789141297.14 total 30.02 last 122.22) total: 4500 /unlink in 35.87 seconds: 125.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 692 (time 1789141319.42 total 10.04 last 68.95) - open/close 1316 (time 1789141329.43 total 20.05 last 62.31) - open/close 2041 (time 1789141339.43 total 30.05 last 72.48) - open/close 2785 (time 1789141349.45 total 40.07 last 74.31) - open/close 3539 (time 1789141359.45 total 50.07 last 75.39) - open/close 4323 (time 1789141369.45 total 60.07 last 78.40) total: 4500 open/close in 62.76 seconds: 71.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1469 (time 1789141387.03 total 10.00 last 146.83) - /unlink 2764 (time 1789141397.04 total 20.01 last 129.42) - /unlink 4087 (time 1789141407.06 total 30.03 last 132.06) total: 4500 /unlink in 33.57 seconds: 134.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 645 (time 1789141426.88 total 10.02 last 64.36) - open/close 1245 (time 1789141436.89 total 20.03 last 59.97) - open/close 2041 (time 1789141446.89 total 30.03 last 79.60) - open/close 2817 (time 1789141456.89 total 40.04 last 77.54) - open/close 3509 (time 1789141466.91 total 50.06 last 69.07) - open/close 4310 (time 1789141476.91 total 60.06 last 80.10) total: 4500 open/close in 62.35 seconds: 72.18 ops/second - /unlink 1450 (time 1789141493.25 total 10.00 last 144.95) - /unlink 2709 (time 1789141503.25 total 20.00 last 125.90) - /unlink 4022 (time 1789141513.25 total 30.01 last 131.25) total: 4500 /unlink in 33.23 seconds: 135.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 661 (time 1789141532.43 total 10.00 last 66.07) - open/close 1413 (time 1789141542.44 total 20.01 last 75.13) - open/close 2113 (time 1789141552.44 total 30.02 last 69.98) - open/close 2822 (time 1789141562.44 total 40.02 last 70.88) - open/close 3551 (time 1789141572.45 total 50.03 last 72.85) - open/close 4289 (time 1789141582.45 total 60.03 last 73.79) total: 4500 open/close in 62.81 seconds: 71.64 ops/second - /unlink 1327 (time 1789141599.74 total 10.01 last 132.55) - /unlink 2569 (time 1789141609.75 total 20.02 last 124.14) - /unlink 3919 (time 1789141619.75 total 30.02 last 134.99) total: 4500 /unlink in 33.76 seconds: 133.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 679 (time 1789141639.54 total 10.00 last 67.90) - open/close 1237 (time 1789141649.55 total 20.01 last 55.74) - open/close 1903 (time 1789141659.56 total 30.02 last 66.51) - open/close 2621 (time 1789141669.56 total 40.02 last 71.80) - open/close 3277 (time 1789141679.57 total 50.03 last 65.55) - open/close 3975 (time 1789141689.60 total 60.06 last 69.60) total: 4500 open/close in 67.93 seconds: 66.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1328 (time 1789141712.93 total 10.01 last 132.69) - /unlink 2572 (time 1789141722.93 total 20.01 last 124.40) - /unlink 3757 (time 1789141732.93 total 30.01 last 118.48) total: 4500 /unlink in 35.08 seconds: 128.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 708 (time 1789141753.96 total 10.00 last 70.79) - open/close 1468 (time 1789141763.97 total 20.01 last 75.93) - open/close 2258 (time 1789141773.98 total 30.02 last 78.93) - open/close 3006 (time 1789141783.99 total 40.02 last 74.76) - open/close 3631 (time 1789141793.99 total 50.03 last 62.46) - open/close 4350 (time 1789141804.00 total 60.04 last 71.85) total: 4500 open/close in 62.22 seconds: 72.33 ops/second - /unlink 1344 (time 1789141820.62 total 10.00 last 134.40) - /unlink 2684 (time 1789141830.62 total 20.00 last 134.00) - /unlink 4022 (time 1789141840.63 total 30.00 last 133.77) total: 4500 /unlink in 34.06 seconds: 132.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 694 (time 1789141859.76 total 10.00 last 69.39) - open/close 1237 (time 1789141869.79 total 20.03 last 54.13) - open/close 1770 (time 1789141879.80 total 30.05 last 53.23) - open/close 2349 (time 1789141889.81 total 40.06 last 57.83) - open/close 3112 (time 1789141899.82 total 50.06 last 76.27) - open/close 3787 (time 1789141909.82 total 60.06 last 67.49) total: 4500 open/close in 68.45 seconds: 65.74 ops/second - /unlink 1392 (time 1789141933.01 total 10.01 last 139.11) - /unlink 2862 (time 1789141943.02 total 20.01 last 146.94) - /unlink 4271 (time 1789141953.02 total 30.01 last 140.88) total: 4500 /unlink in 31.96 seconds: 140.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 741 (time 1789141970.44 total 10.01 last 74.02) - open/close 1381 (time 1789141980.45 total 20.02 last 63.95) - open/close 1948 (time 1789141990.46 total 30.02 last 56.67) - open/close 2516 (time 1789142000.47 total 40.03 last 56.73) - open/close 3236 (time 1789142010.47 total 50.04 last 71.95) - open/close 3957 (time 1789142020.48 total 60.05 last 72.04) total: 4500 open/close in 68.34 seconds: 65.85 ops/second - /unlink 1213 (time 1789142044.55 total 10.02 last 121.04) - /unlink 2687 (time 1789142054.55 total 20.02 last 147.39) - /unlink 3998 (time 1789142064.56 total 30.03 last 131.02) total: 4500 /unlink in 34.30 seconds: 131.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 661 (time 1789142084.28 total 10.01 last 66.06) - open/close 1354 (time 1789142094.28 total 20.01 last 69.26) - open/close 2094 (time 1789142104.29 total 30.02 last 73.94) - open/close 2588 (time 1789142114.30 total 40.03 last 49.36) - open/close 3251 (time 1789142124.31 total 50.04 last 66.21) - open/close 3905 (time 1789142134.32 total 60.05 last 65.34) total: 4500 open/close in 69.65 seconds: 64.61 ops/second - /unlink 989 (time 1789142158.86 total 10.00 last 98.87) - /unlink 2186 (time 1789142168.87 total 20.01 last 119.59) - /unlink 3288 (time 1789142178.87 total 30.01 last 110.17) - /unlink 4401 (time 1789142188.87 total 40.02 last 111.28) total: 4500 /unlink in 41.12 seconds: 109.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 638 (time 1789142205.67 total 10.00 last 63.80) - open/close 1293 (time 1789142215.67 total 20.00 last 65.49) - open/close 1954 (time 1789142225.68 total 30.01 last 66.05) - open/close 2723 (time 1789142235.68 total 40.02 last 76.83) - open/close 3503 (time 1789142245.69 total 50.02 last 78.00) - open/close 4197 (time 1789142255.70 total 60.04 last 69.26) total: 4500 open/close in 64.46 seconds: 69.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1367 (time 1789142274.66 total 10.00 last 136.68) - /unlink 2581 (time 1789142284.67 total 20.01 last 121.27) - /unlink 3966 (time 1789142294.67 total 30.02 last 138.39) total: 4500 /unlink in 34.30 seconds: 131.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 778 (time 1789142314.14 total 10.02 last 77.66) - open/close 1431 (time 1789142324.16 total 20.03 last 65.21) - open/close 2269 (time 1789142334.17 total 30.04 last 83.73) - open/close 3007 (time 1789142344.17 total 40.04 last 73.78) - open/close 3551 (time 1789142354.18 total 50.06 last 54.33) - open/close 4292 (time 1789142364.19 total 60.06 last 74.04) total: 4500 open/close in 62.86 seconds: 71.58 ops/second - /unlink 1357 (time 1789142381.66 total 10.01 last 135.54) - /unlink 2629 (time 1789142391.67 total 20.02 last 127.10) - /unlink 4060 (time 1789142401.68 total 30.03 last 142.95) total: 4500 /unlink in 33.29 seconds: 135.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 631 (time 1789142420.95 total 10.05 last 62.79) - open/close 1297 (time 1789142430.96 total 20.06 last 66.51) - open/close 1879 (time 1789142440.97 total 30.07 last 58.17) - open/close 2499 (time 1789142450.97 total 40.07 last 61.99) - open/close 3168 (time 1789142460.98 total 50.08 last 66.82) - open/close 3754 (time 1789142471.00 total 60.10 last 58.51) - open/close 4327 (time 1789142481.02 total 70.12 last 57.18) total: 4500 open/close in 72.45 seconds: 62.11 ops/second - /unlink 1131 (time 1789142498.29 total 10.00 last 113.10) - /unlink 2254 (time 1789142508.30 total 20.01 last 112.23) - /unlink 3685 (time 1789142518.30 total 30.01 last 143.00) total: 4500 /unlink in 36.56 seconds: 123.10 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95349 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: modules unloaded.