== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:20:21 (1787401221) Stopping clients: oleg619-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg619-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg619-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg619-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg619-server oleg619-server: oleg619-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing load_modules_local oleg619-server: Loading modules from /home/green/git/lustre-release/lustre oleg619-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg619-server: detected 4 online CPUs by sysfs oleg619-server: MODOPTS_LIBCFS= oleg619-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg619-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg619-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg619-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug -1 all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 753 (time 1787401411.60 total 10.00 last 75.30) - open/close 1495 (time 1787401421.61 total 20.01 last 74.10) - open/close 2325 (time 1787401431.61 total 30.02 last 82.98) - open/close 3068 (time 1787401441.61 total 40.02 last 74.29) - open/close 3860 (time 1787401451.62 total 50.02 last 79.16) total: 4500 open/close in 57.56 seconds: 78.18 ops/second - /unlink 1413 (time 1787401474.14 total 10.01 last 141.22) - /unlink 2831 (time 1787401484.15 total 20.01 last 141.78) - /unlink 4238 (time 1787401494.15 total 30.01 last 140.66) total: 4500 /unlink in 32.06 seconds: 140.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 795 (time 1787401510.22 total 10.02 last 79.31) - open/close 1560 (time 1787401520.22 total 20.02 last 76.50) - open/close 2368 (time 1787401530.23 total 30.04 last 80.71) - open/close 3120 (time 1787401540.25 total 40.05 last 75.07) - open/close 3781 (time 1787401550.25 total 50.06 last 66.08) total: 4500 open/close in 59.43 seconds: 75.72 ops/second - /unlink 1346 (time 1787401574.31 total 10.01 last 134.49) - /unlink 2570 (time 1787401584.32 total 20.01 last 122.36) - /unlink 3873 (time 1787401594.32 total 30.02 last 130.24) total: 4500 /unlink in 34.22 seconds: 131.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 744 (time 1787401613.55 total 10.01 last 74.33) - open/close 1484 (time 1787401623.55 total 20.02 last 73.95) - open/close 2238 (time 1787401633.56 total 30.02 last 75.38) - open/close 2974 (time 1787401643.59 total 40.05 last 73.38) - open/close 3650 (time 1787401653.60 total 50.07 last 67.48) - open/close 4310 (time 1787401663.62 total 60.08 last 65.92) total: 4500 open/close in 62.37 seconds: 72.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1178 (time 1787401680.88 total 10.00 last 117.79) - /unlink 2430 (time 1787401690.88 total 20.00 last 125.20) - /unlink 3548 (time 1787401700.88 total 30.00 last 111.79) total: 4500 /unlink in 36.50 seconds: 123.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 608 (time 1787401723.61 total 10.01 last 60.71) - open/close 1129 (time 1787401733.62 total 20.02 last 52.08) - open/close 1761 (time 1787401743.62 total 30.03 last 63.15) - open/close 2453 (time 1787401753.64 total 40.04 last 69.11) - open/close 3077 (time 1787401763.64 total 50.05 last 62.36) - open/close 3792 (time 1787401773.65 total 60.05 last 71.44) - open/close 4383 (time 1787401783.65 total 70.06 last 59.08) total: 4500 open/close in 71.64 seconds: 62.82 ops/second - /unlink 1235 (time 1787401800.14 total 10.00 last 123.45) - /unlink 2416 (time 1787401810.15 total 20.01 last 118.06) - /unlink 3564 (time 1787401820.15 total 30.01 last 114.73) total: 4500 /unlink in 37.54 seconds: 119.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 768 (time 1787401842.68 total 10.01 last 76.73) - open/close 1501 (time 1787401852.70 total 20.03 last 73.14) - open/close 2105 (time 1787401862.70 total 30.03 last 60.40) - open/close 2760 (time 1787401872.71 total 40.04 last 65.47) - open/close 3451 (time 1787401882.72 total 50.04 last 69.04) - open/close 4208 (time 1787401892.72 total 60.05 last 75.66) total: 4500 open/close in 64.11 seconds: 70.19 ops/second - /unlink 1081 (time 1787401912.91 total 10.04 last 107.71) - /unlink 1958 (time 1787401922.92 total 20.04 last 87.67) - /unlink 2917 (time 1787401932.92 total 30.04 last 95.88) - /unlink 4018 (time 1787401942.92 total 40.04 last 110.07) total: 4500 /unlink in 44.45 seconds: 101.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 704 (time 1787401961.75 total 10.00 last 70.39) - open/close 1368 (time 1787401971.76 total 20.01 last 66.36) - open/close 2015 (time 1787401981.77 total 30.02 last 64.64) - open/close 2606 (time 1787401991.77 total 40.03 last 59.05) - open/close 3206 (time 1787402001.78 total 50.03 last 59.99) - open/close 3920 (time 1787402011.78 total 60.04 last 71.34) total: 4500 open/close in 68.57 seconds: 65.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1239 (time 1787402035.29 total 10.00 last 123.87) - /unlink 2497 (time 1787402045.29 total 20.00 last 125.77) - /unlink 3788 (time 1787402055.29 total 30.01 last 129.04) total: 4500 /unlink in 35.04 seconds: 128.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 644 (time 1787402075.10 total 10.00 last 64.39) - open/close 1283 (time 1787402085.10 total 20.00 last 63.89) - open/close 2016 (time 1787402095.11 total 30.01 last 73.27) - open/close 2739 (time 1787402105.12 total 40.01 last 72.25) - open/close 3393 (time 1787402115.13 total 50.02 last 65.33) - open/close 4122 (time 1787402125.13 total 60.03 last 72.88) total: 4500 open/close in 65.42 seconds: 68.79 ops/second - /unlink 1203 (time 1787402144.67 total 10.00 last 120.27) - /unlink 2524 (time 1787402154.67 total 20.01 last 132.02) - /unlink 3746 (time 1787402164.69 total 30.02 last 122.06) total: 4500 /unlink in 35.49 seconds: 126.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 673 (time 1787402185.74 total 10.00 last 67.29) - open/close 1369 (time 1787402195.75 total 20.01 last 69.53) - open/close 1993 (time 1787402205.76 total 30.02 last 62.35) - open/close 2668 (time 1787402215.77 total 40.03 last 67.44) - open/close 3344 (time 1787402225.78 total 50.04 last 67.51) - open/close 4094 (time 1787402235.78 total 60.04 last 74.99) total: 4500 open/close in 65.65 seconds: 68.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1300 (time 1787402256.26 total 10.00 last 129.94) - /unlink 2594 (time 1787402266.26 total 20.01 last 129.35) - /unlink 3640 (time 1787402276.26 total 30.01 last 104.57) total: 4500 /unlink in 39.89 seconds: 112.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 599 (time 1787402302.66 total 10.01 last 59.87) - open/close 1231 (time 1787402312.66 total 20.01 last 63.20) - open/close 1946 (time 1787402322.69 total 30.03 last 71.33) - open/close 2611 (time 1787402332.69 total 40.03 last 66.50) - open/close 3294 (time 1787402342.70 total 50.04 last 68.22) - open/close 4005 (time 1787402352.72 total 60.06 last 70.95) total: 4500 open/close in 67.42 seconds: 66.74 ops/second - /unlink 1172 (time 1787402375.02 total 10.00 last 117.19) - /unlink 2383 (time 1787402385.02 total 20.00 last 121.07) - /unlink 3762 (time 1787402395.02 total 30.01 last 137.85) total: 4500 /unlink in 35.55 seconds: 126.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 732 (time 1787402415.32 total 10.02 last 73.09) - open/close 1457 (time 1787402425.32 total 20.02 last 72.47) - open/close 2203 (time 1787402435.33 total 30.02 last 74.57) - open/close 2888 (time 1787402445.33 total 40.03 last 68.49) - open/close 3558 (time 1787402455.35 total 50.05 last 66.84) - open/close 4087 (time 1787402465.35 total 60.05 last 52.89) total: 4500 open/close in 68.93 seconds: 65.28 ops/second - /unlink 1200 (time 1787402490.61 total 10.00 last 119.97) - /unlink 2419 (time 1787402500.62 total 20.01 last 121.86) - /unlink 3731 (time 1787402510.62 total 30.01 last 131.17) total: 4500 /unlink in 36.15 seconds: 124.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 745 (time 1787402531.55 total 10.00 last 74.49) - open/close 1537 (time 1787402541.55 total 20.01 last 79.17) - open/close 2265 (time 1787402551.55 total 30.01 last 72.76) - open/close 3016 (time 1787402561.56 total 40.02 last 75.02) - open/close 3806 (time 1787402571.57 total 50.02 last 78.97) total: 4500 open/close in 59.32 seconds: 75.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1288 (time 1787402595.47 total 10.00 last 128.78) - /unlink 2634 (time 1787402605.47 total 20.00 last 134.57) - /unlink 3944 (time 1787402615.47 total 30.01 last 130.96) total: 4500 /unlink in 34.24 seconds: 131.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 731 (time 1787402634.06 total 10.01 last 73.06) - open/close 1536 (time 1787402644.06 total 20.01 last 80.47) - open/close 2209 (time 1787402654.07 total 30.02 last 67.26) - open/close 2938 (time 1787402664.07 total 40.02 last 72.88) - open/close 3677 (time 1787402674.08 total 50.03 last 73.84) - open/close 4440 (time 1787402684.08 total 60.03 last 76.24) total: 4500 open/close in 60.80 seconds: 74.01 ops/second - /unlink 1557 (time 1787402698.51 total 10.00 last 155.69) - /unlink 2991 (time 1787402708.52 total 20.01 last 143.24) - /unlink 4481 (time 1787402718.53 total 30.02 last 148.81) total: 4500 /unlink in 30.23 seconds: 148.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 806 (time 1787402732.58 total 10.00 last 80.58) - open/close 1586 (time 1787402742.61 total 20.04 last 77.71) - open/close 2351 (time 1787402752.61 total 30.04 last 76.48) - open/close 3080 (time 1787402762.62 total 40.05 last 72.84) - open/close 3835 (time 1787402772.63 total 50.06 last 75.42) total: 4500 open/close in 59.19 seconds: 76.03 ops/second - /unlink 1381 (time 1787402795.73 total 10.01 last 138.03) - /unlink 2666 (time 1787402805.73 total 20.01 last 128.49) - /unlink 4027 (time 1787402815.73 total 30.01 last 136.09) total: 4500 /unlink in 33.64 seconds: 133.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 777 (time 1787402833.45 total 10.01 last 77.64) - open/close 1500 (time 1787402843.46 total 20.02 last 72.23) - open/close 2274 (time 1787402853.47 total 30.03 last 77.31) - open/close 2921 (time 1787402863.48 total 40.03 last 64.67) - open/close 3480 (time 1787402873.48 total 50.04 last 55.86) - open/close 4049 (time 1787402883.49 total 60.05 last 56.87) total: 4500 open/close in 66.43 seconds: 67.74 ops/second - /unlink 1247 (time 1787402904.42 total 10.01 last 124.64) - /unlink 2545 (time 1787402914.42 total 20.01 last 129.78) - /unlink 3927 (time 1787402924.42 total 30.01 last 138.16) total: 4500 /unlink in 33.91 seconds: 132.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 737 (time 1787402942.22 total 10.01 last 73.65) - open/close 1496 (time 1787402952.22 total 20.01 last 75.87) - open/close 2187 (time 1787402962.22 total 30.01 last 69.10) - open/close 2816 (time 1787402972.23 total 40.02 last 62.84) - open/close 3466 (time 1787402982.24 total 50.03 last 64.95) - open/close 4169 (time 1787402992.26 total 60.04 last 70.19) total: 4500 open/close in 64.55 seconds: 69.71 ops/second - /unlink 1303 (time 1787403011.67 total 10.00 last 130.29) - /unlink 2695 (time 1787403021.67 total 20.00 last 139.20) - /unlink 4065 (time 1787403031.67 total 30.01 last 136.91) total: 4500 /unlink in 33.64 seconds: 133.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 581 (time 1787403050.34 total 10.00 last 58.09) - open/close 1169 (time 1787403060.37 total 20.03 last 58.64) - open/close 1828 (time 1787403070.37 total 30.03 last 65.86) - open/close 2402 (time 1787403080.38 total 40.04 last 57.38) - open/close 3017 (time 1787403090.40 total 50.06 last 61.39) - open/close 3681 (time 1787403100.41 total 60.07 last 66.33) - open/close 4394 (time 1787403110.41 total 70.07 last 71.24) total: 4500 open/close in 71.57 seconds: 62.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1275 (time 1787403125.75 total 10.00 last 127.49) - /unlink 2622 (time 1787403135.75 total 20.01 last 134.62) - /unlink 3896 (time 1787403145.76 total 30.01 last 127.37) total: 4500 /unlink in 34.55 seconds: 130.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 720 (time 1787403164.85 total 10.01 last 71.95) - open/close 1461 (time 1787403174.85 total 20.01 last 74.07) - open/close 2146 (time 1787403184.86 total 30.02 last 68.44) - open/close 2866 (time 1787403194.86 total 40.02 last 71.98) - open/close 3601 (time 1787403204.86 total 50.03 last 73.49) - open/close 4297 (time 1787403214.87 total 60.03 last 69.58) total: 4500 open/close in 63.16 seconds: 71.25 ops/second - /unlink 1171 (time 1787403232.39 total 10.00 last 117.09) - /unlink 2387 (time 1787403242.40 total 20.01 last 121.43) - /unlink 3723 (time 1787403252.42 total 30.03 last 133.40) total: 4500 /unlink in 36.30 seconds: 123.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 596 (time 1787403274.17 total 10.01 last 59.55) - open/close 1230 (time 1787403284.17 total 20.01 last 63.40) - open/close 1936 (time 1787403294.18 total 30.01 last 70.59) - open/close 2665 (time 1787403304.18 total 40.01 last 72.90) - open/close 3393 (time 1787403314.21 total 50.04 last 72.58) - open/close 4045 (time 1787403324.21 total 60.05 last 65.15) total: 4500 open/close in 66.80 seconds: 67.37 ops/second - /unlink 1171 (time 1787403345.58 total 10.00 last 117.07) - /unlink 2407 (time 1787403355.58 total 20.01 last 123.55) - /unlink 3544 (time 1787403365.59 total 30.01 last 113.65) total: 4500 /unlink in 37.79 seconds: 119.08 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94846 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg619-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg619-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg619-server stop mds service on oleg619-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg619-server stop mds service on oleg619-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg619-server unloading modules via unload_modules_local on: 'oleg619-server' oleg619-server: oleg619-server.virtnet: executing unload_modules_local oleg619-server: modules unloaded.