== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:58:03 (1788573483) Stopping clients: oleg621-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg621-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg621-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg621-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg621-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg621-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg621-server oleg621-server: oleg621-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing load_modules_local oleg621-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg621-server: Loading modules from /home/green/git/lustre-release/lustre oleg621-server: detected 4 online CPUs by sysfs oleg621-server: MODOPTS_LIBCFS= oleg621-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg621-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg621-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg621-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 721 (time 1788573698.19 total 10.01 last 72.04) - open/close 1480 (time 1788573708.21 total 20.02 last 75.80) - open/close 2251 (time 1788573718.21 total 30.03 last 77.07) - open/close 3022 (time 1788573728.22 total 40.04 last 77.01) - open/close 3767 (time 1788573738.23 total 50.04 last 74.45) total: 4500 open/close in 59.80 seconds: 75.25 ops/second - /unlink 1363 (time 1788573762.61 total 10.00 last 136.29) - /unlink 2638 (time 1788573772.61 total 20.00 last 127.46) - /unlink 3931 (time 1788573782.62 total 30.01 last 129.21) total: 4500 /unlink in 34.23 seconds: 131.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 695 (time 1788573801.38 total 10.01 last 69.40) - open/close 1352 (time 1788573811.39 total 20.02 last 65.65) - open/close 2013 (time 1788573821.40 total 30.04 last 65.99) - open/close 2751 (time 1788573831.41 total 40.04 last 73.77) - open/close 3491 (time 1788573841.42 total 50.05 last 73.92) - open/close 4289 (time 1788573851.42 total 60.06 last 79.77) total: 4500 open/close in 62.89 seconds: 71.56 ops/second - /unlink 1350 (time 1788573868.64 total 10.00 last 134.95) - /unlink 2739 (time 1788573878.64 total 20.00 last 138.90) - /unlink 3989 (time 1788573888.64 total 30.01 last 124.98) total: 4500 /unlink in 34.06 seconds: 132.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 662 (time 1788573907.24 total 10.02 last 66.09) - open/close 1286 (time 1788573917.24 total 20.02 last 62.38) - open/close 1886 (time 1788573927.25 total 30.02 last 59.97) - open/close 2570 (time 1788573937.26 total 40.03 last 68.35) - open/close 3302 (time 1788573947.27 total 50.05 last 73.09) - open/close 3929 (time 1788573957.27 total 60.05 last 62.70) total: 4500 open/close in 69.40 seconds: 64.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1220 (time 1788573981.69 total 10.01 last 121.82) - /unlink 2325 (time 1788573991.70 total 20.02 last 110.39) - /unlink 3518 (time 1788574001.70 total 30.03 last 119.29) total: 4500 /unlink in 37.16 seconds: 121.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 752 (time 1788574023.72 total 10.00 last 75.17) - open/close 1498 (time 1788574033.72 total 20.01 last 74.55) - open/close 2201 (time 1788574043.73 total 30.02 last 70.22) - open/close 2784 (time 1788574053.73 total 40.02 last 58.30) - open/close 3477 (time 1788574063.75 total 50.03 last 69.21) - open/close 4237 (time 1788574073.76 total 60.05 last 75.91) total: 4500 open/close in 63.21 seconds: 71.19 ops/second - /unlink 1314 (time 1788574091.35 total 10.01 last 131.31) - /unlink 2665 (time 1788574101.36 total 20.01 last 135.00) - /unlink 3932 (time 1788574111.36 total 30.02 last 126.68) total: 4500 /unlink in 34.18 seconds: 131.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 795 (time 1788574130.58 total 10.01 last 79.43) - open/close 1524 (time 1788574140.58 total 20.01 last 72.90) - open/close 2129 (time 1788574150.59 total 30.02 last 60.41) - open/close 2756 (time 1788574160.61 total 40.04 last 62.61) - open/close 3411 (time 1788574170.61 total 50.04 last 65.48) - open/close 4013 (time 1788574180.62 total 60.05 last 60.16) total: 4500 open/close in 66.33 seconds: 67.84 ops/second - /unlink 1385 (time 1788574201.08 total 10.01 last 138.34) - /unlink 2696 (time 1788574211.08 total 20.01 last 131.06) - /unlink 4054 (time 1788574221.08 total 30.02 last 135.76) total: 4500 /unlink in 33.19 seconds: 135.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 724 (time 1788574238.89 total 10.01 last 72.36) - open/close 1415 (time 1788574248.89 total 20.01 last 69.09) - open/close 2104 (time 1788574258.91 total 30.03 last 68.75) - open/close 2779 (time 1788574268.92 total 40.04 last 67.43) - open/close 3536 (time 1788574278.92 total 50.04 last 75.70) - open/close 4190 (time 1788574288.94 total 60.05 last 65.30) total: 4500 open/close in 64.44 seconds: 69.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1266 (time 1788574307.88 total 10.01 last 126.48) - /unlink 2320 (time 1788574317.89 total 20.02 last 105.28) - /unlink 3486 (time 1788574327.89 total 30.03 last 116.54) total: 4500 /unlink in 39.79 seconds: 113.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 468 (time 1788574354.27 total 10.00 last 46.79) - open/close 1118 (time 1788574364.28 total 20.01 last 64.93) - open/close 1775 (time 1788574374.28 total 30.01 last 65.69) - open/close 2417 (time 1788574384.29 total 40.03 last 64.12) - open/close 3053 (time 1788574394.30 total 50.03 last 63.55) - open/close 3712 (time 1788574404.31 total 60.05 last 65.82) - open/close 4338 (time 1788574414.33 total 70.07 last 62.46) total: 4500 open/close in 73.10 seconds: 61.56 ops/second - /unlink 1301 (time 1788574432.66 total 10.00 last 130.05) - /unlink 2691 (time 1788574442.66 total 20.01 last 138.92) - /unlink 4016 (time 1788574452.67 total 30.01 last 132.46) total: 4500 /unlink in 33.27 seconds: 135.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 713 (time 1788574470.94 total 10.01 last 71.23) - open/close 1410 (time 1788574480.95 total 20.02 last 69.64) - open/close 2205 (time 1788574490.95 total 30.02 last 79.49) - open/close 2945 (time 1788574500.96 total 40.03 last 73.91) - open/close 3708 (time 1788574510.96 total 50.03 last 76.28) - open/close 4422 (time 1788574520.97 total 60.04 last 71.36) total: 4500 open/close in 60.91 seconds: 73.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1301 (time 1788574536.68 total 10.00 last 130.09) - /unlink 2570 (time 1788574546.68 total 20.00 last 126.89) - /unlink 3930 (time 1788574556.68 total 30.00 last 135.97) total: 4500 /unlink in 34.15 seconds: 131.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 693 (time 1788574575.92 total 10.00 last 69.28) - open/close 1290 (time 1788574585.92 total 20.01 last 59.67) - open/close 1823 (time 1788574595.94 total 30.03 last 53.21) - open/close 2453 (time 1788574605.96 total 40.05 last 62.88) - open/close 3151 (time 1788574615.96 total 50.05 last 69.77) - open/close 3871 (time 1788574625.97 total 60.06 last 71.96) total: 4500 open/close in 69.73 seconds: 64.53 ops/second - /unlink 1282 (time 1788574651.25 total 10.00 last 128.18) - /unlink 2401 (time 1788574661.26 total 20.01 last 111.84) - /unlink 3675 (time 1788574671.26 total 30.01 last 127.36) total: 4500 /unlink in 36.32 seconds: 123.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 670 (time 1788574692.40 total 10.01 last 66.93) - open/close 1329 (time 1788574702.40 total 20.02 last 65.87) - open/close 2047 (time 1788574712.41 total 30.02 last 71.74) - open/close 2749 (time 1788574722.43 total 40.04 last 70.07) - open/close 3447 (time 1788574732.45 total 50.06 last 69.67) - open/close 4101 (time 1788574742.46 total 60.07 last 65.32) total: 4500 open/close in 66.54 seconds: 67.63 ops/second - /unlink 1057 (time 1788574764.13 total 10.00 last 105.67) - /unlink 2328 (time 1788574774.13 total 20.00 last 127.08) - /unlink 3603 (time 1788574784.13 total 30.01 last 127.44) total: 4500 /unlink in 36.74 seconds: 122.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 670 (time 1788574805.18 total 10.01 last 66.94) - open/close 1342 (time 1788574815.19 total 20.02 last 67.14) - open/close 2093 (time 1788574825.19 total 30.02 last 75.08) - open/close 2772 (time 1788574835.20 total 40.03 last 67.84) - open/close 3512 (time 1788574845.21 total 50.04 last 73.92) - open/close 4242 (time 1788574855.21 total 60.04 last 72.99) total: 4500 open/close in 64.34 seconds: 69.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1284 (time 1788574873.89 total 10.01 last 128.28) - /unlink 2662 (time 1788574883.90 total 20.01 last 137.74) - /unlink 4050 (time 1788574893.90 total 30.02 last 138.79) total: 4500 /unlink in 33.33 seconds: 135.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 749 (time 1788574911.89 total 10.01 last 74.80) - open/close 1493 (time 1788574921.89 total 20.02 last 74.37) - open/close 2219 (time 1788574931.90 total 30.02 last 72.55) - open/close 2937 (time 1788574941.90 total 40.03 last 71.78) - open/close 3629 (time 1788574951.92 total 50.04 last 69.08) - open/close 4374 (time 1788574961.93 total 60.06 last 74.41) total: 4500 open/close in 61.98 seconds: 72.60 ops/second - /unlink 1227 (time 1788574978.78 total 10.00 last 122.65) - /unlink 2369 (time 1788574988.79 total 20.02 last 114.02) - /unlink 3451 (time 1788574998.80 total 30.03 last 108.13) - /unlink 4459 (time 1788575008.80 total 40.03 last 100.79) total: 4500 /unlink in 40.67 seconds: 110.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 576 (time 1788575025.59 total 10.04 last 57.36) - open/close 1240 (time 1788575035.60 total 20.05 last 66.37) - open/close 1943 (time 1788575045.60 total 30.05 last 70.29) - open/close 2661 (time 1788575055.60 total 40.05 last 71.80) - open/close 3364 (time 1788575065.60 total 50.05 last 70.28) - open/close 4022 (time 1788575075.61 total 60.06 last 65.75) total: 4500 open/close in 67.53 seconds: 66.63 ops/second - /unlink 1189 (time 1788575097.75 total 10.00 last 118.85) - /unlink 2320 (time 1788575107.76 total 20.02 last 112.93) - /unlink 3486 (time 1788575117.79 total 30.05 last 116.25) total: 4500 /unlink in 37.82 seconds: 118.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 671 (time 1788575141.21 total 10.01 last 67.04) - open/close 1296 (time 1788575151.22 total 20.02 last 62.40) - open/close 1932 (time 1788575161.25 total 30.05 last 63.42) - open/close 2560 (time 1788575171.26 total 40.06 last 62.78) - open/close 3176 (time 1788575181.26 total 50.06 last 61.56) - open/close 3865 (time 1788575191.28 total 60.08 last 68.76) total: 4500 open/close in 68.17 seconds: 66.01 ops/second - /unlink 1322 (time 1788575214.07 total 10.00 last 132.14) - /unlink 2475 (time 1788575224.07 total 20.01 last 115.24) - /unlink 3548 (time 1788575234.08 total 30.01 last 107.28) - /unlink 4433 (time 1788575244.08 total 40.02 last 88.47) total: 4500 /unlink in 41.04 seconds: 109.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 561 (time 1788575261.15 total 10.01 last 56.03) - open/close 1154 (time 1788575271.16 total 20.03 last 59.22) - open/close 1728 (time 1788575281.17 total 30.03 last 57.35) - open/close 2260 (time 1788575291.17 total 40.04 last 53.19) - open/close 2871 (time 1788575301.18 total 50.04 last 61.04) - open/close 3466 (time 1788575311.18 total 60.05 last 59.49) - open/close 3979 (time 1788575321.19 total 70.06 last 51.23) total: 4500 open/close in 78.11 seconds: 57.61 ops/second - /unlink 1169 (time 1788575344.61 total 10.01 last 116.76) - /unlink 2395 (time 1788575354.61 total 20.02 last 122.51) - /unlink 3653 (time 1788575364.61 total 30.02 last 125.79) total: 4500 /unlink in 36.93 seconds: 121.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 728 (time 1788575386.69 total 10.01 last 72.74) - open/close 1445 (time 1788575396.70 total 20.01 last 71.66) - open/close 2141 (time 1788575406.72 total 30.03 last 69.45) - open/close 2754 (time 1788575416.72 total 40.04 last 61.30) - open/close 3322 (time 1788575426.72 total 50.04 last 56.79) - open/close 3926 (time 1788575436.73 total 60.05 last 60.33) - open/close 4494 (time 1788575446.74 total 70.06 last 56.75) total: 4500 open/close in 70.12 seconds: 64.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1199 (time 1788575461.66 total 10.00 last 119.87) - /unlink 2440 (time 1788575471.66 total 20.00 last 124.10) - /unlink 3733 (time 1788575481.66 total 30.00 last 129.30) total: 4500 /unlink in 36.74 seconds: 122.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 725 (time 1788575503.53 total 10.01 last 72.44) - open/close 1355 (time 1788575513.54 total 20.02 last 62.95) - open/close 2037 (time 1788575523.56 total 30.04 last 68.04) - open/close 2780 (time 1788575533.57 total 40.05 last 74.25) - open/close 3430 (time 1788575543.57 total 50.05 last 64.99) - open/close 4115 (time 1788575553.58 total 60.05 last 68.46) total: 4500 open/close in 65.58 seconds: 68.62 ops/second - /unlink 1359 (time 1788575573.83 total 10.00 last 135.88) - /unlink 2515 (time 1788575583.83 total 20.01 last 115.56) - /unlink 3791 (time 1788575593.84 total 30.01 last 127.52) total: 4500 /unlink in 36.56 seconds: 123.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 687 (time 1788575616.11 total 10.00 last 68.69) - open/close 1296 (time 1788575626.12 total 20.01 last 60.85) - open/close 2001 (time 1788575636.13 total 30.02 last 70.45) - open/close 2650 (time 1788575646.13 total 40.02 last 64.87) - open/close 3244 (time 1788575656.13 total 50.02 last 59.40) - open/close 3965 (time 1788575666.15 total 60.04 last 71.96) total: 4500 open/close in 68.13 seconds: 66.05 ops/second - /unlink 1252 (time 1788575688.96 total 10.00 last 125.16) - /unlink 2543 (time 1788575698.96 total 20.01 last 129.07) - /unlink 3918 (time 1788575708.97 total 30.01 last 137.42) total: 4500 /unlink in 34.46 seconds: 130.57 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95367 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg621-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg621-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg621-server stop mds service on oleg621-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg621-server stop mds service on oleg621-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg621-server unloading modules via unload_modules_local on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing unload_modules_local oleg621-server: modules unloaded.