== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:07:54 (1783141674) Stopping clients: oleg413-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg413-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg413-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg413-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg413-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg413-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg413-server oleg413-server: oleg413-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing load_modules_local oleg413-server: Loading modules from /home/green/git/lustre-release/lustre oleg413-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg413-server: detected 4 online CPUs by sysfs oleg413-server: MODOPTS_LIBCFS= oleg413-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg413-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg413-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg413-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 561 (time 1783141910.76 total 10.00 last 56.10) - open/close 1161 (time 1783141920.76 total 20.01 last 59.97) - open/close 1785 (time 1783141930.78 total 30.02 last 62.28) - open/close 2328 (time 1783141940.79 total 40.04 last 54.23) - open/close 2892 (time 1783141950.81 total 50.05 last 56.31) - open/close 3507 (time 1783141960.83 total 60.07 last 61.39) - open/close 4048 (time 1783141970.83 total 70.08 last 54.08) total: 4500 open/close in 77.17 seconds: 58.31 ops/second - /unlink 1052 (time 1783141992.84 total 10.00 last 105.19) - /unlink 2178 (time 1783142002.84 total 20.00 last 112.58) - /unlink 3250 (time 1783142012.85 total 30.01 last 107.12) - /unlink 4291 (time 1783142022.86 total 40.02 last 104.01) total: 4500 /unlink in 42.13 seconds: 106.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 613 (time 1783142040.82 total 10.01 last 61.27) - open/close 1219 (time 1783142050.83 total 20.02 last 60.54) - open/close 1858 (time 1783142060.84 total 30.02 last 63.89) - open/close 2429 (time 1783142070.84 total 40.02 last 57.09) - open/close 3024 (time 1783142080.84 total 50.02 last 59.48) - open/close 3625 (time 1783142090.85 total 60.03 last 60.04) - open/close 4234 (time 1783142100.89 total 70.07 last 60.69) total: 4500 open/close in 73.96 seconds: 60.84 ops/second - /unlink 1136 (time 1783142119.62 total 10.00 last 113.56) - /unlink 2237 (time 1783142129.62 total 20.00 last 110.09) - /unlink 3356 (time 1783142139.63 total 30.01 last 111.80) - /unlink 4442 (time 1783142149.65 total 40.03 last 108.44) total: 4500 /unlink in 40.78 seconds: 110.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 608 (time 1783142166.40 total 10.01 last 60.74) - open/close 1134 (time 1783142176.41 total 20.02 last 52.55) - open/close 1789 (time 1783142186.41 total 30.03 last 65.45) - open/close 2416 (time 1783142196.41 total 40.03 last 62.69) - open/close 3068 (time 1783142206.41 total 50.03 last 65.20) - open/close 3714 (time 1783142216.43 total 60.04 last 64.52) - open/close 4338 (time 1783142226.43 total 70.04 last 62.39) total: 4500 open/close in 72.46 seconds: 62.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1177 (time 1783142243.69 total 10.00 last 117.69) - /unlink 2351 (time 1783142253.69 total 20.00 last 117.36) - /unlink 3521 (time 1783142263.70 total 30.01 last 116.94) total: 4500 /unlink in 38.13 seconds: 118.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 670 (time 1783142287.44 total 10.00 last 66.98) - open/close 1301 (time 1783142297.45 total 20.01 last 63.06) - open/close 1942 (time 1783142307.46 total 30.02 last 64.02) - open/close 2568 (time 1783142317.47 total 40.02 last 62.59) - open/close 3191 (time 1783142327.50 total 50.05 last 62.11) - open/close 3871 (time 1783142337.51 total 60.06 last 67.93) - open/close 4448 (time 1783142347.51 total 70.07 last 57.66) total: 4500 open/close in 70.87 seconds: 63.50 ops/second - /unlink 1209 (time 1783142362.57 total 10.00 last 120.86) - /unlink 2379 (time 1783142372.58 total 20.00 last 116.99) - /unlink 3510 (time 1783142382.58 total 30.01 last 113.03) total: 4500 /unlink in 37.96 seconds: 118.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 679 (time 1783142404.86 total 10.01 last 67.80) - open/close 1345 (time 1783142414.87 total 20.02 last 66.56) - open/close 2013 (time 1783142424.88 total 30.03 last 66.75) - open/close 2681 (time 1783142434.89 total 40.04 last 66.70) - open/close 3380 (time 1783142444.90 total 50.06 last 69.82) - open/close 4089 (time 1783142454.91 total 60.06 last 70.85) total: 4500 open/close in 66.07 seconds: 68.11 ops/second - /unlink 1238 (time 1783142474.84 total 10.00 last 123.78) - /unlink 2400 (time 1783142484.85 total 20.01 last 116.08) - /unlink 3666 (time 1783142494.85 total 30.02 last 126.55) total: 4500 /unlink in 36.86 seconds: 122.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 715 (time 1783142516.09 total 10.00 last 71.48) - open/close 1389 (time 1783142526.10 total 20.02 last 67.29) - open/close 2051 (time 1783142536.12 total 30.03 last 66.12) - open/close 2743 (time 1783142546.12 total 40.03 last 69.17) - open/close 3399 (time 1783142556.13 total 50.04 last 65.56) - open/close 4077 (time 1783142566.15 total 60.06 last 67.65) total: 4500 open/close in 65.85 seconds: 68.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1313 (time 1783142586.14 total 10.00 last 131.28) - /unlink 2581 (time 1783142596.14 total 20.00 last 126.80) - /unlink 3794 (time 1783142606.14 total 30.00 last 121.27) total: 4500 /unlink in 35.37 seconds: 127.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 672 (time 1783142625.71 total 10.01 last 67.16) - open/close 1307 (time 1783142635.71 total 20.01 last 63.49) - open/close 1978 (time 1783142645.72 total 30.01 last 67.05) - open/close 2629 (time 1783142655.72 total 40.02 last 65.07) - open/close 3299 (time 1783142665.74 total 50.03 last 66.90) - open/close 3946 (time 1783142675.74 total 60.04 last 64.66) total: 4500 open/close in 68.15 seconds: 66.03 ops/second - /unlink 1364 (time 1783142697.30 total 10.00 last 136.40) - /unlink 2679 (time 1783142707.30 total 20.01 last 131.42) - /unlink 3980 (time 1783142717.30 total 30.01 last 130.09) total: 4500 /unlink in 33.76 seconds: 133.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 641 (time 1783142735.34 total 10.00 last 64.09) - open/close 1364 (time 1783142745.34 total 20.00 last 72.28) - open/close 2120 (time 1783142755.35 total 30.01 last 75.54) - open/close 2855 (time 1783142765.37 total 40.03 last 73.37) - open/close 3520 (time 1783142775.38 total 50.04 last 66.42) - open/close 4193 (time 1783142785.39 total 60.05 last 67.25) total: 4500 open/close in 64.57 seconds: 69.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1350 (time 1783142803.74 total 10.00 last 134.98) - /unlink 2647 (time 1783142813.75 total 20.01 last 129.62) - /unlink 3966 (time 1783142823.76 total 30.02 last 131.79) total: 4500 /unlink in 34.48 seconds: 130.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 656 (time 1783142842.46 total 10.02 last 65.50) - open/close 1363 (time 1783142852.47 total 20.02 last 70.64) - open/close 2026 (time 1783142862.49 total 30.05 last 66.15) - open/close 2698 (time 1783142872.51 total 40.07 last 67.05) - open/close 3400 (time 1783142882.51 total 50.07 last 70.19) - open/close 4118 (time 1783142892.52 total 60.08 last 71.75) total: 4500 open/close in 65.13 seconds: 69.10 ops/second - /unlink 1338 (time 1783142911.39 total 10.00 last 133.79) - /unlink 2670 (time 1783142921.39 total 20.00 last 133.16) - /unlink 3900 (time 1783142931.39 total 30.00 last 122.99) total: 4500 /unlink in 34.41 seconds: 130.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 710 (time 1783142950.10 total 10.01 last 70.96) - open/close 1436 (time 1783142960.11 total 20.02 last 72.53) - open/close 2113 (time 1783142970.13 total 30.03 last 67.57) - open/close 2828 (time 1783142980.13 total 40.04 last 71.48) - open/close 3577 (time 1783142990.14 total 50.04 last 74.86) - open/close 4370 (time 1783143000.15 total 60.05 last 79.22) total: 4500 open/close in 61.61 seconds: 73.04 ops/second - /unlink 1476 (time 1783143014.54 total 10.00 last 147.55) - /unlink 2787 (time 1783143024.54 total 20.00 last 131.09) - /unlink 4184 (time 1783143034.54 total 30.01 last 139.64) total: 4500 /unlink in 32.06 seconds: 140.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 693 (time 1783143050.03 total 10.01 last 69.23) - open/close 1384 (time 1783143060.05 total 20.02 last 69.00) - open/close 2130 (time 1783143070.05 total 30.03 last 74.59) - open/close 2855 (time 1783143080.06 total 40.04 last 72.42) - open/close 3566 (time 1783143090.07 total 50.05 last 71.04) - open/close 4364 (time 1783143100.07 total 60.05 last 79.79) total: 4500 open/close in 61.91 seconds: 72.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1441 (time 1783143115.50 total 10.01 last 143.94) - /unlink 2944 (time 1783143125.50 total 20.01 last 150.30) - /unlink 4454 (time 1783143135.50 total 30.01 last 150.96) total: 4500 /unlink in 30.55 seconds: 147.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 793 (time 1783143149.38 total 10.00 last 79.29) - open/close 1592 (time 1783143159.39 total 20.01 last 79.85) - open/close 2402 (time 1783143169.40 total 30.02 last 80.92) - open/close 3214 (time 1783143179.42 total 40.03 last 81.06) - open/close 4032 (time 1783143189.42 total 50.04 last 81.76) total: 4500 open/close in 55.86 seconds: 80.56 ops/second - /unlink 1390 (time 1783143208.63 total 10.01 last 138.85) - /unlink 2873 (time 1783143218.64 total 20.02 last 148.22) - /unlink 4401 (time 1783143228.65 total 30.03 last 152.65) total: 4500 /unlink in 30.83 seconds: 145.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 812 (time 1783143243.07 total 10.01 last 81.12) - open/close 1540 (time 1783143253.09 total 20.03 last 72.66) - open/close 2299 (time 1783143263.10 total 30.04 last 75.80) - open/close 3086 (time 1783143273.10 total 40.05 last 78.66) - open/close 3844 (time 1783143283.11 total 50.06 last 75.72) total: 4500 open/close in 59.00 seconds: 76.27 ops/second - /unlink 1423 (time 1783143305.96 total 10.00 last 142.23) - /unlink 2887 (time 1783143315.96 total 20.00 last 146.40) - /unlink 4345 (time 1783143325.96 total 30.01 last 145.78) total: 4500 /unlink in 31.12 seconds: 144.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 819 (time 1783143340.14 total 10.00 last 81.89) - open/close 1662 (time 1783143350.15 total 20.01 last 84.24) - open/close 2487 (time 1783143360.15 total 30.01 last 82.48) - open/close 3299 (time 1783143370.16 total 40.01 last 81.18) - open/close 4165 (time 1783143380.16 total 50.02 last 86.58) total: 4500 open/close in 53.79 seconds: 83.66 ops/second - /unlink 1669 (time 1783143396.84 total 10.00 last 166.89) - /unlink 3273 (time 1783143406.84 total 20.00 last 160.40) total: 4500 /unlink in 28.87 seconds: 155.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 791 (time 1783143429.22 total 10.00 last 79.08) - open/close 1602 (time 1783143439.23 total 20.02 last 80.99) - open/close 2451 (time 1783143449.23 total 30.02 last 84.90) - open/close 3316 (time 1783143459.24 total 40.03 last 86.42) - open/close 4158 (time 1783143469.24 total 50.03 last 84.18) total: 4500 open/close in 53.82 seconds: 83.61 ops/second - /unlink 1789 (time 1783143485.72 total 10.00 last 178.87) - /unlink 3841 (time 1783143495.72 total 20.00 last 205.18) total: 4500 /unlink in 23.44 seconds: 192.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1036 (time 1783143511.67 total 10.00 last 103.57) - open/close 2043 (time 1783143521.68 total 20.01 last 100.60) - open/close 3167 (time 1783143531.68 total 30.02 last 112.38) - open/close 4301 (time 1783143541.68 total 40.02 last 113.40) total: 4500 open/close in 41.80 seconds: 107.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1957 (time 1783143555.77 total 10.00 last 195.67) - /unlink 3763 (time 1783143565.77 total 20.00 last 180.59) total: 4500 /unlink in 24.42 seconds: 184.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 984 (time 1783143582.75 total 10.00 last 98.38) - open/close 2093 (time 1783143592.76 total 20.01 last 110.87) - open/close 3134 (time 1783143602.76 total 30.01 last 104.02) - open/close 4158 (time 1783143612.77 total 40.02 last 102.35) total: 4500 open/close in 43.27 seconds: 104.01 ops/second - /unlink 2170 (time 1783143628.50 total 10.00 last 216.98) - /unlink 4290 (time 1783143638.51 total 20.01 last 211.85) total: 4500 /unlink in 21.13 seconds: 212.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1066 (time 1783143652.15 total 10.01 last 106.53) - open/close 2207 (time 1783143662.15 total 20.01 last 114.07) - open/close 3410 (time 1783143672.16 total 30.02 last 120.19) - open/close 4457 (time 1783143682.16 total 40.02 last 104.66) total: 4500 open/close in 40.41 seconds: 111.35 ops/second - /unlink 2093 (time 1783143695.16 total 10.00 last 209.26) - /unlink 4013 (time 1783143705.17 total 20.01 last 191.92) total: 4500 /unlink in 22.53 seconds: 199.73 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94884 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg413-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg413-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg413-server stop mds service on oleg413-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg413-server stop mds service on oleg413-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg413-server unloading modules via unload_modules_local on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing unload_modules_local oleg413-server: modules unloaded.