== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:27:55 (1789133275) Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg422-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server oleg422-server: oleg422-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg422-server: detected 4 online CPUs by sysfs oleg422-server: MODOPTS_LIBCFS= oleg422-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg422-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg422-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg422-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 747 (time 1789133476.09 total 10.01 last 74.61) - open/close 1498 (time 1789133486.10 total 20.02 last 75.05) - open/close 2224 (time 1789133496.11 total 30.02 last 72.56) - open/close 2852 (time 1789133506.11 total 40.03 last 62.75) - open/close 3519 (time 1789133516.13 total 50.04 last 66.63) - open/close 4318 (time 1789133526.13 total 60.04 last 79.88) total: 4500 open/close in 62.42 seconds: 72.09 ops/second - /unlink 1546 (time 1789133543.03 total 10.01 last 154.49) - /unlink 2994 (time 1789133553.03 total 20.01 last 144.76) - /unlink 4403 (time 1789133563.05 total 30.03 last 140.66) total: 4500 /unlink in 31.03 seconds: 145.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 784 (time 1789133577.90 total 10.01 last 78.32) - open/close 1307 (time 1789133587.90 total 20.02 last 52.26) - open/close 1952 (time 1789133597.91 total 30.02 last 64.46) - open/close 2659 (time 1789133607.92 total 40.03 last 70.63) - open/close 3338 (time 1789133617.93 total 50.04 last 67.83) - open/close 3934 (time 1789133627.97 total 60.08 last 59.38) total: 4500 open/close in 68.22 seconds: 65.97 ops/second - /unlink 1523 (time 1789133650.93 total 10.00 last 152.24) - /unlink 3108 (time 1789133660.93 total 20.01 last 158.43) total: 4500 /unlink in 29.57 seconds: 152.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 771 (time 1789133684.82 total 10.00 last 77.10) - open/close 1552 (time 1789133694.82 total 20.00 last 78.09) - open/close 2359 (time 1789133704.82 total 30.00 last 80.68) - open/close 3182 (time 1789133714.83 total 40.01 last 82.25) - open/close 4020 (time 1789133724.83 total 50.02 last 83.73) total: 4500 open/close in 56.39 seconds: 79.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1356 (time 1789133745.32 total 10.00 last 135.58) - /unlink 2424 (time 1789133755.34 total 20.02 last 106.62) - /unlink 3519 (time 1789133765.35 total 30.03 last 109.32) total: 4500 /unlink in 37.20 seconds: 120.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 813 (time 1789133787.63 total 10.01 last 81.23) - open/close 1619 (time 1789133797.63 total 20.01 last 80.60) - open/close 2341 (time 1789133807.66 total 30.04 last 71.99) - open/close 2983 (time 1789133817.67 total 40.04 last 64.17) - open/close 3647 (time 1789133827.67 total 50.04 last 66.39) - open/close 4367 (time 1789133837.68 total 60.05 last 71.92) total: 4500 open/close in 61.75 seconds: 72.87 ops/second - /unlink 1422 (time 1789133853.42 total 10.00 last 142.20) - /unlink 2767 (time 1789133863.43 total 20.00 last 134.45) - /unlink 3837 (time 1789133873.44 total 30.01 last 106.89) total: 4500 /unlink in 36.03 seconds: 124.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 679 (time 1789133894.20 total 10.01 last 67.84) - open/close 1478 (time 1789133904.22 total 20.03 last 79.77) - open/close 2084 (time 1789133914.24 total 30.04 last 60.48) - open/close 2728 (time 1789133924.25 total 40.06 last 64.32) - open/close 3435 (time 1789133934.26 total 50.06 last 70.65) - open/close 4128 (time 1789133944.26 total 60.07 last 69.29) total: 4500 open/close in 66.21 seconds: 67.96 ops/second - /unlink 1189 (time 1789133964.45 total 10.00 last 118.84) - /unlink 2084 (time 1789133974.46 total 20.01 last 89.49) - /unlink 3284 (time 1789133984.46 total 30.01 last 119.98) total: 4500 /unlink in 39.94 seconds: 112.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 728 (time 1789134008.92 total 10.00 last 72.79) - open/close 1406 (time 1789134018.93 total 20.02 last 67.69) - open/close 2019 (time 1789134028.96 total 30.04 last 61.14) - open/close 2643 (time 1789134038.96 total 40.05 last 62.39) - open/close 3284 (time 1789134048.96 total 50.05 last 64.09) - open/close 3961 (time 1789134058.97 total 60.06 last 67.65) total: 4500 open/close in 66.70 seconds: 67.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1213 (time 1789134080.41 total 10.00 last 121.28) - /unlink 2449 (time 1789134090.41 total 20.00 last 123.60) - /unlink 3864 (time 1789134100.42 total 30.01 last 141.38) total: 4500 /unlink in 34.72 seconds: 129.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 519 (time 1789134120.46 total 10.03 last 51.77) - open/close 983 (time 1789134130.47 total 20.03 last 46.38) - open/close 1530 (time 1789134140.47 total 30.04 last 54.65) - open/close 2206 (time 1789134150.47 total 40.04 last 67.60) - open/close 2890 (time 1789134160.47 total 50.04 last 68.40) - open/close 3745 (time 1789134170.49 total 60.05 last 85.41) - open/close 4471 (time 1789134180.49 total 70.05 last 72.59) total: 4500 open/close in 70.40 seconds: 63.92 ops/second - /unlink 1418 (time 1789134195.01 total 10.01 last 141.72) - /unlink 2896 (time 1789134205.02 total 20.01 last 147.71) - /unlink 4365 (time 1789134215.02 total 30.01 last 146.87) total: 4500 /unlink in 31.02 seconds: 145.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 747 (time 1789134230.83 total 10.00 last 74.68) - open/close 1370 (time 1789134240.83 total 20.00 last 62.30) - open/close 1867 (time 1789134250.85 total 30.02 last 49.64) - open/close 2499 (time 1789134260.86 total 40.03 last 63.10) - open/close 3194 (time 1789134270.87 total 50.04 last 69.46) - open/close 3890 (time 1789134280.87 total 60.04 last 69.56) - open/close 4460 (time 1789134290.89 total 70.06 last 56.89) total: 4500 open/close in 70.68 seconds: 63.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1289 (time 1789134306.05 total 10.00 last 128.88) - /unlink 2533 (time 1789134316.06 total 20.01 last 124.30) - /unlink 4039 (time 1789134326.08 total 30.02 last 150.39) total: 4500 /unlink in 34.04 seconds: 132.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 615 (time 1789134345.63 total 10.00 last 61.48) - open/close 1279 (time 1789134355.64 total 20.01 last 66.34) - open/close 1888 (time 1789134365.65 total 30.02 last 60.83) - open/close 2440 (time 1789134375.66 total 40.03 last 55.18) - open/close 3185 (time 1789134385.66 total 50.03 last 74.45) - open/close 3852 (time 1789134395.68 total 60.05 last 66.61) total: 4500 open/close in 68.67 seconds: 65.53 ops/second - /unlink 1410 (time 1789134418.74 total 10.01 last 140.85) - /unlink 2562 (time 1789134428.74 total 20.01 last 115.18) - /unlink 3885 (time 1789134438.75 total 30.02 last 132.24) total: 4500 /unlink in 34.75 seconds: 129.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 732 (time 1789134458.67 total 10.00 last 73.20) - open/close 1501 (time 1789134468.67 total 20.01 last 76.86) - open/close 2342 (time 1789134478.68 total 30.01 last 84.04) - open/close 2916 (time 1789134488.69 total 40.02 last 57.36) - open/close 3573 (time 1789134498.69 total 50.02 last 65.68) - open/close 4327 (time 1789134508.69 total 60.03 last 75.38) total: 4500 open/close in 62.27 seconds: 72.26 ops/second - /unlink 1489 (time 1789134525.10 total 10.02 last 148.63) - /unlink 2757 (time 1789134535.10 total 20.02 last 126.76) - /unlink 4061 (time 1789134545.11 total 30.03 last 130.30) total: 4500 /unlink in 33.46 seconds: 134.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 732 (time 1789134565.04 total 10.01 last 73.14) - open/close 1532 (time 1789134575.05 total 20.02 last 79.90) - open/close 2220 (time 1789134585.06 total 30.03 last 68.72) - open/close 2852 (time 1789134595.07 total 40.04 last 63.13) - open/close 3511 (time 1789134605.08 total 50.05 last 65.87) - open/close 4241 (time 1789134615.09 total 60.06 last 72.93) total: 4500 open/close in 63.51 seconds: 70.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1415 (time 1789134633.64 total 10.00 last 141.49) - /unlink 2725 (time 1789134643.64 total 20.00 last 130.95) - /unlink 4188 (time 1789134653.64 total 30.01 last 146.25) total: 4500 /unlink in 32.61 seconds: 137.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 728 (time 1789134672.14 total 10.01 last 72.72) - open/close 1480 (time 1789134682.15 total 20.02 last 75.11) - open/close 2186 (time 1789134692.16 total 30.04 last 70.52) - open/close 2996 (time 1789134702.17 total 40.04 last 80.95) - open/close 3716 (time 1789134712.18 total 50.05 last 71.95) - open/close 4498 (time 1789134722.18 total 60.05 last 78.19) total: 4500 open/close in 60.08 seconds: 74.90 ops/second - /unlink 1449 (time 1789134736.99 total 10.02 last 144.61) - /unlink 2790 (time 1789134746.99 total 20.02 last 134.08) - /unlink 4170 (time 1789134757.00 total 30.03 last 137.92) total: 4500 /unlink in 32.22 seconds: 139.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 654 (time 1789134775.56 total 10.03 last 65.23) - open/close 1423 (time 1789134785.58 total 20.04 last 76.80) - open/close 2133 (time 1789134795.58 total 30.04 last 70.99) - open/close 2786 (time 1789134805.58 total 40.04 last 65.30) - open/close 3362 (time 1789134815.62 total 50.08 last 57.35) - open/close 3992 (time 1789134825.62 total 60.08 last 63.00) total: 4500 open/close in 67.97 seconds: 66.21 ops/second - /unlink 1360 (time 1789134849.06 total 10.00 last 135.96) - /unlink 2747 (time 1789134859.07 total 20.01 last 138.67) - /unlink 4126 (time 1789134869.07 total 30.01 last 137.79) total: 4500 /unlink in 32.94 seconds: 136.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 410 (time 1789134889.84 total 10.00 last 40.98) - open/close 877 (time 1789134899.85 total 20.01 last 46.66) - open/close 1354 (time 1789134909.85 total 30.01 last 47.70) - open/close 1813 (time 1789134919.87 total 40.03 last 45.81) - open/close 2407 (time 1789134929.88 total 50.04 last 59.34) - open/close 3039 (time 1789134939.89 total 60.05 last 63.16) - open/close 3663 (time 1789134949.89 total 70.05 last 62.40) - open/close 4345 (time 1789134959.90 total 80.06 last 68.13) total: 4500 open/close in 82.58 seconds: 54.49 ops/second - /unlink 901 (time 1789134976.86 total 10.01 last 90.03) - /unlink 2121 (time 1789134986.86 total 20.01 last 121.98) - /unlink 3462 (time 1789134996.87 total 30.02 last 133.94) total: 4500 /unlink in 39.87 seconds: 112.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 562 (time 1789135022.74 total 10.03 last 56.04) - open/close 1074 (time 1789135032.75 total 20.04 last 51.14) - open/close 1688 (time 1789135042.76 total 30.06 last 61.31) - open/close 2337 (time 1789135052.77 total 40.06 last 64.85) - open/close 3131 (time 1789135062.78 total 50.07 last 79.32) - open/close 3779 (time 1789135072.78 total 60.07 last 64.79) - open/close 4303 (time 1789135082.79 total 70.08 last 52.36) total: 4500 open/close in 72.91 seconds: 61.72 ops/second - /unlink 1352 (time 1789135101.71 total 10.00 last 135.15) - /unlink 2705 (time 1789135111.72 total 20.01 last 135.26) - /unlink 4126 (time 1789135121.72 total 30.01 last 142.10) total: 4500 /unlink in 33.17 seconds: 135.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 675 (time 1789135140.50 total 10.01 last 67.42) - open/close 1410 (time 1789135150.51 total 20.02 last 73.47) - open/close 2007 (time 1789135160.51 total 30.02 last 59.68) - open/close 2799 (time 1789135170.51 total 40.03 last 79.15) - open/close 3488 (time 1789135180.52 total 50.03 last 68.85) - open/close 4267 (time 1789135190.53 total 60.05 last 77.80) total: 4500 open/close in 64.48 seconds: 69.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1207 (time 1789135210.66 total 10.00 last 120.66) - /unlink 2448 (time 1789135220.66 total 20.00 last 124.09) - /unlink 3652 (time 1789135230.67 total 30.01 last 120.28) total: 4500 /unlink in 36.59 seconds: 122.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 702 (time 1789135252.67 total 10.00 last 70.19) - open/close 1366 (time 1789135262.68 total 20.01 last 66.35) - open/close 2149 (time 1789135272.68 total 30.01 last 78.27) - open/close 2883 (time 1789135282.69 total 40.01 last 73.39) - open/close 3582 (time 1789135292.69 total 50.02 last 69.89) - open/close 4272 (time 1789135302.69 total 60.02 last 68.97) total: 4500 open/close in 63.98 seconds: 70.33 ops/second - /unlink 1480 (time 1789135321.03 total 10.01 last 147.87) - /unlink 2854 (time 1789135331.03 total 20.01 last 137.40) - /unlink 4270 (time 1789135341.03 total 30.01 last 141.57) total: 4500 /unlink in 31.60 seconds: 142.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 752 (time 1789135356.77 total 10.00 last 75.19) - open/close 1611 (time 1789135366.77 total 20.01 last 85.84) - open/close 2341 (time 1789135376.78 total 30.01 last 72.98) - open/close 3181 (time 1789135386.79 total 40.02 last 83.94) - open/close 3917 (time 1789135396.81 total 50.04 last 73.45) total: 4500 open/close in 59.16 seconds: 76.06 ops/second - /unlink 1070 (time 1789135420.41 total 10.00 last 106.98) - /unlink 2082 (time 1789135430.41 total 20.01 last 101.16) - /unlink 2840 (time 1789135440.43 total 30.02 last 75.67) - /unlink 3882 (time 1789135450.44 total 40.03 last 104.11) total: 4500 /unlink in 46.19 seconds: 97.42 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95332 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server unloading modules via unload_modules_local on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing unload_modules_local oleg422-server: modules unloaded.