== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:43:25 (1787384605) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg451-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg451-server oleg451-server: oleg451-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 620 (time 1787384831.88 total 10.01 last 61.95) - open/close 1268 (time 1787384841.91 total 20.04 last 64.61) - open/close 1972 (time 1787384851.93 total 30.05 last 70.29) - open/close 2607 (time 1787384861.94 total 40.07 last 63.42) - open/close 3251 (time 1787384871.94 total 50.07 last 64.40) - open/close 3930 (time 1787384881.94 total 60.07 last 67.89) total: 4500 open/close in 68.21 seconds: 65.98 ops/second - /unlink 1283 (time 1787384905.48 total 10.00 last 128.27) - /unlink 2504 (time 1787384915.48 total 20.00 last 122.08) - /unlink 3654 (time 1787384925.49 total 30.01 last 114.90) total: 4500 /unlink in 37.29 seconds: 120.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 654 (time 1787384947.89 total 10.00 last 65.39) - open/close 1280 (time 1787384957.90 total 20.02 last 62.50) - open/close 1980 (time 1787384967.90 total 30.02 last 69.98) - open/close 2531 (time 1787384977.91 total 40.02 last 55.08) - open/close 3122 (time 1787384987.91 total 50.03 last 59.07) - open/close 3798 (time 1787384997.93 total 60.04 last 67.49) total: 4500 open/close in 69.98 seconds: 64.30 ops/second - /unlink 1247 (time 1787385023.13 total 10.00 last 124.66) - /unlink 2481 (time 1787385033.14 total 20.01 last 123.29) - /unlink 3725 (time 1787385043.14 total 30.01 last 124.37) total: 4500 /unlink in 36.17 seconds: 124.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 580 (time 1787385064.72 total 10.01 last 57.97) - open/close 1245 (time 1787385074.72 total 20.01 last 66.47) - open/close 1966 (time 1787385084.72 total 30.01 last 72.08) - open/close 2693 (time 1787385094.73 total 40.02 last 72.63) - open/close 3270 (time 1787385104.74 total 50.03 last 57.64) - open/close 3988 (time 1787385114.75 total 60.04 last 71.76) total: 4500 open/close in 67.94 seconds: 66.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1240 (time 1787385137.21 total 10.00 last 123.99) - /unlink 2383 (time 1787385147.22 total 20.01 last 114.21) - /unlink 3689 (time 1787385157.22 total 30.01 last 130.59) total: 4500 /unlink in 36.73 seconds: 122.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 685 (time 1787385178.94 total 10.03 last 68.29) - open/close 1326 (time 1787385188.94 total 20.03 last 64.09) - open/close 1989 (time 1787385198.95 total 30.04 last 66.24) - open/close 2630 (time 1787385208.96 total 40.06 last 64.00) - open/close 3283 (time 1787385218.97 total 50.06 last 65.28) - open/close 3933 (time 1787385228.97 total 60.06 last 64.97) total: 4500 open/close in 68.93 seconds: 65.28 ops/second - /unlink 1167 (time 1787385253.23 total 10.02 last 116.51) - /unlink 2336 (time 1787385263.23 total 20.02 last 116.86) - /unlink 3571 (time 1787385273.24 total 30.03 last 123.43) total: 4500 /unlink in 37.41 seconds: 120.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 652 (time 1787385295.43 total 10.01 last 65.14) - open/close 1214 (time 1787385305.44 total 20.02 last 56.16) - open/close 1906 (time 1787385315.45 total 30.03 last 69.11) - open/close 2580 (time 1787385325.47 total 40.04 last 67.30) - open/close 3217 (time 1787385335.47 total 50.05 last 63.69) - open/close 3856 (time 1787385345.48 total 60.05 last 63.85) total: 4500 open/close in 69.92 seconds: 64.36 ops/second - /unlink 1318 (time 1787385369.32 total 10.00 last 131.78) - /unlink 2517 (time 1787385379.32 total 20.00 last 119.86) - /unlink 3773 (time 1787385389.32 total 30.01 last 125.55) total: 4500 /unlink in 36.61 seconds: 122.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 686 (time 1787385411.03 total 10.01 last 68.56) - open/close 1351 (time 1787385421.05 total 20.02 last 66.39) - open/close 2064 (time 1787385431.05 total 30.02 last 71.28) - open/close 2705 (time 1787385441.06 total 40.03 last 64.08) - open/close 3361 (time 1787385451.08 total 50.05 last 65.48) - open/close 4030 (time 1787385461.09 total 60.06 last 66.84) total: 4500 open/close in 67.13 seconds: 67.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1159 (time 1787385482.75 total 10.00 last 115.90) - /unlink 2462 (time 1787385492.75 total 20.00 last 130.29) - /unlink 3758 (time 1787385502.76 total 30.01 last 129.53) total: 4500 /unlink in 35.66 seconds: 126.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 656 (time 1787385522.73 total 10.01 last 65.51) - open/close 1296 (time 1787385532.74 total 20.02 last 63.96) - open/close 1981 (time 1787385542.75 total 30.03 last 68.42) - open/close 2612 (time 1787385552.77 total 40.05 last 62.98) - open/close 3272 (time 1787385562.80 total 50.08 last 65.83) - open/close 3956 (time 1787385572.80 total 60.08 last 68.35) total: 4500 open/close in 67.98 seconds: 66.20 ops/second - /unlink 1144 (time 1787385595.32 total 10.00 last 114.35) - /unlink 2183 (time 1787385605.33 total 20.02 last 103.78) - /unlink 3275 (time 1787385615.33 total 30.02 last 109.17) - /unlink 4331 (time 1787385625.33 total 40.02 last 105.58) total: 4500 /unlink in 41.47 seconds: 108.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 610 (time 1787385641.99 total 10.02 last 60.88) - open/close 1274 (time 1787385651.99 total 20.03 last 66.37) - open/close 1885 (time 1787385662.00 total 30.04 last 61.03) - open/close 2600 (time 1787385672.01 total 40.05 last 71.41) - open/close 3255 (time 1787385682.01 total 50.05 last 65.50) - open/close 3986 (time 1787385692.02 total 60.05 last 73.08) total: 4500 open/close in 67.52 seconds: 66.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1178 (time 1787385714.96 total 10.01 last 117.69) - /unlink 2433 (time 1787385724.97 total 20.02 last 125.40) - /unlink 3719 (time 1787385734.98 total 30.03 last 128.48) total: 4500 /unlink in 36.37 seconds: 123.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 688 (time 1787385755.78 total 10.01 last 68.76) - open/close 1341 (time 1787385765.78 total 20.01 last 65.28) - open/close 2025 (time 1787385775.79 total 30.02 last 68.32) - open/close 2740 (time 1787385785.80 total 40.03 last 71.46) - open/close 3463 (time 1787385795.80 total 50.03 last 72.30) - open/close 4173 (time 1787385805.80 total 60.03 last 70.99) total: 4500 open/close in 65.13 seconds: 69.10 ops/second - /unlink 1113 (time 1787385825.75 total 10.00 last 111.30) - /unlink 2331 (time 1787385835.75 total 20.00 last 121.77) - /unlink 3554 (time 1787385845.76 total 30.01 last 122.23) total: 4500 /unlink in 38.08 seconds: 118.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 618 (time 1787385868.50 total 10.01 last 61.74) - open/close 1217 (time 1787385878.50 total 20.02 last 59.85) - open/close 1812 (time 1787385888.51 total 30.02 last 59.49) - open/close 2442 (time 1787385898.51 total 40.02 last 62.99) - open/close 3100 (time 1787385908.52 total 50.03 last 65.71) - open/close 3779 (time 1787385918.53 total 60.04 last 67.84) - open/close 4474 (time 1787385928.53 total 70.05 last 69.46) total: 4500 open/close in 70.40 seconds: 63.92 ops/second - /unlink 1270 (time 1787385942.94 total 10.01 last 126.82) - /unlink 2497 (time 1787385952.95 total 20.02 last 122.64) - /unlink 3698 (time 1787385962.95 total 30.02 last 120.05) total: 4500 /unlink in 36.54 seconds: 123.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 644 (time 1787385984.17 total 10.00 last 64.37) - open/close 1318 (time 1787385994.19 total 20.02 last 67.31) - open/close 1976 (time 1787386004.19 total 30.02 last 65.78) - open/close 2628 (time 1787386014.21 total 40.04 last 65.11) - open/close 3278 (time 1787386024.21 total 50.04 last 64.98) - open/close 3904 (time 1787386034.22 total 60.05 last 62.51) total: 4500 open/close in 68.88 seconds: 65.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1256 (time 1787386057.83 total 10.00 last 125.59) - /unlink 2501 (time 1787386067.84 total 20.01 last 124.37) - /unlink 3720 (time 1787386077.84 total 30.01 last 121.90) total: 4500 /unlink in 36.93 seconds: 121.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 683 (time 1787386099.30 total 10.00 last 68.27) - open/close 1312 (time 1787386109.30 total 20.00 last 62.90) - open/close 1961 (time 1787386119.32 total 30.02 last 64.78) - open/close 2542 (time 1787386129.34 total 40.04 last 58.00) - open/close 3107 (time 1787386139.34 total 50.04 last 56.48) - open/close 3813 (time 1787386149.34 total 60.05 last 70.58) total: 4500 open/close in 70.00 seconds: 64.28 ops/second - /unlink 1299 (time 1787386173.87 total 10.01 last 129.83) - /unlink 2509 (time 1787386183.87 total 20.01 last 120.99) - /unlink 3676 (time 1787386193.87 total 30.01 last 116.68) total: 4500 /unlink in 36.70 seconds: 122.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 649 (time 1787386216.20 total 10.01 last 64.81) - open/close 1291 (time 1787386226.22 total 20.03 last 64.10) - open/close 1875 (time 1787386236.22 total 30.03 last 58.37) - open/close 2512 (time 1787386246.23 total 40.04 last 63.65) - open/close 3126 (time 1787386256.24 total 50.05 last 61.37) - open/close 3713 (time 1787386266.25 total 60.06 last 58.60) - open/close 4440 (time 1787386276.27 total 70.08 last 72.60) total: 4500 open/close in 71.06 seconds: 63.32 ops/second - /unlink 1134 (time 1787386292.69 total 10.04 last 112.99) - /unlink 2307 (time 1787386302.69 total 20.04 last 117.21) - /unlink 3604 (time 1787386312.70 total 30.05 last 129.61) total: 4500 /unlink in 36.84 seconds: 122.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 641 (time 1787386334.15 total 10.03 last 63.93) - open/close 1260 (time 1787386344.15 total 20.03 last 61.88) - open/close 1936 (time 1787386354.16 total 30.04 last 67.55) - open/close 2562 (time 1787386364.16 total 40.04 last 62.58) - open/close 3195 (time 1787386374.17 total 50.05 last 63.25) - open/close 3904 (time 1787386384.18 total 60.06 last 70.83) total: 4500 open/close in 69.22 seconds: 65.01 ops/second - /unlink 1296 (time 1787386407.94 total 10.00 last 129.60) - /unlink 2653 (time 1787386417.95 total 20.01 last 135.59) - /unlink 3810 (time 1787386427.96 total 30.02 last 115.56) total: 4500 /unlink in 35.92 seconds: 125.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 603 (time 1787386448.35 total 10.00 last 60.29) - open/close 1264 (time 1787386458.35 total 20.00 last 66.08) - open/close 1969 (time 1787386468.35 total 30.01 last 70.48) - open/close 2657 (time 1787386478.36 total 40.01 last 68.75) - open/close 3331 (time 1787386488.38 total 50.03 last 67.28) - open/close 3990 (time 1787386498.38 total 60.03 last 65.88) total: 4500 open/close in 67.36 seconds: 66.81 ops/second - /unlink 1307 (time 1787386520.33 total 10.00 last 130.65) - /unlink 2426 (time 1787386530.33 total 20.00 last 111.90) - /unlink 3720 (time 1787386540.33 total 30.00 last 129.39) total: 4500 /unlink in 36.34 seconds: 123.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 663 (time 1787386562.10 total 10.00 last 66.28) - open/close 1268 (time 1787386572.11 total 20.01 last 60.44) - open/close 1990 (time 1787386582.12 total 30.03 last 72.09) - open/close 2740 (time 1787386592.13 total 40.03 last 74.98) - open/close 3399 (time 1787386602.13 total 50.04 last 65.87) - open/close 4039 (time 1787386612.13 total 60.04 last 63.99) total: 4500 open/close in 66.95 seconds: 67.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1132 (time 1787386634.02 total 10.00 last 113.16) - /unlink 2233 (time 1787386644.03 total 20.01 last 110.02) - /unlink 3499 (time 1787386654.04 total 30.02 last 126.53) total: 4500 /unlink in 38.92 seconds: 115.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 653 (time 1787386677.86 total 10.00 last 65.29) - open/close 1359 (time 1787386687.86 total 20.01 last 70.56) - open/close 2006 (time 1787386697.87 total 30.01 last 64.66) - open/close 2730 (time 1787386707.88 total 40.02 last 72.32) - open/close 3453 (time 1787386717.89 total 50.03 last 72.25) - open/close 4116 (time 1787386727.91 total 60.05 last 66.15) total: 4500 open/close in 66.56 seconds: 67.61 ops/second - /unlink 1165 (time 1787386749.78 total 10.00 last 116.44) - /unlink 2373 (time 1787386759.79 total 20.01 last 120.75) - /unlink 3654 (time 1787386769.79 total 30.02 last 128.02) total: 4500 /unlink in 37.39 seconds: 120.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 678 (time 1787386792.50 total 10.00 last 67.77) - open/close 1347 (time 1787386802.50 total 20.01 last 66.88) - open/close 2028 (time 1787386812.50 total 30.01 last 68.10) - open/close 2764 (time 1787386822.50 total 40.01 last 73.60) - open/close 3461 (time 1787386832.51 total 50.02 last 69.64) - open/close 4099 (time 1787386842.52 total 60.02 last 63.76) total: 4500 open/close in 66.53 seconds: 67.64 ops/second - /unlink 1038 (time 1787386864.44 total 10.02 last 103.63) - /unlink 2116 (time 1787386874.44 total 20.02 last 107.74) - /unlink 3246 (time 1787386884.44 total 30.03 last 112.96) - /unlink 4422 (time 1787386894.45 total 40.03 last 117.56) total: 4500 /unlink in 40.97 seconds: 109.84 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95040 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: modules unloaded.