== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:37:06 (1787377026) Stopping clients: oleg115-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg115-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg115-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg115-server oleg115-server: oleg115-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing load_modules_local oleg115-server: Loading modules from /home/green/git/lustre-release/lustre oleg115-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg115-server: detected 4 online CPUs by sysfs oleg115-server: MODOPTS_LIBCFS= oleg115-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg115-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg115-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 780 (time 1787377212.10 total 10.01 last 77.89) - open/close 1592 (time 1787377222.10 total 20.02 last 81.14) - open/close 2392 (time 1787377232.10 total 30.02 last 79.99) - open/close 3218 (time 1787377242.11 total 40.03 last 82.57) - open/close 3988 (time 1787377252.12 total 50.04 last 76.93) total: 4500 open/close in 57.24 seconds: 78.62 ops/second - /unlink 1400 (time 1787377274.72 total 10.01 last 139.89) - /unlink 2613 (time 1787377284.72 total 20.01 last 121.28) - /unlink 4107 (time 1787377294.73 total 30.01 last 149.35) total: 4500 /unlink in 32.49 seconds: 138.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 759 (time 1787377312.14 total 10.00 last 75.87) - open/close 1496 (time 1787377322.14 total 20.00 last 73.69) - open/close 2215 (time 1787377332.15 total 30.01 last 71.86) - open/close 2866 (time 1787377342.17 total 40.03 last 64.99) - open/close 3523 (time 1787377352.19 total 50.05 last 65.56) - open/close 4277 (time 1787377362.19 total 60.05 last 75.36) total: 4500 open/close in 63.82 seconds: 70.51 ops/second - /unlink 1322 (time 1787377380.66 total 10.01 last 132.13) - /unlink 2638 (time 1787377390.67 total 20.01 last 131.51) - /unlink 3959 (time 1787377400.67 total 30.02 last 132.02) total: 4500 /unlink in 34.00 seconds: 132.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 724 (time 1787377420.68 total 10.00 last 72.38) - open/close 1448 (time 1787377430.69 total 20.01 last 72.33) - open/close 2273 (time 1787377440.70 total 30.02 last 82.44) - open/close 3092 (time 1787377450.70 total 40.02 last 81.89) - open/close 3865 (time 1787377460.70 total 50.02 last 77.28) total: 4500 open/close in 58.36 seconds: 77.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1441 (time 1787377483.67 total 10.00 last 144.08) - /unlink 2677 (time 1787377493.68 total 20.01 last 123.50) - /unlink 4058 (time 1787377503.68 total 30.01 last 138.09) total: 4500 /unlink in 33.67 seconds: 133.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 653 (time 1787377523.47 total 10.02 last 65.18) - open/close 1301 (time 1787377533.47 total 20.02 last 64.79) - open/close 1974 (time 1787377543.47 total 30.02 last 67.29) - open/close 2609 (time 1787377553.47 total 40.03 last 63.47) - open/close 3143 (time 1787377563.49 total 50.04 last 53.31) - open/close 3827 (time 1787377573.50 total 60.05 last 68.35) - open/close 4417 (time 1787377583.51 total 70.06 last 58.95) total: 4500 open/close in 71.36 seconds: 63.06 ops/second - /unlink 1119 (time 1787377601.08 total 10.02 last 111.64) - /unlink 2070 (time 1787377611.08 total 20.03 last 95.09) - /unlink 3262 (time 1787377621.08 total 30.03 last 119.18) total: 4500 /unlink in 37.93 seconds: 118.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 688 (time 1787377645.30 total 10.01 last 68.70) - open/close 1294 (time 1787377655.31 total 20.02 last 60.54) - open/close 1919 (time 1787377665.33 total 30.04 last 62.38) - open/close 2426 (time 1787377675.33 total 40.04 last 50.69) - open/close 3025 (time 1787377685.33 total 50.05 last 59.89) - open/close 3638 (time 1787377695.34 total 60.06 last 61.24) - open/close 4296 (time 1787377705.37 total 70.08 last 65.62) total: 4500 open/close in 73.37 seconds: 61.34 ops/second - /unlink 1445 (time 1787377723.62 total 10.00 last 144.45) - /unlink 2827 (time 1787377733.62 total 20.01 last 138.15) - /unlink 4118 (time 1787377743.63 total 30.01 last 129.02) total: 4500 /unlink in 32.92 seconds: 136.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 696 (time 1787377761.80 total 10.00 last 69.59) - open/close 1285 (time 1787377771.82 total 20.02 last 58.81) - open/close 1933 (time 1787377781.83 total 30.03 last 64.71) - open/close 2711 (time 1787377791.85 total 40.04 last 77.70) - open/close 3408 (time 1787377801.85 total 50.05 last 69.65) - open/close 4143 (time 1787377811.86 total 60.06 last 73.44) total: 4500 open/close in 64.52 seconds: 69.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1227 (time 1787377831.34 total 10.01 last 122.62) - /unlink 2378 (time 1787377841.34 total 20.01 last 115.03) - /unlink 3687 (time 1787377851.35 total 30.01 last 130.88) total: 4500 /unlink in 37.85 seconds: 118.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 705 (time 1787377874.44 total 10.00 last 70.48) - open/close 1421 (time 1787377884.45 total 20.02 last 71.50) - open/close 2263 (time 1787377894.46 total 30.02 last 84.14) - open/close 3064 (time 1787377904.47 total 40.04 last 80.01) - open/close 3873 (time 1787377914.48 total 50.04 last 80.86) total: 4500 open/close in 58.86 seconds: 76.46 ops/second - /unlink 1335 (time 1787377937.99 total 10.00 last 133.47) - /unlink 2812 (time 1787377947.99 total 20.00 last 147.67) - /unlink 4253 (time 1787377958.00 total 30.01 last 143.99) total: 4500 /unlink in 32.00 seconds: 140.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 719 (time 1787377975.09 total 10.00 last 71.87) - open/close 1361 (time 1787377985.10 total 20.02 last 64.13) - open/close 1928 (time 1787377995.12 total 30.03 last 56.62) - open/close 2516 (time 1787378005.12 total 40.03 last 58.79) - open/close 3230 (time 1787378015.13 total 50.04 last 71.35) - open/close 3927 (time 1787378025.13 total 60.04 last 69.70) total: 4500 open/close in 69.23 seconds: 65.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1170 (time 1787378051.38 total 10.00 last 116.99) - /unlink 2434 (time 1787378061.38 total 20.00 last 126.38) - /unlink 3932 (time 1787378071.38 total 30.01 last 149.75) total: 4500 /unlink in 35.38 seconds: 127.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 662 (time 1787378092.04 total 10.01 last 66.12) - open/close 1460 (time 1787378102.04 total 20.01 last 79.78) - open/close 2212 (time 1787378112.04 total 30.02 last 75.17) - open/close 2988 (time 1787378122.04 total 40.02 last 77.58) - open/close 3770 (time 1787378132.05 total 50.03 last 78.15) total: 4500 open/close in 59.02 seconds: 76.24 ops/second - /unlink 1478 (time 1787378157.07 total 10.00 last 147.80) - /unlink 3016 (time 1787378167.07 total 20.00 last 153.74) - /unlink 4328 (time 1787378177.08 total 30.01 last 131.17) total: 4500 /unlink in 31.49 seconds: 142.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 741 (time 1787378193.23 total 10.00 last 74.07) - open/close 1423 (time 1787378203.24 total 20.01 last 68.16) - open/close 2084 (time 1787378213.26 total 30.03 last 65.97) - open/close 2661 (time 1787378223.26 total 40.03 last 57.68) - open/close 3316 (time 1787378233.27 total 50.05 last 65.40) - open/close 3931 (time 1787378243.28 total 60.06 last 61.45) - open/close 4495 (time 1787378253.30 total 70.08 last 56.29) total: 4500 open/close in 70.17 seconds: 64.13 ops/second - /unlink 1421 (time 1787378268.40 total 10.01 last 141.99) - /unlink 2836 (time 1787378278.41 total 20.01 last 141.44) - /unlink 4153 (time 1787378288.41 total 30.01 last 131.66) total: 4500 /unlink in 32.38 seconds: 138.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 751 (time 1787378305.61 total 10.01 last 75.04) - open/close 1397 (time 1787378315.62 total 20.02 last 64.53) - open/close 2141 (time 1787378325.62 total 30.02 last 74.39) - open/close 2975 (time 1787378335.62 total 40.02 last 83.37) - open/close 3737 (time 1787378345.63 total 50.03 last 76.13) total: 4500 open/close in 59.54 seconds: 75.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1385 (time 1787378369.94 total 10.01 last 138.30) - /unlink 2770 (time 1787378379.94 total 20.01 last 138.49) - /unlink 4178 (time 1787378389.95 total 30.02 last 140.70) total: 4500 /unlink in 32.52 seconds: 138.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 689 (time 1787378408.10 total 10.01 last 68.86) - open/close 1371 (time 1787378418.10 total 20.01 last 68.20) - open/close 2171 (time 1787378428.10 total 30.01 last 79.96) - open/close 2933 (time 1787378438.12 total 40.03 last 76.07) - open/close 3705 (time 1787378448.13 total 50.04 last 77.14) - open/close 4477 (time 1787378458.14 total 60.04 last 77.13) total: 4500 open/close in 60.36 seconds: 74.55 ops/second - /unlink 1375 (time 1787378473.32 total 10.00 last 137.45) - /unlink 2847 (time 1787378483.32 total 20.01 last 147.15) - /unlink 4281 (time 1787378493.33 total 30.01 last 143.33) total: 4500 /unlink in 31.33 seconds: 143.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 855 (time 1787378508.70 total 10.01 last 85.41) - open/close 1676 (time 1787378518.71 total 20.03 last 81.96) - open/close 2428 (time 1787378528.73 total 30.04 last 75.12) - open/close 3104 (time 1787378538.74 total 40.06 last 67.48) - open/close 3884 (time 1787378548.75 total 50.06 last 77.94) total: 4500 open/close in 58.35 seconds: 77.12 ops/second - /unlink 1445 (time 1787378571.84 total 10.01 last 144.42) - /unlink 2770 (time 1787378581.85 total 20.01 last 132.47) - /unlink 4335 (time 1787378591.85 total 30.01 last 156.44) total: 4500 /unlink in 31.29 seconds: 143.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 811 (time 1787378609.13 total 10.01 last 81.06) - open/close 1511 (time 1787378619.14 total 20.01 last 69.98) - open/close 2291 (time 1787378629.14 total 30.01 last 77.99) - open/close 3021 (time 1787378639.14 total 40.02 last 72.95) - open/close 3740 (time 1787378649.16 total 50.03 last 71.80) total: 4500 open/close in 59.01 seconds: 76.26 ops/second - /unlink 1185 (time 1787378672.73 total 10.01 last 118.43) - /unlink 2605 (time 1787378682.73 total 20.01 last 142.00) - /unlink 4007 (time 1787378692.73 total 30.01 last 140.12) total: 4500 /unlink in 33.57 seconds: 134.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 884 (time 1787378711.28 total 10.01 last 88.33) - open/close 1657 (time 1787378721.29 total 20.01 last 77.25) - open/close 2561 (time 1787378731.30 total 30.02 last 90.35) - open/close 3436 (time 1787378741.30 total 40.02 last 87.48) - open/close 4261 (time 1787378751.31 total 50.03 last 82.44) total: 4500 open/close in 52.90 seconds: 85.07 ops/second - /unlink 1456 (time 1787378768.68 total 10.01 last 145.44) - /unlink 2881 (time 1787378778.68 total 20.01 last 142.50) - /unlink 4364 (time 1787378788.69 total 30.01 last 148.29) total: 4500 /unlink in 30.56 seconds: 147.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 706 (time 1787378803.92 total 10.00 last 70.58) - open/close 1521 (time 1787378813.93 total 20.01 last 81.42) - open/close 2295 (time 1787378823.93 total 30.01 last 77.39) - open/close 3040 (time 1787378833.95 total 40.03 last 74.41) - open/close 3787 (time 1787378843.95 total 50.03 last 74.64) total: 4500 open/close in 58.69 seconds: 76.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1373 (time 1787378866.78 total 10.00 last 137.26) - /unlink 2811 (time 1787378876.79 total 20.01 last 143.71) - /unlink 4378 (time 1787378886.79 total 30.01 last 156.62) total: 4500 /unlink in 30.90 seconds: 145.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 778 (time 1787378902.60 total 10.01 last 77.70) - open/close 1510 (time 1787378912.61 total 20.02 last 73.16) - open/close 2315 (time 1787378922.61 total 30.02 last 80.49) - open/close 3156 (time 1787378932.61 total 40.02 last 84.07) - open/close 3945 (time 1787378942.62 total 50.03 last 78.87) total: 4500 open/close in 59.19 seconds: 76.03 ops/second - /unlink 1335 (time 1787378967.31 total 10.00 last 133.48) - /unlink 2759 (time 1787378977.32 total 20.00 last 142.36) - /unlink 4188 (time 1787378987.32 total 30.01 last 142.86) total: 4500 /unlink in 32.33 seconds: 139.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 810 (time 1787379003.20 total 10.03 last 80.79) - open/close 1612 (time 1787379013.20 total 20.03 last 80.19) - open/close 2475 (time 1787379023.21 total 30.04 last 86.22) - open/close 3219 (time 1787379033.22 total 40.04 last 74.34) - open/close 3977 (time 1787379043.22 total 50.05 last 75.79) total: 4500 open/close in 56.86 seconds: 79.14 ops/second - /unlink 1425 (time 1787379064.53 total 10.00 last 142.49) - /unlink 2701 (time 1787379074.53 total 20.00 last 127.59) - /unlink 4064 (time 1787379084.53 total 30.01 last 136.24) total: 4500 /unlink in 33.19 seconds: 135.60 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95268 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg115-server unloading modules via unload_modules_local on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local oleg115-server: modules unloaded.