== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:57:57 (1787785077) Stopping clients: oleg658-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg658-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg658-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg658-server oleg658-server: oleg658-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing load_modules_local oleg658-server: Loading modules from /home/green/git/lustre-release/lustre oleg658-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg658-server: detected 4 online CPUs by sysfs oleg658-server: MODOPTS_LIBCFS= oleg658-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg658-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg658-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg658-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg658-client.virtnet: -o user_xattr,flock 192.168.206.158@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 649 (time 1787785274.86 total 10.00 last 64.88) - open/close 1382 (time 1787785284.87 total 20.01 last 73.27) - open/close 2103 (time 1787785294.88 total 30.02 last 72.00) - open/close 2839 (time 1787785304.89 total 40.03 last 73.56) - open/close 3573 (time 1787785314.89 total 50.03 last 73.37) - open/close 4191 (time 1787785324.90 total 60.04 last 61.75) total: 4500 open/close in 64.07 seconds: 70.23 ops/second - /unlink 1430 (time 1787785343.18 total 10.01 last 142.92) - /unlink 2875 (time 1787785353.18 total 20.01 last 144.48) - /unlink 4343 (time 1787785363.19 total 30.01 last 146.75) total: 4500 /unlink in 31.27 seconds: 143.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 699 (time 1787785379.07 total 10.01 last 69.81) - open/close 1440 (time 1787785389.08 total 20.02 last 74.06) - open/close 2192 (time 1787785399.08 total 30.03 last 75.15) - open/close 2885 (time 1787785409.09 total 40.03 last 69.26) - open/close 3600 (time 1787785419.11 total 50.05 last 71.36) - open/close 4372 (time 1787785429.11 total 60.05 last 77.19) total: 4500 open/close in 61.69 seconds: 72.95 ops/second - /unlink 1473 (time 1787785444.92 total 10.00 last 147.27) - /unlink 3020 (time 1787785454.93 total 20.01 last 154.61) - /unlink 4393 (time 1787785464.93 total 30.01 last 137.30) total: 4500 /unlink in 30.72 seconds: 146.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 668 (time 1787785479.70 total 10.02 last 66.65) - open/close 1416 (time 1787785489.70 total 20.02 last 74.80) - open/close 2149 (time 1787785499.73 total 30.05 last 73.07) - open/close 2861 (time 1787785509.74 total 40.06 last 71.15) - open/close 3609 (time 1787785519.75 total 50.08 last 74.69) - open/close 4351 (time 1787785529.76 total 60.08 last 74.14) total: 4500 open/close in 62.43 seconds: 72.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1407 (time 1787785546.18 total 10.00 last 140.65) - /unlink 2870 (time 1787785556.18 total 20.01 last 146.26) - /unlink 4091 (time 1787785566.18 total 30.01 last 122.05) total: 4500 /unlink in 33.21 seconds: 135.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 709 (time 1787785584.32 total 10.02 last 70.73) - open/close 1362 (time 1787785594.33 total 20.03 last 65.28) - open/close 2095 (time 1787785604.35 total 30.05 last 73.12) - open/close 2881 (time 1787785614.35 total 40.06 last 78.58) - open/close 3638 (time 1787785624.36 total 50.06 last 75.66) - open/close 4354 (time 1787785634.37 total 60.08 last 71.49) total: 4500 open/close in 61.57 seconds: 73.09 ops/second - /unlink 1342 (time 1787785650.83 total 10.00 last 134.19) - /unlink 2727 (time 1787785660.84 total 20.01 last 138.33) - /unlink 4177 (time 1787785670.85 total 30.02 last 144.95) total: 4500 /unlink in 32.34 seconds: 139.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 688 (time 1787785688.05 total 10.00 last 68.78) - open/close 1346 (time 1787785698.07 total 20.02 last 65.69) - open/close 1984 (time 1787785708.07 total 30.02 last 63.79) - open/close 2656 (time 1787785718.08 total 40.03 last 67.15) - open/close 3377 (time 1787785728.08 total 50.03 last 72.09) - open/close 4058 (time 1787785738.09 total 60.04 last 68.03) total: 4500 open/close in 66.70 seconds: 67.46 ops/second - /unlink 1132 (time 1787785759.64 total 10.00 last 113.19) - /unlink 2412 (time 1787785769.65 total 20.01 last 127.92) - /unlink 3644 (time 1787785779.65 total 30.01 last 123.16) total: 4500 /unlink in 36.18 seconds: 124.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 719 (time 1787785801.18 total 10.02 last 71.77) - open/close 1429 (time 1787785811.20 total 20.03 last 70.92) - open/close 2122 (time 1787785821.20 total 30.04 last 69.24) - open/close 2851 (time 1787785831.21 total 40.04 last 72.86) - open/close 3535 (time 1787785841.21 total 50.04 last 68.40) - open/close 4210 (time 1787785851.21 total 60.05 last 67.48) total: 4500 open/close in 64.71 seconds: 69.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1247 (time 1787785870.94 total 10.00 last 124.65) - /unlink 2560 (time 1787785880.94 total 20.01 last 131.26) - /unlink 3996 (time 1787785890.95 total 30.01 last 143.53) total: 4500 /unlink in 33.94 seconds: 132.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 690 (time 1787785908.88 total 10.01 last 68.92) - open/close 1424 (time 1787785918.88 total 20.01 last 73.39) - open/close 2124 (time 1787785928.88 total 30.02 last 69.95) - open/close 2793 (time 1787785938.89 total 40.03 last 66.84) - open/close 3548 (time 1787785948.90 total 50.03 last 75.47) - open/close 4285 (time 1787785958.90 total 60.04 last 73.68) total: 4500 open/close in 63.00 seconds: 71.43 ops/second - /unlink 1438 (time 1787785975.84 total 10.00 last 143.80) - /unlink 2867 (time 1787785985.85 total 20.00 last 142.87) - /unlink 4264 (time 1787785995.85 total 30.01 last 139.59) total: 4500 /unlink in 31.71 seconds: 141.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 820 (time 1787786011.47 total 10.01 last 81.93) - open/close 1577 (time 1787786021.49 total 20.03 last 75.56) - open/close 2315 (time 1787786031.50 total 30.03 last 73.76) - open/close 3037 (time 1787786041.51 total 40.05 last 72.11) - open/close 3752 (time 1787786051.52 total 50.06 last 71.41) - open/close 4485 (time 1787786061.53 total 60.07 last 73.24) total: 4500 open/close in 60.32 seconds: 74.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1441 (time 1787786076.30 total 10.00 last 144.04) - /unlink 2815 (time 1787786086.30 total 20.01 last 137.38) - /unlink 4165 (time 1787786096.31 total 30.01 last 134.91) total: 4500 /unlink in 32.80 seconds: 137.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 652 (time 1787786115.16 total 10.00 last 65.17) - open/close 1370 (time 1787786125.17 total 20.01 last 71.77) - open/close 2196 (time 1787786135.17 total 30.01 last 82.58) - open/close 2930 (time 1787786145.17 total 40.01 last 73.40) - open/close 3622 (time 1787786155.17 total 50.02 last 69.17) - open/close 4344 (time 1787786165.18 total 60.02 last 72.15) total: 4500 open/close in 62.04 seconds: 72.54 ops/second - /unlink 1396 (time 1787786181.52 total 10.00 last 139.57) - /unlink 2630 (time 1787786191.52 total 20.00 last 123.38) - /unlink 4006 (time 1787786201.52 total 30.01 last 137.56) total: 4500 /unlink in 33.83 seconds: 133.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 807 (time 1787786219.13 total 10.01 last 80.63) - open/close 1539 (time 1787786229.14 total 20.01 last 73.18) - open/close 2301 (time 1787786239.14 total 30.02 last 76.15) - open/close 3043 (time 1787786249.15 total 40.02 last 74.17) - open/close 3843 (time 1787786259.15 total 50.03 last 79.96) total: 4500 open/close in 58.52 seconds: 76.89 ops/second - /unlink 1495 (time 1787786281.49 total 10.00 last 149.45) - /unlink 2940 (time 1787786291.49 total 20.01 last 144.46) - /unlink 4359 (time 1787786301.49 total 30.01 last 141.82) total: 4500 /unlink in 31.14 seconds: 144.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 730 (time 1787786316.87 total 10.01 last 72.95) - open/close 1492 (time 1787786326.88 total 20.02 last 76.09) - open/close 2260 (time 1787786336.88 total 30.02 last 76.78) - open/close 3052 (time 1787786346.89 total 40.03 last 79.13) - open/close 3830 (time 1787786356.90 total 50.04 last 77.73) total: 4500 open/close in 58.33 seconds: 77.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1457 (time 1787786379.07 total 10.00 last 145.70) - /unlink 2901 (time 1787786389.08 total 20.00 last 144.38) - /unlink 4271 (time 1787786399.10 total 30.02 last 136.71) total: 4500 /unlink in 31.86 seconds: 141.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 743 (time 1787786416.21 total 10.01 last 74.22) - open/close 1501 (time 1787786426.22 total 20.02 last 75.75) - open/close 2257 (time 1787786436.22 total 30.03 last 75.54) - open/close 2896 (time 1787786446.25 total 40.05 last 63.73) - open/close 3579 (time 1787786456.26 total 50.06 last 68.22) - open/close 4204 (time 1787786466.30 total 60.10 last 62.25) total: 4500 open/close in 64.88 seconds: 69.36 ops/second - /unlink 1431 (time 1787786485.29 total 10.00 last 143.07) - /unlink 2818 (time 1787786495.30 total 20.00 last 138.67) - /unlink 4183 (time 1787786505.32 total 30.02 last 136.25) total: 4500 /unlink in 32.17 seconds: 139.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 743 (time 1787786521.63 total 10.01 last 74.19) - open/close 1457 (time 1787786531.63 total 20.02 last 71.39) - open/close 2173 (time 1787786541.64 total 30.02 last 71.57) - open/close 2958 (time 1787786551.64 total 40.02 last 78.47) - open/close 3589 (time 1787786561.66 total 50.04 last 63.01) - open/close 4166 (time 1787786571.66 total 60.05 last 57.67) total: 4500 open/close in 64.63 seconds: 69.63 ops/second - /unlink 1336 (time 1787786590.53 total 10.00 last 133.58) - /unlink 2710 (time 1787786600.53 total 20.00 last 137.40) - /unlink 3998 (time 1787786610.53 total 30.01 last 128.69) total: 4500 /unlink in 33.77 seconds: 133.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 780 (time 1787786628.68 total 10.00 last 77.99) - open/close 1463 (time 1787786638.70 total 20.02 last 68.15) - open/close 2254 (time 1787786648.71 total 30.04 last 78.99) - open/close 2977 (time 1787786658.73 total 40.05 last 72.17) - open/close 3719 (time 1787786668.73 total 50.06 last 74.15) - open/close 4354 (time 1787786678.74 total 60.06 last 63.50) total: 4500 open/close in 62.42 seconds: 72.10 ops/second - /unlink 1372 (time 1787786695.14 total 10.00 last 137.16) - /unlink 2780 (time 1787786705.15 total 20.01 last 140.68) - /unlink 4013 (time 1787786715.15 total 30.01 last 123.30) total: 4500 /unlink in 33.63 seconds: 133.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 740 (time 1787786733.98 total 10.02 last 73.84) - open/close 1398 (time 1787786743.99 total 20.03 last 65.74) - open/close 2139 (time 1787786754.02 total 30.05 last 73.92) - open/close 2833 (time 1787786764.02 total 40.06 last 69.36) - open/close 3627 (time 1787786774.02 total 50.06 last 79.40) - open/close 4357 (time 1787786784.03 total 60.06 last 73.00) total: 4500 open/close in 62.01 seconds: 72.57 ops/second - /unlink 1368 (time 1787786800.33 total 10.01 last 136.73) - /unlink 2789 (time 1787786810.33 total 20.01 last 142.07) - /unlink 4240 (time 1787786820.33 total 30.01 last 145.08) total: 4500 /unlink in 31.92 seconds: 140.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 767 (time 1787786836.80 total 10.00 last 76.67) - open/close 1465 (time 1787786846.82 total 20.02 last 69.66) - open/close 2250 (time 1787786856.82 total 30.02 last 78.49) - open/close 3029 (time 1787786866.83 total 40.03 last 77.82) - open/close 3784 (time 1787786876.88 total 50.08 last 75.19) total: 4500 open/close in 59.69 seconds: 75.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1396 (time 1787786900.43 total 10.00 last 139.59) - /unlink 2840 (time 1787786910.45 total 20.01 last 144.24) - /unlink 4281 (time 1787786920.45 total 30.02 last 143.99) total: 4500 /unlink in 31.65 seconds: 142.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 829 (time 1787786936.72 total 10.01 last 82.80) - open/close 1622 (time 1787786946.74 total 20.03 last 79.18) - open/close 2437 (time 1787786956.76 total 30.06 last 81.28) - open/close 3224 (time 1787786966.78 total 40.07 last 78.55) - open/close 3916 (time 1787786976.80 total 50.09 last 69.10) total: 4500 open/close in 57.71 seconds: 77.97 ops/second - /unlink 1426 (time 1787786998.73 total 10.01 last 142.53) - /unlink 2799 (time 1787787008.73 total 20.01 last 137.28) - /unlink 4338 (time 1787787018.73 total 30.01 last 153.88) total: 4500 /unlink in 31.37 seconds: 143.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 688 (time 1787787034.63 total 10.02 last 68.69) - open/close 1392 (time 1787787044.64 total 20.03 last 70.34) - open/close 2149 (time 1787787054.65 total 30.03 last 75.66) - open/close 2858 (time 1787787064.65 total 40.04 last 70.86) - open/close 3633 (time 1787787074.66 total 50.04 last 77.44) - open/close 4303 (time 1787787084.67 total 60.06 last 66.92) total: 4500 open/close in 62.94 seconds: 71.49 ops/second - /unlink 1202 (time 1787787103.16 total 10.00 last 120.18) - /unlink 2480 (time 1787787113.16 total 20.00 last 127.78) - /unlink 3967 (time 1787787123.17 total 30.01 last 148.57) total: 4500 /unlink in 33.99 seconds: 132.40 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95321 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg658-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg658-server unloading modules via unload_modules_local on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing unload_modules_local oleg658-server: modules unloaded.