== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:23:47 (1786008227) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg131-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg131-server oleg131-server: oleg131-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg131-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg131-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 744 (time 1786008456.04 total 10.01 last 74.34) - open/close 1509 (time 1786008466.04 total 20.01 last 76.47) - open/close 2262 (time 1786008476.05 total 30.02 last 75.21) - open/close 2909 (time 1786008486.07 total 40.04 last 64.57) - open/close 3627 (time 1786008496.08 total 50.05 last 71.76) - open/close 4254 (time 1786008506.08 total 60.05 last 62.69) total: 4500 open/close in 63.72 seconds: 70.62 ops/second - /unlink 1296 (time 1786008525.00 total 10.01 last 129.49) - /unlink 2541 (time 1786008535.00 total 20.01 last 124.48) - /unlink 3595 (time 1786008545.01 total 30.02 last 105.30) total: 4500 /unlink in 38.79 seconds: 116.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 599 (time 1786008568.87 total 10.01 last 59.83) - open/close 1291 (time 1786008578.91 total 20.05 last 68.96) - open/close 2048 (time 1786008588.91 total 30.05 last 75.66) - open/close 2763 (time 1786008598.92 total 40.06 last 71.41) - open/close 3474 (time 1786008608.92 total 50.06 last 71.10) - open/close 4237 (time 1786008618.92 total 60.06 last 76.30) total: 4500 open/close in 63.27 seconds: 71.13 ops/second - /unlink 1199 (time 1786008637.21 total 10.00 last 119.86) - /unlink 2581 (time 1786008647.22 total 20.01 last 138.16) - /unlink 4023 (time 1786008657.22 total 30.01 last 144.19) total: 4500 /unlink in 33.15 seconds: 135.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 705 (time 1786008677.17 total 10.00 last 70.50) - open/close 1578 (time 1786008687.17 total 20.00 last 87.28) - open/close 2300 (time 1786008697.18 total 30.01 last 72.13) - open/close 2952 (time 1786008707.19 total 40.02 last 65.16) - open/close 3655 (time 1786008717.20 total 50.03 last 70.22) total: 4500 open/close in 59.07 seconds: 76.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1360 (time 1786008741.19 total 10.00 last 135.96) - /unlink 2483 (time 1786008751.19 total 20.01 last 112.27) - /unlink 3325 (time 1786008761.21 total 30.02 last 84.05) - /unlink 4379 (time 1786008771.22 total 40.03 last 105.35) total: 4500 /unlink in 41.11 seconds: 109.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 665 (time 1786008786.49 total 10.00 last 66.47) - open/close 1324 (time 1786008796.49 total 20.01 last 65.89) - open/close 1999 (time 1786008806.50 total 30.02 last 67.43) - open/close 2754 (time 1786008816.51 total 40.02 last 75.44) - open/close 3565 (time 1786008826.51 total 50.03 last 81.06) - open/close 4353 (time 1786008836.54 total 60.05 last 78.60) total: 4500 open/close in 61.96 seconds: 72.63 ops/second - /unlink 1526 (time 1786008853.31 total 10.00 last 152.58) - /unlink 3039 (time 1786008863.31 total 20.00 last 151.28) total: 4500 /unlink in 29.90 seconds: 150.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 654 (time 1786008889.33 total 10.00 last 65.39) - open/close 1392 (time 1786008899.34 total 20.01 last 73.73) - open/close 2154 (time 1786008909.34 total 30.01 last 76.18) - open/close 2957 (time 1786008919.35 total 40.02 last 80.26) - open/close 3719 (time 1786008929.35 total 50.02 last 76.17) - open/close 4370 (time 1786008939.36 total 60.03 last 65.03) total: 4500 open/close in 62.05 seconds: 72.53 ops/second - /unlink 1244 (time 1786008956.55 total 10.01 last 124.33) - /unlink 2279 (time 1786008966.55 total 20.01 last 103.47) - /unlink 3527 (time 1786008976.56 total 30.02 last 124.69) total: 4500 /unlink in 39.00 seconds: 115.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 718 (time 1786009001.44 total 10.02 last 71.69) - open/close 1344 (time 1786009011.44 total 20.02 last 62.58) - open/close 2099 (time 1786009021.45 total 30.02 last 75.45) - open/close 2778 (time 1786009031.45 total 40.03 last 67.85) - open/close 3472 (time 1786009041.46 total 50.04 last 69.34) - open/close 4209 (time 1786009051.46 total 60.04 last 73.70) total: 4500 open/close in 63.55 seconds: 70.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1363 (time 1786009069.96 total 10.00 last 136.28) - /unlink 2506 (time 1786009079.97 total 20.01 last 114.22) - /unlink 3723 (time 1786009089.97 total 30.01 last 121.65) total: 4500 /unlink in 36.75 seconds: 122.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 776 (time 1786009112.04 total 10.01 last 77.52) - open/close 1594 (time 1786009122.05 total 20.01 last 81.78) - open/close 2452 (time 1786009132.05 total 30.02 last 85.74) - open/close 3155 (time 1786009142.06 total 40.03 last 70.23) - open/close 3876 (time 1786009152.07 total 50.03 last 72.08) total: 4500 open/close in 58.66 seconds: 76.72 ops/second - /unlink 1442 (time 1786009175.58 total 10.00 last 144.17) - /unlink 2860 (time 1786009185.59 total 20.00 last 141.77) - /unlink 4303 (time 1786009195.59 total 30.01 last 144.22) total: 4500 /unlink in 31.50 seconds: 142.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 795 (time 1786009212.51 total 10.00 last 79.48) - open/close 1250 (time 1786009222.51 total 20.00 last 45.49) - open/close 1938 (time 1786009232.52 total 30.01 last 68.76) - open/close 2541 (time 1786009242.53 total 40.03 last 60.20) - open/close 2942 (time 1786009252.53 total 50.03 last 40.09) - open/close 3631 (time 1786009262.54 total 60.03 last 68.89) - open/close 4430 (time 1786009272.57 total 70.06 last 79.64) total: 4500 open/close in 71.22 seconds: 63.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1339 (time 1786009288.66 total 10.00 last 133.84) - /unlink 2720 (time 1786009298.66 total 20.01 last 138.07) - /unlink 4201 (time 1786009308.66 total 30.01 last 148.08) total: 4500 /unlink in 32.65 seconds: 137.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 802 (time 1786009326.83 total 10.01 last 80.09) - open/close 1575 (time 1786009336.83 total 20.02 last 77.25) - open/close 2316 (time 1786009346.84 total 30.02 last 74.07) - open/close 3172 (time 1786009356.84 total 40.03 last 85.56) - open/close 3985 (time 1786009366.84 total 50.03 last 81.27) total: 4500 open/close in 57.73 seconds: 77.96 ops/second - /unlink 1139 (time 1786009389.00 total 10.00 last 113.87) - /unlink 2331 (time 1786009399.00 total 20.00 last 119.20) - /unlink 3416 (time 1786009409.01 total 30.01 last 108.41) total: 4500 /unlink in 38.56 seconds: 116.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 679 (time 1786009433.69 total 10.00 last 67.90) - open/close 1540 (time 1786009443.70 total 20.01 last 86.02) - open/close 2378 (time 1786009453.70 total 30.01 last 83.80) - open/close 3228 (time 1786009463.71 total 40.01 last 84.98) - open/close 4001 (time 1786009473.71 total 50.01 last 77.30) total: 4500 open/close in 56.27 seconds: 79.97 ops/second - /unlink 1126 (time 1786009497.07 total 10.01 last 112.52) - /unlink 2570 (time 1786009507.08 total 20.01 last 144.29) - /unlink 4200 (time 1786009517.09 total 30.02 last 162.93) total: 4500 /unlink in 33.04 seconds: 136.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 916 (time 1786009536.46 total 10.00 last 91.57) - open/close 1636 (time 1786009546.47 total 20.01 last 71.94) - open/close 2345 (time 1786009556.47 total 30.02 last 70.86) - open/close 3004 (time 1786009566.47 total 40.02 last 65.89) - open/close 3616 (time 1786009576.48 total 50.02 last 61.15) - open/close 4191 (time 1786009586.48 total 60.03 last 57.50) total: 4500 open/close in 63.75 seconds: 70.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1525 (time 1786009604.42 total 10.00 last 152.48) - /unlink 3069 (time 1786009614.42 total 20.01 last 154.32) - /unlink 4470 (time 1786009624.42 total 30.01 last 140.08) total: 4500 /unlink in 30.31 seconds: 148.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 839 (time 1786009639.57 total 10.01 last 83.85) - open/close 1691 (time 1786009649.57 total 20.01 last 85.14) - open/close 2510 (time 1786009659.58 total 30.02 last 81.87) - open/close 3347 (time 1786009669.58 total 40.02 last 83.65) - open/close 4087 (time 1786009679.59 total 50.03 last 73.93) total: 4500 open/close in 55.87 seconds: 80.54 ops/second - /unlink 1169 (time 1786009700.30 total 10.01 last 116.82) - /unlink 2413 (time 1786009710.30 total 20.01 last 124.38) - /unlink 3790 (time 1786009720.31 total 30.02 last 137.58) total: 4500 /unlink in 35.00 seconds: 128.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 799 (time 1786009740.30 total 10.02 last 79.75) - open/close 1367 (time 1786009750.30 total 20.02 last 56.80) - open/close 2169 (time 1786009760.32 total 30.03 last 80.09) - open/close 2920 (time 1786009770.32 total 40.04 last 75.05) - open/close 3736 (time 1786009780.33 total 50.05 last 81.51) total: 4500 open/close in 58.51 seconds: 76.91 ops/second - /unlink 1376 (time 1786009803.56 total 10.00 last 137.57) - /unlink 2591 (time 1786009813.57 total 20.01 last 121.42) - /unlink 3921 (time 1786009823.57 total 30.01 last 132.94) total: 4500 /unlink in 35.92 seconds: 125.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 552 (time 1786009845.16 total 10.00 last 55.19) - open/close 1194 (time 1786009855.21 total 20.05 last 63.87) - open/close 1797 (time 1786009865.22 total 30.06 last 60.26) - open/close 2584 (time 1786009875.23 total 40.07 last 78.63) - open/close 3300 (time 1786009885.23 total 50.08 last 71.56) - open/close 4087 (time 1786009895.23 total 60.08 last 78.70) total: 4500 open/close in 66.30 seconds: 67.88 ops/second - /unlink 1544 (time 1786009916.01 total 10.01 last 154.23) - /unlink 2822 (time 1786009926.01 total 20.02 last 127.73) - /unlink 4210 (time 1786009936.01 total 30.02 last 138.77) total: 4500 /unlink in 32.15 seconds: 139.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 809 (time 1786009953.73 total 10.00 last 80.86) - open/close 1583 (time 1786009963.76 total 20.03 last 77.21) - open/close 2177 (time 1786009973.76 total 30.03 last 59.38) - open/close 2781 (time 1786009983.77 total 40.04 last 60.35) - open/close 3429 (time 1786009993.80 total 50.07 last 64.62) - open/close 3935 (time 1786010003.81 total 60.09 last 50.51) total: 4500 open/close in 67.67 seconds: 66.50 ops/second - /unlink 1511 (time 1786010026.68 total 10.00 last 151.04) - /unlink 2778 (time 1786010036.68 total 20.01 last 126.67) - /unlink 3999 (time 1786010046.69 total 30.01 last 122.08) total: 4500 /unlink in 34.44 seconds: 130.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 770 (time 1786010066.19 total 10.00 last 76.98) - open/close 1464 (time 1786010076.19 total 20.00 last 69.39) - open/close 1848 (time 1786010086.22 total 30.04 last 38.27) - open/close 2232 (time 1786010096.22 total 40.04 last 38.39) - open/close 2723 (time 1786010106.24 total 50.06 last 49.02) - open/close 3465 (time 1786010116.25 total 60.06 last 74.16) - open/close 4211 (time 1786010126.26 total 70.07 last 74.50) total: 4500 open/close in 73.66 seconds: 61.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1367 (time 1786010144.96 total 10.00 last 136.69) - /unlink 2746 (time 1786010154.97 total 20.01 last 137.82) - /unlink 4234 (time 1786010164.97 total 30.01 last 148.75) total: 4500 /unlink in 31.72 seconds: 141.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 611 (time 1786010182.29 total 10.00 last 61.10) - open/close 1248 (time 1786010192.30 total 20.00 last 63.67) - open/close 1943 (time 1786010202.32 total 30.02 last 69.37) - open/close 2585 (time 1786010212.32 total 40.02 last 64.18) - open/close 3122 (time 1786010222.34 total 50.05 last 53.58) - open/close 3693 (time 1786010232.35 total 60.06 last 57.05) - open/close 4215 (time 1786010242.36 total 70.07 last 52.14) total: 4500 open/close in 75.84 seconds: 59.34 ops/second - /unlink 1431 (time 1786010265.05 total 10.00 last 143.06) - /unlink 2761 (time 1786010275.05 total 20.01 last 132.95) - /unlink 4234 (time 1786010285.06 total 30.01 last 147.26) total: 4500 /unlink in 32.18 seconds: 139.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 830 (time 1786010302.52 total 10.01 last 82.95) - open/close 1452 (time 1786010312.53 total 20.01 last 62.16) - open/close 2042 (time 1786010322.54 total 30.02 last 58.93) - open/close 2524 (time 1786010332.58 total 40.06 last 48.02) - open/close 2969 (time 1786010342.60 total 50.08 last 44.42) - open/close 3377 (time 1786010352.61 total 60.09 last 40.75) - open/close 3840 (time 1786010362.63 total 70.12 last 46.18) - open/close 4400 (time 1786010372.64 total 80.12 last 55.99) total: 4500 open/close in 81.64 seconds: 55.12 ops/second - /unlink 1402 (time 1786010389.43 total 10.02 last 139.95) - /unlink 2577 (time 1786010399.43 total 20.02 last 117.46) - /unlink 3622 (time 1786010409.43 total 30.03 last 104.46) total: 4500 /unlink in 36.35 seconds: 123.79 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95297 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: modules unloaded.