== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:51:41 (1787802701) Stopping clients: oleg119-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg119-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg119-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg119-server oleg119-server: oleg119-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg119-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg119-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 790 (time 1787802914.58 total 10.01 last 78.92) - open/close 1573 (time 1787802924.60 total 20.03 last 78.17) - open/close 2293 (time 1787802934.60 total 30.03 last 72.00) - open/close 3042 (time 1787802944.60 total 40.03 last 74.87) - open/close 3747 (time 1787802954.61 total 50.03 last 70.47) - open/close 4430 (time 1787802964.61 total 60.04 last 68.26) total: 4500 open/close in 61.17 seconds: 73.56 ops/second - /unlink 1469 (time 1787802980.33 total 10.01 last 146.78) - /unlink 3022 (time 1787802990.33 total 20.01 last 155.25) - /unlink 4473 (time 1787803000.34 total 30.02 last 144.97) total: 4500 /unlink in 30.31 seconds: 148.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 879 (time 1787803015.79 total 10.00 last 87.88) - open/close 1513 (time 1787803025.79 total 20.01 last 63.34) - open/close 2141 (time 1787803035.80 total 30.02 last 62.77) - open/close 2871 (time 1787803045.81 total 40.03 last 72.92) - open/close 3525 (time 1787803055.83 total 50.04 last 65.30) - open/close 4203 (time 1787803065.83 total 60.04 last 67.79) total: 4500 open/close in 63.99 seconds: 70.32 ops/second - /unlink 1436 (time 1787803084.72 total 10.00 last 143.57) - /unlink 2613 (time 1787803094.72 total 20.01 last 117.62) - /unlink 4052 (time 1787803104.72 total 30.01 last 143.87) total: 4500 /unlink in 32.99 seconds: 136.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 680 (time 1787803123.81 total 10.00 last 67.98) - open/close 1417 (time 1787803133.81 total 20.01 last 73.65) - open/close 2098 (time 1787803143.82 total 30.02 last 68.04) - open/close 2831 (time 1787803153.83 total 40.02 last 73.29) - open/close 3689 (time 1787803163.84 total 50.03 last 85.69) - open/close 4438 (time 1787803173.85 total 60.05 last 74.81) total: 4500 open/close in 60.68 seconds: 74.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1488 (time 1787803189.09 total 10.01 last 148.63) - /unlink 3074 (time 1787803199.11 total 20.02 last 158.41) total: 4500 /unlink in 29.02 seconds: 155.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 765 (time 1787803221.99 total 10.01 last 76.43) - open/close 1411 (time 1787803232.00 total 20.02 last 64.56) - open/close 1935 (time 1787803242.00 total 30.02 last 52.35) - open/close 2419 (time 1787803252.01 total 40.03 last 48.35) - open/close 3178 (time 1787803262.02 total 50.04 last 75.83) - open/close 3814 (time 1787803272.03 total 60.05 last 63.59) total: 4500 open/close in 68.73 seconds: 65.48 ops/second - /unlink 1409 (time 1787803295.85 total 10.00 last 140.90) - /unlink 2882 (time 1787803305.85 total 20.00 last 147.27) - /unlink 4245 (time 1787803315.87 total 30.02 last 136.06) total: 4500 /unlink in 31.94 seconds: 140.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 763 (time 1787803332.49 total 10.01 last 76.22) - open/close 1440 (time 1787803342.50 total 20.02 last 67.63) - open/close 2255 (time 1787803352.50 total 30.02 last 81.49) - open/close 3158 (time 1787803362.51 total 40.03 last 90.27) - open/close 3984 (time 1787803372.52 total 50.04 last 82.51) total: 4500 open/close in 56.56 seconds: 79.56 ops/second - /unlink 1214 (time 1787803394.24 total 10.01 last 121.28) - /unlink 2376 (time 1787803404.24 total 20.01 last 116.17) - /unlink 3422 (time 1787803414.24 total 30.02 last 104.55) - /unlink 4355 (time 1787803424.25 total 40.02 last 93.26) total: 4500 /unlink in 42.00 seconds: 107.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 592 (time 1787803442.61 total 10.01 last 59.15) - open/close 997 (time 1787803452.63 total 20.03 last 40.43) - open/close 1460 (time 1787803462.63 total 30.03 last 46.30) - open/close 2175 (time 1787803472.64 total 40.04 last 71.41) - open/close 2732 (time 1787803482.66 total 50.06 last 55.61) - open/close 3357 (time 1787803492.68 total 60.08 last 62.36) - open/close 3999 (time 1787803502.69 total 70.08 last 64.16) total: 4500 open/close in 79.52 seconds: 56.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 852 (time 1787803529.15 total 10.00 last 85.19) - /unlink 1616 (time 1787803539.16 total 20.01 last 76.35) - /unlink 2611 (time 1787803549.16 total 30.01 last 99.47) - /unlink 4156 (time 1787803559.16 total 40.01 last 154.49) total: 4500 /unlink in 42.24 seconds: 106.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 782 (time 1787803576.19 total 10.00 last 78.17) - open/close 1407 (time 1787803586.22 total 20.03 last 62.31) - open/close 1957 (time 1787803596.24 total 30.05 last 54.92) - open/close 2555 (time 1787803606.24 total 40.05 last 59.80) - open/close 3218 (time 1787803616.25 total 50.06 last 66.25) - open/close 3926 (time 1787803626.25 total 60.06 last 70.78) total: 4500 open/close in 68.38 seconds: 65.81 ops/second - /unlink 1363 (time 1787803649.32 total 10.01 last 136.22) - /unlink 2685 (time 1787803659.32 total 20.01 last 132.18) - /unlink 4003 (time 1787803669.33 total 30.01 last 131.72) total: 4500 /unlink in 33.35 seconds: 134.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 790 (time 1787803687.73 total 10.01 last 78.92) - open/close 1584 (time 1787803697.73 total 20.01 last 79.38) - open/close 2359 (time 1787803707.73 total 30.01 last 77.49) - open/close 2953 (time 1787803717.75 total 40.03 last 59.28) - open/close 3494 (time 1787803727.79 total 50.07 last 53.91) - open/close 4222 (time 1787803737.79 total 60.08 last 72.76) total: 4500 open/close in 64.32 seconds: 69.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1279 (time 1787803756.88 total 10.04 last 127.34) - /unlink 2250 (time 1787803766.88 total 20.04 last 97.09) - /unlink 3460 (time 1787803776.88 total 30.05 last 121.00) total: 4500 /unlink in 38.63 seconds: 116.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 777 (time 1787803801.36 total 10.01 last 77.62) - open/close 1486 (time 1787803811.39 total 20.03 last 70.73) - open/close 2046 (time 1787803821.39 total 30.04 last 55.98) - open/close 2588 (time 1787803831.44 total 40.09 last 53.93) - open/close 2994 (time 1787803842.96 total 51.60 last 35.26) - open/close 3434 (time 1787803852.96 total 61.61 last 43.97) - open/close 4039 (time 1787803862.97 total 71.62 last 60.45) total: 4500 open/close in 80.06 seconds: 56.21 ops/second - /unlink 1503 (time 1787803887.63 total 10.01 last 150.14) - /unlink 2714 (time 1787803897.64 total 20.02 last 121.01) - /unlink 3911 (time 1787803907.64 total 30.02 last 119.69) total: 4500 /unlink in 33.63 seconds: 133.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 644 (time 1787803926.37 total 10.00 last 64.37) - open/close 1330 (time 1787803936.38 total 20.01 last 68.55) - open/close 1936 (time 1787803946.39 total 30.03 last 60.52) - open/close 2450 (time 1787803956.41 total 40.05 last 51.29) - open/close 3190 (time 1787803966.42 total 50.06 last 73.93) - open/close 3893 (time 1787803976.43 total 60.06 last 70.26) total: 4500 open/close in 68.46 seconds: 65.73 ops/second - /unlink 989 (time 1787804000.93 total 10.01 last 98.79) - /unlink 1939 (time 1787804010.93 total 20.01 last 95.00) - /unlink 2994 (time 1787804020.93 total 30.02 last 105.42) - /unlink 4126 (time 1787804030.96 total 40.05 last 112.91) total: 4500 /unlink in 43.39 seconds: 103.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 695 (time 1787804048.91 total 10.02 last 69.36) - open/close 1279 (time 1787804058.92 total 20.03 last 58.34) - open/close 1984 (time 1787804068.92 total 30.03 last 70.49) - open/close 2589 (time 1787804078.92 total 40.04 last 60.46) - open/close 3215 (time 1787804088.93 total 50.05 last 62.53) - open/close 3896 (time 1787804098.95 total 60.06 last 68.00) total: 4500 open/close in 68.22 seconds: 65.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1676 (time 1787804121.48 total 10.00 last 167.57) - /unlink 3147 (time 1787804131.48 total 20.00 last 147.09) - /unlink 4497 (time 1787804141.49 total 30.01 last 134.91) total: 4500 /unlink in 30.03 seconds: 149.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 649 (time 1787804156.81 total 10.01 last 64.81) - open/close 1346 (time 1787804166.82 total 20.02 last 69.64) - open/close 1817 (time 1787804176.88 total 30.08 last 46.83) - open/close 2347 (time 1787804186.88 total 40.09 last 52.97) - open/close 2983 (time 1787804196.90 total 50.10 last 63.50) - open/close 3576 (time 1787804206.94 total 60.14 last 59.05) - open/close 4144 (time 1787804216.98 total 70.18 last 56.59) total: 4500 open/close in 77.16 seconds: 58.32 ops/second - /unlink 1269 (time 1787804240.14 total 10.00 last 126.90) - /unlink 2371 (time 1787804250.14 total 20.00 last 110.17) - /unlink 3393 (time 1787804260.16 total 30.03 last 101.96) - /unlink 4405 (time 1787804270.16 total 40.03 last 101.17) total: 4500 /unlink in 41.06 seconds: 109.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 648 (time 1787804287.25 total 10.00 last 64.78) - open/close 1320 (time 1787804297.26 total 20.01 last 67.15) - open/close 1968 (time 1787804307.27 total 30.02 last 64.75) - open/close 2464 (time 1787804317.27 total 40.02 last 49.57) - open/close 2973 (time 1787804327.28 total 50.03 last 50.87) - open/close 3523 (time 1787804337.28 total 60.03 last 54.97) - open/close 4325 (time 1787804347.31 total 70.06 last 80.00) total: 4500 open/close in 72.28 seconds: 62.25 ops/second - /unlink 1482 (time 1787804364.31 total 10.00 last 148.19) - /unlink 2795 (time 1787804374.31 total 20.00 last 131.29) - /unlink 4249 (time 1787804384.31 total 30.00 last 145.39) total: 4500 /unlink in 31.57 seconds: 142.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 752 (time 1787804400.62 total 10.01 last 75.12) - open/close 1569 (time 1787804410.63 total 20.02 last 81.59) - open/close 2412 (time 1787804420.64 total 30.04 last 84.20) - open/close 3228 (time 1787804430.64 total 40.04 last 81.59) - open/close 3876 (time 1787804440.65 total 50.04 last 64.78) total: 4500 open/close in 57.47 seconds: 78.30 ops/second - /unlink 1499 (time 1787804462.41 total 10.00 last 149.84) - /unlink 2853 (time 1787804472.41 total 20.01 last 135.31) - /unlink 4408 (time 1787804482.42 total 30.02 last 155.41) total: 4500 /unlink in 30.74 seconds: 146.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 786 (time 1787804497.05 total 10.02 last 78.43) - open/close 1697 (time 1787804507.05 total 20.02 last 91.09) - open/close 2572 (time 1787804517.05 total 30.03 last 87.45) - open/close 3435 (time 1787804527.05 total 40.03 last 86.28) - open/close 4358 (time 1787804537.05 total 50.03 last 92.29) total: 4500 open/close in 51.70 seconds: 87.03 ops/second - /unlink 1600 (time 1787804553.03 total 10.01 last 159.89) - /unlink 3266 (time 1787804563.03 total 20.02 last 166.44) total: 4500 /unlink in 26.62 seconds: 169.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 919 (time 1787804583.31 total 10.02 last 91.74) - open/close 1646 (time 1787804593.31 total 20.02 last 72.69) - open/close 2561 (time 1787804603.31 total 30.02 last 91.45) - open/close 3476 (time 1787804613.32 total 40.03 last 91.44) - open/close 4458 (time 1787804623.32 total 50.03 last 98.17) total: 4500 open/close in 50.63 seconds: 88.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1627 (time 1787804637.86 total 10.00 last 162.66) - /unlink 3143 (time 1787804647.86 total 20.00 last 151.58) total: 4500 /unlink in 29.58 seconds: 152.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 926 (time 1787804672.65 total 10.00 last 92.57) - open/close 1810 (time 1787804682.66 total 20.01 last 88.34) - open/close 2598 (time 1787804692.67 total 30.03 last 78.68) - open/close 3323 (time 1787804702.68 total 40.03 last 72.45) - open/close 4250 (time 1787804712.68 total 50.03 last 92.68) total: 4500 open/close in 53.09 seconds: 84.77 ops/second - /unlink 1554 (time 1787804730.85 total 10.00 last 155.33) - /unlink 3194 (time 1787804740.85 total 20.01 last 163.89) total: 4500 /unlink in 27.06 seconds: 166.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 738 (time 1787804762.04 total 10.02 last 73.68) - open/close 1518 (time 1787804772.05 total 20.02 last 77.96) - open/close 2307 (time 1787804782.07 total 30.05 last 78.70) - open/close 3175 (time 1787804792.09 total 40.06 last 86.64) - open/close 4028 (time 1787804802.10 total 50.07 last 85.23) total: 4500 open/close in 56.24 seconds: 80.01 ops/second - /unlink 1689 (time 1787804822.55 total 10.00 last 168.86) - /unlink 3172 (time 1787804832.56 total 20.01 last 148.11) total: 4500 /unlink in 27.82 seconds: 161.78 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95131 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: modules unloaded.