== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:52:12 (1779702732) Stopping clients: oleg205-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg205-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg205-server oleg205-server: oleg205-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing load_modules_local oleg205-server: Loading modules from /home/green/git/lustre-release/lustre oleg205-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg205-server: detected 4 online CPUs by sysfs oleg205-server: MODOPTS_LIBCFS= oleg205-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg205-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg205-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg205-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug -1 all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 897 (time 1779702963.25 total 10.03 last 89.44) - open/close 1708 (time 1779702973.25 total 20.03 last 81.06) - open/close 2470 (time 1779702983.27 total 30.05 last 76.09) - open/close 3292 (time 1779702993.28 total 40.06 last 82.10) - open/close 3955 (time 1779703003.29 total 50.07 last 66.24) total: 4500 open/close in 58.44 seconds: 77.00 ops/second - /unlink 1628 (time 1779703026.16 total 10.00 last 162.76) - /unlink 3071 (time 1779703036.16 total 20.01 last 144.24) - /unlink 4487 (time 1779703046.16 total 30.01 last 141.54) total: 4500 /unlink in 30.15 seconds: 149.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 672 (time 1779703062.61 total 10.01 last 67.16) - open/close 1352 (time 1779703072.61 total 20.01 last 67.98) - open/close 2140 (time 1779703082.62 total 30.02 last 78.74) - open/close 2863 (time 1779703092.63 total 40.03 last 72.19) - open/close 3472 (time 1779703102.64 total 50.04 last 60.87) - open/close 4018 (time 1779703112.65 total 60.05 last 54.54) total: 4500 open/close in 67.92 seconds: 66.26 ops/second - /unlink 1074 (time 1779703136.65 total 10.00 last 107.40) - /unlink 2471 (time 1779703146.65 total 20.00 last 139.70) - /unlink 3823 (time 1779703156.65 total 30.00 last 135.19) total: 4500 /unlink in 34.63 seconds: 129.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 865 (time 1779703176.59 total 10.00 last 86.49) - open/close 1550 (time 1779703186.59 total 20.00 last 68.48) - open/close 2104 (time 1779703196.59 total 30.01 last 55.39) - open/close 2697 (time 1779703206.60 total 40.01 last 59.25) - open/close 3408 (time 1779703216.61 total 50.02 last 71.04) - open/close 4097 (time 1779703226.62 total 60.03 last 68.85) total: 4500 open/close in 65.38 seconds: 68.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1267 (time 1779703246.91 total 10.00 last 126.67) - /unlink 2266 (time 1779703256.92 total 20.01 last 99.87) - /unlink 3314 (time 1779703266.92 total 30.01 last 104.77) total: 4500 /unlink in 38.55 seconds: 116.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 567 (time 1779703291.40 total 10.03 last 56.50) - open/close 1129 (time 1779703301.42 total 20.05 last 56.11) - open/close 1817 (time 1779703311.44 total 30.07 last 68.67) - open/close 2504 (time 1779703321.44 total 40.07 last 68.70) - open/close 3151 (time 1779703331.46 total 50.09 last 64.59) - open/close 3895 (time 1779703341.47 total 60.10 last 74.31) - open/close 4492 (time 1779703351.47 total 70.10 last 59.67) total: 4500 open/close in 70.21 seconds: 64.09 ops/second - /unlink 1047 (time 1779703367.50 total 10.02 last 104.51) - /unlink 2109 (time 1779703377.51 total 20.03 last 106.12) - /unlink 3067 (time 1779703387.51 total 30.03 last 95.79) - /unlink 4099 (time 1779703397.51 total 40.03 last 103.13) total: 4500 /unlink in 43.56 seconds: 103.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 793 (time 1779703417.47 total 10.01 last 79.25) - open/close 1371 (time 1779703427.47 total 20.01 last 57.79) - open/close 2080 (time 1779703437.49 total 30.02 last 70.79) - open/close 2855 (time 1779703447.50 total 40.03 last 77.42) - open/close 3515 (time 1779703457.51 total 50.04 last 65.95) - open/close 4209 (time 1779703467.51 total 60.04 last 69.39) total: 4500 open/close in 63.70 seconds: 70.64 ops/second - /unlink 1514 (time 1779703487.13 total 10.00 last 151.34) - /unlink 2837 (time 1779703497.14 total 20.01 last 132.25) - /unlink 4168 (time 1779703507.14 total 30.01 last 133.05) total: 4500 /unlink in 32.30 seconds: 139.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 598 (time 1779703525.63 total 10.00 last 59.78) - open/close 1037 (time 1779703535.64 total 20.02 last 43.85) - open/close 1544 (time 1779703545.64 total 30.02 last 50.69) - open/close 2006 (time 1779703555.65 total 40.02 last 46.17) - open/close 2532 (time 1779703565.67 total 50.04 last 52.51) - open/close 3105 (time 1779703575.67 total 60.04 last 57.30) - open/close 3833 (time 1779703585.68 total 70.05 last 72.73) - open/close 4489 (time 1779703595.68 total 80.06 last 65.57) total: 4500 open/close in 80.26 seconds: 56.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1375 (time 1779703611.46 total 10.00 last 137.46) - /unlink 2675 (time 1779703621.47 total 20.01 last 129.94) - /unlink 3720 (time 1779703631.47 total 30.01 last 104.44) total: 4500 /unlink in 36.53 seconds: 123.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 799 (time 1779703654.32 total 10.00 last 79.89) - open/close 1440 (time 1779703664.32 total 20.01 last 64.07) - open/close 2075 (time 1779703674.33 total 30.01 last 63.47) - open/close 2774 (time 1779703684.33 total 40.02 last 69.88) - open/close 3417 (time 1779703694.34 total 50.02 last 64.27) - open/close 4054 (time 1779703704.34 total 60.02 last 63.70) total: 4500 open/close in 65.76 seconds: 68.43 ops/second - /unlink 1443 (time 1779703725.55 total 10.00 last 144.27) - /unlink 2669 (time 1779703735.55 total 20.00 last 122.57) - /unlink 4037 (time 1779703745.57 total 30.03 last 136.51) total: 4500 /unlink in 34.22 seconds: 131.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 652 (time 1779703765.76 total 10.00 last 65.18) - open/close 1380 (time 1779703775.76 total 20.01 last 72.77) - open/close 2094 (time 1779703785.76 total 30.01 last 71.39) - open/close 2762 (time 1779703795.76 total 40.01 last 66.79) - open/close 3265 (time 1779703805.77 total 50.01 last 50.29) - open/close 3885 (time 1779703815.77 total 60.01 last 61.99) total: 4500 open/close in 68.34 seconds: 65.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1472 (time 1779703839.38 total 10.01 last 147.06) - /unlink 2842 (time 1779703849.38 total 20.01 last 136.94) - /unlink 4116 (time 1779703859.39 total 30.02 last 127.34) total: 4500 /unlink in 33.80 seconds: 133.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 596 (time 1779703880.05 total 10.03 last 59.42) - open/close 1170 (time 1779703890.06 total 20.04 last 57.35) - open/close 1794 (time 1779703900.06 total 30.04 last 62.40) - open/close 2352 (time 1779703910.08 total 40.06 last 55.67) - open/close 2966 (time 1779703920.09 total 50.07 last 61.37) - open/close 3733 (time 1779703930.09 total 60.07 last 76.69) - open/close 4474 (time 1779703940.10 total 70.09 last 73.97) total: 4500 open/close in 70.46 seconds: 63.86 ops/second - /unlink 943 (time 1779703957.93 total 10.00 last 94.27) - /unlink 2318 (time 1779703967.93 total 20.00 last 137.49) - /unlink 3560 (time 1779703977.95 total 30.02 last 124.03) total: 4500 /unlink in 39.07 seconds: 115.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 705 (time 1779704002.44 total 10.00 last 70.48) - open/close 1498 (time 1779704012.44 total 20.01 last 79.26) - open/close 2304 (time 1779704022.45 total 30.01 last 80.56) - open/close 2895 (time 1779704032.45 total 40.02 last 59.06) - open/close 3379 (time 1779704042.49 total 50.06 last 48.23) - open/close 4052 (time 1779704052.50 total 60.06 last 67.25) total: 4500 open/close in 67.16 seconds: 67.00 ops/second - /unlink 1443 (time 1779704075.19 total 10.01 last 144.18) - /unlink 2695 (time 1779704085.19 total 20.01 last 125.14) - /unlink 4008 (time 1779704095.20 total 30.02 last 131.24) total: 4500 /unlink in 33.41 seconds: 134.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 799 (time 1779704114.03 total 10.01 last 79.83) - open/close 1431 (time 1779704124.04 total 20.02 last 63.12) - open/close 2151 (time 1779704134.05 total 30.03 last 71.91) - open/close 2859 (time 1779704144.06 total 40.04 last 70.75) - open/close 3514 (time 1779704154.06 total 50.04 last 65.50) - open/close 4136 (time 1779704164.06 total 60.04 last 62.19) total: 4500 open/close in 65.99 seconds: 68.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 876 (time 1779704186.25 total 10.01 last 87.52) - /unlink 2345 (time 1779704196.26 total 20.02 last 146.81) - /unlink 3607 (time 1779704206.26 total 30.02 last 126.14) total: 4500 /unlink in 37.15 seconds: 121.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 543 (time 1779704228.86 total 10.00 last 54.29) - open/close 1228 (time 1779704238.86 total 20.00 last 68.49) - open/close 1877 (time 1779704248.89 total 30.02 last 64.77) - open/close 2481 (time 1779704258.90 total 40.03 last 60.34) - open/close 3182 (time 1779704268.91 total 50.05 last 69.99) - open/close 3811 (time 1779704278.92 total 60.06 last 62.86) - open/close 4461 (time 1779704288.92 total 70.06 last 64.98) total: 4500 open/close in 70.47 seconds: 63.86 ops/second - /unlink 1353 (time 1779704304.88 total 10.00 last 135.30) - /unlink 2425 (time 1779704314.90 total 20.02 last 106.99) - /unlink 3810 (time 1779704324.90 total 30.02 last 138.46) total: 4500 /unlink in 36.14 seconds: 124.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 623 (time 1779704346.82 total 10.00 last 62.27) - open/close 1271 (time 1779704356.84 total 20.02 last 64.67) - open/close 1883 (time 1779704366.84 total 30.03 last 61.17) - open/close 2581 (time 1779704376.85 total 40.03 last 69.77) - open/close 3349 (time 1779704386.86 total 50.04 last 76.74) - open/close 4082 (time 1779704396.86 total 60.05 last 73.26) total: 4500 open/close in 66.89 seconds: 67.28 ops/second - /unlink 1128 (time 1779704419.29 total 10.00 last 112.79) - /unlink 2355 (time 1779704429.29 total 20.00 last 122.67) - /unlink 3461 (time 1779704439.30 total 30.01 last 110.56) total: 4500 /unlink in 38.61 seconds: 116.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 573 (time 1779704464.39 total 10.01 last 57.25) - open/close 1225 (time 1779704474.40 total 20.01 last 65.18) - open/close 1902 (time 1779704484.40 total 30.01 last 67.69) - open/close 2635 (time 1779704494.41 total 40.02 last 73.22) - open/close 3217 (time 1779704504.42 total 50.03 last 58.16) - open/close 3645 (time 1779704514.44 total 60.05 last 42.71) - open/close 4210 (time 1779704524.44 total 70.05 last 56.49) total: 4500 open/close in 74.86 seconds: 60.11 ops/second - /unlink 1215 (time 1779704544.42 total 10.00 last 121.46) - /unlink 2215 (time 1779704554.43 total 20.01 last 99.92) - /unlink 3349 (time 1779704564.44 total 30.02 last 113.31) - /unlink 4444 (time 1779704574.44 total 40.02 last 109.46) total: 4500 /unlink in 40.62 seconds: 110.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 678 (time 1779704590.16 total 10.00 last 67.77) - open/close 1317 (time 1779704600.16 total 20.01 last 63.86) - open/close 2000 (time 1779704610.17 total 30.02 last 68.23) - open/close 2693 (time 1779704620.18 total 40.03 last 69.25) - open/close 3321 (time 1779704630.19 total 50.04 last 62.74) - open/close 4040 (time 1779704640.20 total 60.05 last 71.83) total: 4500 open/close in 66.24 seconds: 67.93 ops/second - /unlink 1119 (time 1779704662.04 total 10.01 last 111.76) - /unlink 2290 (time 1779704672.04 total 20.02 last 117.05) - /unlink 3707 (time 1779704682.05 total 30.02 last 141.63) total: 4500 /unlink in 38.12 seconds: 118.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 464 (time 1779704705.49 total 10.02 last 46.31) - open/close 974 (time 1779704715.50 total 20.02 last 50.97) - open/close 1543 (time 1779704725.51 total 30.03 last 56.87) - open/close 2130 (time 1779704735.52 total 40.04 last 58.64) - open/close 2760 (time 1779704745.53 total 50.05 last 62.92) - open/close 3335 (time 1779704755.53 total 60.06 last 57.47) - open/close 4004 (time 1779704765.55 total 70.07 last 66.82) total: 4500 open/close in 77.52 seconds: 58.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1139 (time 1779704788.28 total 10.00 last 113.85) - /unlink 2261 (time 1779704798.28 total 20.00 last 112.20) - /unlink 3571 (time 1779704808.28 total 30.00 last 131.00) total: 4500 /unlink in 37.24 seconds: 120.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 630 (time 1779704831.96 total 10.01 last 62.96) - open/close 1283 (time 1779704841.96 total 20.01 last 65.27) - open/close 1945 (time 1779704851.97 total 30.02 last 66.10) - open/close 2680 (time 1779704861.98 total 40.03 last 73.42) - open/close 3481 (time 1779704872.00 total 50.05 last 80.01) - open/close 4179 (time 1779704882.03 total 60.08 last 69.59) total: 4500 open/close in 65.12 seconds: 69.10 ops/second - /unlink 1136 (time 1779704903.67 total 10.02 last 113.36) - /unlink 2180 (time 1779704913.67 total 20.03 last 104.32) - /unlink 3179 (time 1779704923.68 total 30.03 last 99.87) - /unlink 4177 (time 1779704933.68 total 40.04 last 99.73) total: 4500 /unlink in 44.40 seconds: 101.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 735 (time 1779704954.73 total 10.00 last 73.47) - open/close 1401 (time 1779704964.74 total 20.01 last 66.54) - open/close 2167 (time 1779704974.75 total 30.02 last 76.56) - open/close 2880 (time 1779704984.75 total 40.02 last 71.27) - open/close 3474 (time 1779704994.76 total 50.03 last 59.34) - open/close 4159 (time 1779705004.77 total 60.04 last 68.44) total: 4500 open/close in 64.84 seconds: 69.41 ops/second - /unlink 999 (time 1779705024.11 total 10.00 last 99.88) - /unlink 1782 (time 1779705034.13 total 20.02 last 78.15) - /unlink 2666 (time 1779705044.14 total 30.04 last 88.25) - /unlink 3613 (time 1779705054.15 total 40.04 last 94.66) total: 4500 /unlink in 48.78 seconds: 92.25 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 93953 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg205-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg205-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg205-server stop mds service on oleg205-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg205-server unloading modules via unload_modules_local on: 'oleg205-server' oleg205-server: oleg205-server.virtnet: executing unload_modules_local oleg205-server: modules unloaded.