== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:10:00 (1786309800) Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg202-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server oleg202-server: oleg202-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing load_modules_local oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg202-server: Loading modules from /home/green/git/lustre-release/lustre oleg202-server: detected 4 online CPUs by sysfs oleg202-server: MODOPTS_LIBCFS= oleg202-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg202-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg202-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg202-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 717 (time 1786310020.92 total 10.00 last 71.67) - open/close 1406 (time 1786310030.93 total 20.01 last 68.86) - open/close 2096 (time 1786310040.93 total 30.01 last 68.99) - open/close 2792 (time 1786310050.94 total 40.02 last 69.56) - open/close 3409 (time 1786310060.95 total 50.03 last 61.63) - open/close 4080 (time 1786310070.95 total 60.03 last 67.07) total: 4500 open/close in 65.90 seconds: 68.29 ops/second - /unlink 1419 (time 1786310091.68 total 10.00 last 141.83) - /unlink 2735 (time 1786310101.68 total 20.00 last 131.60) - /unlink 4102 (time 1786310111.69 total 30.01 last 136.65) total: 4500 /unlink in 33.00 seconds: 136.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 655 (time 1786310129.60 total 10.01 last 65.46) - open/close 1236 (time 1786310139.60 total 20.01 last 58.08) - open/close 1980 (time 1786310149.61 total 30.01 last 74.38) - open/close 2710 (time 1786310159.61 total 40.01 last 72.99) - open/close 3425 (time 1786310169.63 total 50.03 last 71.35) - open/close 4168 (time 1786310179.63 total 60.04 last 74.27) total: 4500 open/close in 64.65 seconds: 69.61 ops/second - /unlink 1469 (time 1786310199.21 total 10.00 last 146.86) - /unlink 2727 (time 1786310209.21 total 20.00 last 125.77) - /unlink 3989 (time 1786310219.22 total 30.01 last 126.17) total: 4500 /unlink in 34.51 seconds: 130.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 776 (time 1786310238.80 total 10.01 last 77.55) - open/close 1490 (time 1786310248.81 total 20.02 last 71.33) - open/close 2221 (time 1786310258.82 total 30.02 last 73.05) - open/close 2961 (time 1786310268.82 total 40.03 last 73.98) - open/close 3659 (time 1786310278.82 total 50.03 last 69.78) - open/close 4404 (time 1786310288.83 total 60.04 last 74.42) total: 4500 open/close in 61.21 seconds: 73.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1305 (time 1786310305.62 total 10.01 last 130.41) - /unlink 2694 (time 1786310315.62 total 20.01 last 138.80) - /unlink 4051 (time 1786310325.63 total 30.02 last 135.64) total: 4500 /unlink in 33.19 seconds: 135.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 536 (time 1786310344.35 total 10.00 last 53.57) - open/close 1107 (time 1786310354.35 total 20.01 last 57.09) - open/close 1715 (time 1786310364.35 total 30.01 last 60.80) - open/close 2449 (time 1786310374.36 total 40.02 last 73.30) - open/close 3212 (time 1786310384.37 total 50.03 last 76.26) - open/close 3989 (time 1786310394.38 total 60.04 last 77.60) total: 4500 open/close in 68.39 seconds: 65.80 ops/second - /unlink 1173 (time 1786310418.24 total 10.00 last 117.29) - /unlink 2430 (time 1786310428.24 total 20.00 last 125.67) - /unlink 3865 (time 1786310438.25 total 30.01 last 143.34) total: 4500 /unlink in 34.54 seconds: 130.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 801 (time 1786310457.99 total 10.01 last 80.04) - open/close 1624 (time 1786310467.99 total 20.01 last 82.29) - open/close 2408 (time 1786310477.99 total 30.01 last 78.36) - open/close 3177 (time 1786310488.01 total 40.03 last 76.81) - open/close 4025 (time 1786310498.02 total 50.04 last 84.72) total: 4500 open/close in 55.10 seconds: 81.67 ops/second - /unlink 1533 (time 1786310517.32 total 10.01 last 153.20) - /unlink 2875 (time 1786310527.33 total 20.02 last 134.08) - /unlink 4320 (time 1786310537.33 total 30.02 last 144.49) total: 4500 /unlink in 31.37 seconds: 143.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 731 (time 1786310553.81 total 10.02 last 72.98) - open/close 1461 (time 1786310563.81 total 20.02 last 72.96) - open/close 2261 (time 1786310573.82 total 30.03 last 79.93) - open/close 3101 (time 1786310583.82 total 40.03 last 83.99) - open/close 3845 (time 1786310593.82 total 50.03 last 74.39) total: 4500 open/close in 58.14 seconds: 77.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1157 (time 1786310615.87 total 10.01 last 115.64) - /unlink 2504 (time 1786310625.88 total 20.02 last 134.54) - /unlink 3871 (time 1786310635.89 total 30.02 last 136.65) total: 4500 /unlink in 34.10 seconds: 131.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 811 (time 1786310654.77 total 10.00 last 81.09) - open/close 1604 (time 1786310664.78 total 20.01 last 79.23) - open/close 2399 (time 1786310674.79 total 30.02 last 79.41) - open/close 3174 (time 1786310684.79 total 40.02 last 77.50) - open/close 3911 (time 1786310694.79 total 50.02 last 73.68) total: 4500 open/close in 57.51 seconds: 78.25 ops/second - /unlink 1501 (time 1786310716.60 total 10.00 last 150.07) - /unlink 2960 (time 1786310726.60 total 20.01 last 145.83) - /unlink 4278 (time 1786310736.61 total 30.01 last 131.72) total: 4500 /unlink in 31.91 seconds: 141.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 752 (time 1786310752.09 total 10.00 last 75.20) - open/close 1545 (time 1786310762.11 total 20.02 last 79.11) - open/close 2332 (time 1786310772.12 total 30.03 last 78.64) - open/close 3058 (time 1786310782.13 total 40.05 last 72.50) - open/close 3848 (time 1786310792.13 total 50.05 last 78.99) total: 4500 open/close in 58.86 seconds: 76.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1287 (time 1786310816.10 total 10.01 last 128.58) - /unlink 2790 (time 1786310826.10 total 20.01 last 150.23) - /unlink 4355 (time 1786310836.11 total 30.02 last 156.43) total: 4500 /unlink in 31.14 seconds: 144.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 770 (time 1786310851.75 total 10.02 last 76.83) - open/close 1409 (time 1786310861.76 total 20.03 last 63.86) - open/close 2107 (time 1786310871.76 total 30.03 last 69.79) - open/close 2878 (time 1786310881.77 total 40.04 last 77.04) - open/close 3455 (time 1786310891.78 total 50.05 last 57.64) - open/close 4144 (time 1786310901.80 total 60.07 last 68.71) total: 4500 open/close in 65.56 seconds: 68.64 ops/second - /unlink 1183 (time 1786310923.06 total 10.00 last 118.25) - /unlink 2311 (time 1786310933.07 total 20.02 last 112.61) - /unlink 3161 (time 1786310943.08 total 30.03 last 84.97) - /unlink 3997 (time 1786310953.08 total 40.03 last 83.58) total: 4500 /unlink in 44.27 seconds: 101.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 555 (time 1786310973.20 total 10.02 last 55.41) - open/close 1082 (time 1786310983.20 total 20.02 last 52.66) - open/close 1788 (time 1786310993.23 total 30.05 last 70.39) - open/close 2507 (time 1786311003.24 total 40.06 last 71.82) - open/close 3180 (time 1786311013.28 total 50.10 last 67.04) - open/close 3904 (time 1786311023.29 total 60.11 last 72.34) total: 4500 open/close in 67.85 seconds: 66.33 ops/second - /unlink 1385 (time 1786311045.42 total 10.00 last 138.47) - /unlink 2888 (time 1786311055.42 total 20.00 last 150.28) - /unlink 4049 (time 1786311065.44 total 30.02 last 115.92) total: 4500 /unlink in 33.61 seconds: 133.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 531 (time 1786311084.78 total 10.02 last 52.98) - open/close 1211 (time 1786311094.79 total 20.03 last 67.95) - open/close 1957 (time 1786311104.79 total 30.03 last 74.58) - open/close 2716 (time 1786311114.80 total 40.05 last 75.79) - open/close 3393 (time 1786311124.82 total 50.06 last 67.59) - open/close 4153 (time 1786311134.83 total 60.07 last 75.94) total: 4500 open/close in 65.27 seconds: 68.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1428 (time 1786311154.78 total 10.00 last 142.75) - /unlink 2679 (time 1786311164.78 total 20.01 last 125.04) - /unlink 4020 (time 1786311174.79 total 30.01 last 134.06) total: 4500 /unlink in 34.03 seconds: 132.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 594 (time 1786311193.90 total 10.02 last 59.29) - open/close 1221 (time 1786311203.91 total 20.02 last 62.67) - open/close 1852 (time 1786311213.92 total 30.04 last 63.02) - open/close 2445 (time 1786311223.93 total 40.04 last 59.26) - open/close 3117 (time 1786311233.93 total 50.05 last 67.17) - open/close 3788 (time 1786311243.93 total 60.05 last 67.08) total: 4500 open/close in 69.67 seconds: 64.59 ops/second - /unlink 1204 (time 1786311269.82 total 10.00 last 120.34) - /unlink 2321 (time 1786311279.83 total 20.01 last 111.64) - /unlink 3748 (time 1786311289.83 total 30.01 last 142.67) total: 4500 /unlink in 35.78 seconds: 125.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 607 (time 1786311310.72 total 10.05 last 60.39) - open/close 1244 (time 1786311320.72 total 20.06 last 63.65) - open/close 1909 (time 1786311330.73 total 30.07 last 66.43) - open/close 2643 (time 1786311340.76 total 40.09 last 73.24) - open/close 3398 (time 1786311350.76 total 50.10 last 75.48) - open/close 4050 (time 1786311360.77 total 60.10 last 65.14) total: 4500 open/close in 67.25 seconds: 66.91 ops/second - /unlink 1083 (time 1786311385.18 total 10.01 last 108.21) - /unlink 2283 (time 1786311395.20 total 20.03 last 119.80) - /unlink 3582 (time 1786311405.20 total 30.03 last 129.86) total: 4500 /unlink in 38.04 seconds: 118.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 756 (time 1786311428.68 total 10.01 last 75.55) - open/close 1407 (time 1786311438.71 total 20.05 last 64.85) - open/close 2103 (time 1786311448.72 total 30.05 last 69.59) - open/close 2752 (time 1786311458.72 total 40.05 last 64.86) - open/close 3405 (time 1786311468.73 total 50.06 last 65.27) - open/close 4011 (time 1786311478.73 total 60.06 last 60.56) total: 4500 open/close in 68.14 seconds: 66.04 ops/second - /unlink 1221 (time 1786311502.43 total 10.00 last 122.07) - /unlink 2375 (time 1786311512.44 total 20.01 last 115.33) - /unlink 3699 (time 1786311522.44 total 30.01 last 132.37) total: 4500 /unlink in 36.98 seconds: 121.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 620 (time 1786311544.93 total 10.00 last 61.98) - open/close 1237 (time 1786311554.93 total 20.01 last 61.69) - open/close 1906 (time 1786311564.94 total 30.01 last 66.89) - open/close 2535 (time 1786311574.94 total 40.01 last 62.88) - open/close 3252 (time 1786311584.94 total 50.01 last 71.69) - open/close 3924 (time 1786311594.95 total 60.02 last 67.13) total: 4500 open/close in 68.81 seconds: 65.39 ops/second - /unlink 1017 (time 1786311619.76 total 10.01 last 101.58) - /unlink 2106 (time 1786311629.77 total 20.02 last 108.85) - /unlink 3411 (time 1786311639.77 total 30.02 last 130.46) - /unlink 4491 (time 1786311649.78 total 40.03 last 107.91) total: 4500 /unlink in 40.12 seconds: 112.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 598 (time 1786311664.82 total 10.00 last 59.79) - open/close 1224 (time 1786311674.82 total 20.00 last 62.60) - open/close 1969 (time 1786311684.83 total 30.01 last 74.44) - open/close 2530 (time 1786311694.83 total 40.02 last 56.07) - open/close 3272 (time 1786311704.84 total 50.03 last 74.13) - open/close 4035 (time 1786311714.88 total 60.06 last 76.01) total: 4500 open/close in 66.92 seconds: 67.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1233 (time 1786311736.72 total 10.00 last 123.28) - /unlink 2385 (time 1786311746.73 total 20.01 last 115.13) - /unlink 3605 (time 1786311756.74 total 30.01 last 121.91) total: 4500 /unlink in 37.14 seconds: 121.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 754 (time 1786311778.83 total 10.01 last 75.33) - open/close 1596 (time 1786311788.83 total 20.01 last 84.17) - open/close 2353 (time 1786311798.84 total 30.02 last 75.62) - open/close 3099 (time 1786311808.85 total 40.03 last 74.55) - open/close 3756 (time 1786311818.86 total 50.05 last 65.59) - open/close 4495 (time 1786311828.86 total 60.05 last 73.90) total: 4500 open/close in 60.11 seconds: 74.86 ops/second - /unlink 1025 (time 1786311844.49 total 10.00 last 102.47) - /unlink 1978 (time 1786311854.51 total 20.02 last 95.14) - /unlink 3312 (time 1786311864.51 total 30.02 last 133.36) total: 4500 /unlink in 38.47 seconds: 116.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 590 (time 1786311889.63 total 10.01 last 58.94) - open/close 1327 (time 1786311899.64 total 20.02 last 73.64) - open/close 1976 (time 1786311909.64 total 30.02 last 64.90) - open/close 2627 (time 1786311919.65 total 40.03 last 65.03) - open/close 3360 (time 1786311929.66 total 50.04 last 73.26) - open/close 4032 (time 1786311939.66 total 60.04 last 67.20) total: 4500 open/close in 66.81 seconds: 67.36 ops/second - /unlink 1013 (time 1786311961.93 total 10.00 last 101.26) - /unlink 2410 (time 1786311971.93 total 20.00 last 139.68) - /unlink 3604 (time 1786311981.95 total 30.02 last 119.24) total: 4500 /unlink in 39.48 seconds: 113.98 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95420 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg202-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server unloading modules via unload_modules_local on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing unload_modules_local oleg202-server: modules unloaded.