== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:43:53 (1787899433) Stopping clients: oleg415-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg415-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg415-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg415-server oleg415-server: oleg415-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg415-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg415-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg415-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 683 (time 1787899651.15 total 10.00 last 68.29) - open/close 1362 (time 1787899661.16 total 20.01 last 67.84) - open/close 1993 (time 1787899671.16 total 30.01 last 63.09) - open/close 2642 (time 1787899681.16 total 40.02 last 64.85) - open/close 3337 (time 1787899691.17 total 50.02 last 69.48) - open/close 4057 (time 1787899701.18 total 60.03 last 71.94) total: 4500 open/close in 66.86 seconds: 67.31 ops/second - /unlink 1223 (time 1787899722.55 total 10.00 last 122.25) - /unlink 2462 (time 1787899732.55 total 20.01 last 123.87) - /unlink 3618 (time 1787899742.55 total 30.01 last 115.57) total: 4500 /unlink in 37.88 seconds: 118.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 621 (time 1787899766.49 total 10.00 last 62.08) - open/close 1174 (time 1787899776.50 total 20.01 last 55.27) - open/close 1770 (time 1787899786.51 total 30.02 last 59.51) - open/close 2234 (time 1787899796.52 total 40.03 last 46.36) - open/close 2742 (time 1787899806.53 total 50.04 last 50.77) - open/close 3320 (time 1787899816.53 total 60.04 last 57.78) - open/close 3871 (time 1787899826.53 total 70.05 last 55.08) - open/close 4458 (time 1787899836.54 total 80.05 last 58.66) total: 4500 open/close in 81.14 seconds: 55.46 ops/second - /unlink 1124 (time 1787899853.42 total 10.00 last 112.35) - /unlink 2157 (time 1787899863.43 total 20.01 last 103.20) - /unlink 3186 (time 1787899873.43 total 30.01 last 102.89) - /unlink 4310 (time 1787899883.43 total 40.02 last 112.31) total: 4500 /unlink in 41.63 seconds: 108.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 687 (time 1787899900.75 total 10.01 last 68.60) - open/close 1378 (time 1787899910.75 total 20.01 last 69.10) - open/close 1973 (time 1787899920.76 total 30.02 last 59.45) - open/close 2680 (time 1787899930.76 total 40.02 last 70.68) - open/close 3320 (time 1787899940.77 total 50.04 last 63.92) - open/close 3972 (time 1787899950.79 total 60.05 last 65.12) total: 4500 open/close in 68.37 seconds: 65.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1077 (time 1787899973.84 total 10.01 last 107.63) - /unlink 2009 (time 1787899983.85 total 20.02 last 93.07) - /unlink 3025 (time 1787899993.85 total 30.02 last 101.60) - /unlink 4190 (time 1787900003.86 total 40.03 last 116.41) total: 4500 /unlink in 42.90 seconds: 104.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 625 (time 1787900021.85 total 10.00 last 62.50) - open/close 1231 (time 1787900031.86 total 20.01 last 60.53) - open/close 1889 (time 1787900041.87 total 30.03 last 65.69) - open/close 2572 (time 1787900051.88 total 40.04 last 68.25) - open/close 3208 (time 1787900061.89 total 50.04 last 63.56) - open/close 3832 (time 1787900071.90 total 60.05 last 62.34) - open/close 4429 (time 1787900081.92 total 70.08 last 59.54) total: 4500 open/close in 71.42 seconds: 63.00 ops/second - /unlink 920 (time 1787900100.21 total 10.01 last 91.89) - /unlink 1951 (time 1787900110.21 total 20.01 last 103.08) - /unlink 2933 (time 1787900120.21 total 30.02 last 98.16) - /unlink 4075 (time 1787900130.22 total 40.02 last 114.17) total: 4500 /unlink in 43.28 seconds: 103.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 661 (time 1787900149.39 total 10.01 last 66.06) - open/close 1305 (time 1787900159.40 total 20.02 last 64.29) - open/close 1933 (time 1787900169.41 total 30.03 last 62.77) - open/close 2548 (time 1787900179.41 total 40.03 last 61.48) - open/close 3137 (time 1787900189.43 total 50.05 last 58.82) - open/close 3785 (time 1787900199.46 total 60.08 last 64.56) - open/close 4395 (time 1787900209.47 total 70.09 last 60.96) total: 4500 open/close in 71.70 seconds: 62.76 ops/second - /unlink 1219 (time 1787900227.15 total 10.00 last 121.85) - /unlink 2229 (time 1787900237.15 total 20.01 last 100.98) - /unlink 3422 (time 1787900247.15 total 30.01 last 119.29) total: 4500 /unlink in 38.63 seconds: 116.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 615 (time 1787900271.38 total 10.00 last 61.49) - open/close 1326 (time 1787900281.40 total 20.01 last 71.01) - open/close 2011 (time 1787900291.40 total 30.02 last 68.47) - open/close 2713 (time 1787900301.40 total 40.02 last 70.18) - open/close 3366 (time 1787900311.42 total 50.04 last 65.20) - open/close 4046 (time 1787900321.43 total 60.05 last 67.94) total: 4500 open/close in 68.08 seconds: 66.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1215 (time 1787900344.58 total 10.00 last 121.47) - /unlink 2272 (time 1787900354.58 total 20.01 last 105.66) - /unlink 3419 (time 1787900364.58 total 30.01 last 114.68) total: 4500 /unlink in 38.85 seconds: 115.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 700 (time 1787900389.07 total 10.01 last 69.96) - open/close 1312 (time 1787900399.08 total 20.02 last 61.14) - open/close 1959 (time 1787900409.08 total 30.02 last 64.68) - open/close 2511 (time 1787900419.09 total 40.02 last 55.18) - open/close 3077 (time 1787900429.12 total 50.05 last 56.43) - open/close 3693 (time 1787900439.13 total 60.06 last 61.55) - open/close 4317 (time 1787900449.15 total 70.08 last 62.26) total: 4500 open/close in 72.96 seconds: 61.68 ops/second - /unlink 1199 (time 1787900467.08 total 10.00 last 119.89) - /unlink 2325 (time 1787900477.08 total 20.00 last 112.59) - /unlink 3515 (time 1787900487.08 total 30.01 last 118.96) total: 4500 /unlink in 38.17 seconds: 117.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 618 (time 1787900511.21 total 10.01 last 61.71) - open/close 1081 (time 1787900521.22 total 20.02 last 46.27) - open/close 1441 (time 1787900531.23 total 30.03 last 35.96) - open/close 2015 (time 1787900541.25 total 40.05 last 57.30) - open/close 2566 (time 1787900551.28 total 50.08 last 54.96) - open/close 3088 (time 1787900561.31 total 60.11 last 52.01) - open/close 3557 (time 1787900571.31 total 70.12 last 46.89) - open/close 4068 (time 1787900581.33 total 80.13 last 51.03) total: 4500 open/close in 87.34 seconds: 51.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1179 (time 1787900604.36 total 10.00 last 117.88) - /unlink 2338 (time 1787900614.37 total 20.01 last 115.85) - /unlink 3486 (time 1787900624.37 total 30.01 last 114.79) total: 4500 /unlink in 38.40 seconds: 117.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 606 (time 1787900647.75 total 10.00 last 60.58) - open/close 1197 (time 1787900657.76 total 20.01 last 59.04) - open/close 1852 (time 1787900667.76 total 30.01 last 65.50) - open/close 2524 (time 1787900677.79 total 40.04 last 67.04) - open/close 3107 (time 1787900687.79 total 50.04 last 58.29) - open/close 3727 (time 1787900697.80 total 60.06 last 61.90) - open/close 4351 (time 1787900707.81 total 70.06 last 62.36) total: 4500 open/close in 72.07 seconds: 62.44 ops/second - /unlink 1321 (time 1787900725.43 total 10.01 last 132.02) - /unlink 2536 (time 1787900735.44 total 20.02 last 121.31) - /unlink 3790 (time 1787900745.45 total 30.03 last 125.35) total: 4500 /unlink in 36.38 seconds: 123.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 685 (time 1787900767.30 total 10.00 last 68.47) - open/close 1308 (time 1787900777.30 total 20.01 last 62.29) - open/close 1938 (time 1787900787.31 total 30.01 last 62.98) - open/close 2574 (time 1787900797.31 total 40.01 last 63.59) - open/close 3099 (time 1787900807.31 total 50.01 last 52.49) - open/close 3687 (time 1787900817.37 total 60.07 last 58.48) - open/close 4214 (time 1787900827.37 total 70.07 last 52.68) total: 4500 open/close in 74.81 seconds: 60.15 ops/second - /unlink 1192 (time 1787900846.99 total 10.01 last 119.08) - /unlink 2344 (time 1787900857.00 total 20.02 last 115.06) - /unlink 3629 (time 1787900867.01 total 30.03 last 128.42) total: 4500 /unlink in 37.30 seconds: 120.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 622 (time 1787900889.79 total 10.03 last 62.01) - open/close 1182 (time 1787900899.80 total 20.04 last 55.95) - open/close 1747 (time 1787900909.80 total 30.05 last 56.46) - open/close 2450 (time 1787900919.80 total 40.05 last 70.30) - open/close 3120 (time 1787900929.82 total 50.06 last 66.92) - open/close 3742 (time 1787900939.82 total 60.07 last 62.15) - open/close 4256 (time 1787900949.84 total 70.08 last 51.34) total: 4500 open/close in 74.35 seconds: 60.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1014 (time 1787900968.84 total 10.01 last 101.33) - /unlink 2035 (time 1787900978.85 total 20.01 last 102.04) - /unlink 3055 (time 1787900988.85 total 30.01 last 101.98) - /unlink 4125 (time 1787900998.86 total 40.02 last 106.96) total: 4500 /unlink in 43.18 seconds: 104.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 473 (time 1787901019.25 total 10.00 last 47.29) - open/close 895 (time 1787901029.25 total 20.00 last 42.19) - open/close 1335 (time 1787901039.26 total 30.01 last 43.96) - open/close 1737 (time 1787901049.28 total 40.03 last 40.12) - open/close 2154 (time 1787901059.28 total 50.03 last 41.70) - open/close 2567 (time 1787901069.29 total 60.04 last 41.26) - open/close 2997 (time 1787901079.29 total 70.05 last 42.99) - open/close 3443 (time 1787901089.30 total 80.05 last 44.58) - open/close 3989 (time 1787901099.30 total 90.05 last 54.58) total: 4500 open/close in 99.03 seconds: 45.44 ops/second - /unlink 1161 (time 1787901124.06 total 10.00 last 116.09) - /unlink 2253 (time 1787901134.07 total 20.01 last 109.10) - /unlink 3369 (time 1787901144.08 total 30.02 last 111.49) total: 4500 /unlink in 39.28 seconds: 114.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 641 (time 1787901167.64 total 10.01 last 64.03) - open/close 1332 (time 1787901177.65 total 20.02 last 69.05) - open/close 1934 (time 1787901187.65 total 30.02 last 60.19) - open/close 2496 (time 1787901197.65 total 40.02 last 56.20) - open/close 3133 (time 1787901207.65 total 50.02 last 63.69) - open/close 3678 (time 1787901217.68 total 60.05 last 54.36) - open/close 4248 (time 1787901227.68 total 70.06 last 56.94) total: 4500 open/close in 74.31 seconds: 60.56 ops/second - /unlink 1192 (time 1787901247.72 total 10.00 last 119.19) - /unlink 2335 (time 1787901257.72 total 20.00 last 114.28) - /unlink 3531 (time 1787901267.72 total 30.00 last 119.58) total: 4500 /unlink in 37.86 seconds: 118.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 622 (time 1787901291.30 total 10.00 last 62.18) - open/close 1251 (time 1787901301.31 total 20.01 last 62.84) - open/close 1933 (time 1787901311.31 total 30.02 last 68.17) - open/close 2605 (time 1787901321.32 total 40.02 last 67.16) - open/close 3292 (time 1787901331.33 total 50.03 last 68.66) - open/close 4007 (time 1787901341.33 total 60.03 last 71.48) total: 4500 open/close in 67.90 seconds: 66.27 ops/second - /unlink 993 (time 1787901365.21 total 10.00 last 99.29) - /unlink 1865 (time 1787901375.22 total 20.01 last 87.10) - /unlink 2966 (time 1787901385.22 total 30.01 last 110.08) - /unlink 4141 (time 1787901395.23 total 40.02 last 117.39) total: 4500 /unlink in 43.10 seconds: 104.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 635 (time 1787901413.67 total 10.00 last 63.49) - open/close 1180 (time 1787901423.69 total 20.02 last 54.41) - open/close 1797 (time 1787901433.71 total 30.04 last 61.55) - open/close 2372 (time 1787901443.72 total 40.05 last 57.45) - open/close 3008 (time 1787901453.73 total 50.06 last 63.51) - open/close 3669 (time 1787901463.74 total 60.08 last 66.03) - open/close 4194 (time 1787901473.75 total 70.08 last 52.48) total: 4500 open/close in 75.87 seconds: 59.31 ops/second - /unlink 1176 (time 1787901494.95 total 10.00 last 117.56) - /unlink 2270 (time 1787901504.96 total 20.01 last 109.34) - /unlink 3501 (time 1787901514.96 total 30.01 last 123.05) total: 4500 /unlink in 39.25 seconds: 114.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 678 (time 1787901540.73 total 10.01 last 67.72) - open/close 1299 (time 1787901550.73 total 20.02 last 62.08) - open/close 1963 (time 1787901560.74 total 30.02 last 66.36) - open/close 2658 (time 1787901570.76 total 40.04 last 69.36) - open/close 3236 (time 1787901580.77 total 50.05 last 57.74) - open/close 3907 (time 1787901590.79 total 60.07 last 67.00) total: 4500 open/close in 68.50 seconds: 65.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1286 (time 1787901615.24 total 10.01 last 128.52) - /unlink 2529 (time 1787901625.25 total 20.01 last 124.21) - /unlink 3682 (time 1787901635.25 total 30.02 last 115.25) total: 4500 /unlink in 36.79 seconds: 122.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 668 (time 1787901657.69 total 10.01 last 66.76) - open/close 1337 (time 1787901667.69 total 20.01 last 66.88) - open/close 1939 (time 1787901677.70 total 30.02 last 60.14) - open/close 2627 (time 1787901687.71 total 40.03 last 68.75) - open/close 3241 (time 1787901697.71 total 50.03 last 61.35) - open/close 3910 (time 1787901707.72 total 60.04 last 66.86) total: 4500 open/close in 69.20 seconds: 65.02 ops/second - /unlink 1265 (time 1787901731.69 total 10.00 last 126.45) - /unlink 2466 (time 1787901741.70 total 20.01 last 120.08) - /unlink 3641 (time 1787901751.70 total 30.01 last 117.48) total: 4500 /unlink in 37.69 seconds: 119.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 559 (time 1787901775.34 total 10.02 last 55.80) - open/close 1119 (time 1787901785.34 total 20.02 last 55.97) - open/close 1654 (time 1787901795.37 total 30.05 last 53.34) - open/close 2199 (time 1787901805.38 total 40.06 last 54.48) - open/close 2787 (time 1787901815.38 total 50.07 last 58.76) - open/close 3478 (time 1787901825.39 total 60.07 last 69.09) - open/close 4116 (time 1787901835.39 total 70.07 last 63.78) total: 4500 open/close in 77.05 seconds: 58.40 ops/second - /unlink 1143 (time 1787901858.19 total 10.01 last 114.15) - /unlink 2049 (time 1787901868.20 total 20.01 last 90.59) - /unlink 3167 (time 1787901878.20 total 30.02 last 111.75) - /unlink 4345 (time 1787901888.20 total 40.02 last 117.78) total: 4500 /unlink in 41.49 seconds: 108.47 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95288 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server unloading modules via unload_modules_local on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local oleg415-server: modules unloaded.