== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:35:13 (1787146513) Stopping clients: oleg334-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg334-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg334-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg334-server oleg334-server: oleg334-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg334-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg334-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 692 (time 1787146723.55 total 10.01 last 69.13) - open/close 1357 (time 1787146733.55 total 20.01 last 66.47) - open/close 1989 (time 1787146743.56 total 30.02 last 63.19) - open/close 2645 (time 1787146753.58 total 40.04 last 65.42) - open/close 3335 (time 1787146763.59 total 50.05 last 68.99) - open/close 3972 (time 1787146773.60 total 60.06 last 63.63) total: 4500 open/close in 69.05 seconds: 65.17 ops/second - /unlink 1109 (time 1787146797.52 total 10.00 last 110.85) - /unlink 2301 (time 1787146807.53 total 20.01 last 119.15) - /unlink 3458 (time 1787146817.53 total 30.01 last 115.69) total: 4500 /unlink in 38.53 seconds: 116.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 663 (time 1787146841.58 total 10.00 last 66.29) - open/close 1326 (time 1787146851.59 total 20.01 last 66.26) - open/close 2022 (time 1787146861.60 total 30.02 last 69.52) - open/close 2718 (time 1787146871.63 total 40.05 last 69.40) - open/close 3441 (time 1787146881.64 total 50.06 last 72.24) - open/close 4138 (time 1787146891.64 total 60.06 last 69.68) total: 4500 open/close in 65.17 seconds: 69.05 ops/second - /unlink 1316 (time 1787146910.95 total 10.01 last 131.51) - /unlink 2554 (time 1787146920.96 total 20.01 last 123.78) - /unlink 3752 (time 1787146930.96 total 30.01 last 119.74) total: 4500 /unlink in 35.68 seconds: 126.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 701 (time 1787146950.95 total 10.01 last 70.06) - open/close 1422 (time 1787146960.96 total 20.01 last 72.04) - open/close 2164 (time 1787146970.97 total 30.02 last 74.14) - open/close 2879 (time 1787146980.97 total 40.03 last 71.43) - open/close 3492 (time 1787146990.98 total 50.04 last 61.27) - open/close 4143 (time 1787147000.98 total 60.04 last 65.10) total: 4500 open/close in 65.95 seconds: 68.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1284 (time 1787147021.58 total 10.00 last 128.37) - /unlink 2475 (time 1787147031.58 total 20.00 last 119.09) - /unlink 3750 (time 1787147041.58 total 30.01 last 127.45) total: 4500 /unlink in 36.43 seconds: 123.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 688 (time 1787147062.59 total 10.01 last 68.71) - open/close 1340 (time 1787147072.61 total 20.03 last 65.10) - open/close 2055 (time 1787147082.62 total 30.04 last 71.41) - open/close 2641 (time 1787147092.62 total 40.04 last 58.59) - open/close 3303 (time 1787147102.63 total 50.05 last 66.16) - open/close 3987 (time 1787147112.63 total 60.05 last 68.38) total: 4500 open/close in 67.33 seconds: 66.84 ops/second - /unlink 1349 (time 1787147133.99 total 10.00 last 134.84) - /unlink 2622 (time 1787147143.99 total 20.01 last 127.29) - /unlink 3840 (time 1787147154.00 total 30.02 last 121.65) total: 4500 /unlink in 35.35 seconds: 127.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 682 (time 1787147173.98 total 10.00 last 68.20) - open/close 1355 (time 1787147183.99 total 20.00 last 67.27) - open/close 2005 (time 1787147194.00 total 30.02 last 64.92) - open/close 2582 (time 1787147204.02 total 40.04 last 57.59) - open/close 3165 (time 1787147214.03 total 50.05 last 58.22) - open/close 3865 (time 1787147224.05 total 60.06 last 69.90) total: 4500 open/close in 68.95 seconds: 65.26 ops/second - /unlink 1402 (time 1787147246.38 total 10.01 last 140.13) - /unlink 2746 (time 1787147256.38 total 20.01 last 134.37) - /unlink 4037 (time 1787147266.38 total 30.01 last 129.09) total: 4500 /unlink in 33.40 seconds: 134.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 674 (time 1787147283.94 total 10.00 last 67.38) - open/close 1262 (time 1787147293.97 total 20.03 last 58.65) - open/close 1822 (time 1787147303.98 total 30.03 last 55.96) - open/close 2323 (time 1787147313.98 total 40.04 last 50.08) - open/close 3049 (time 1787147324.00 total 50.05 last 72.48) - open/close 3676 (time 1787147334.02 total 60.08 last 62.57) - open/close 4314 (time 1787147344.02 total 70.08 last 63.77) total: 4500 open/close in 72.79 seconds: 61.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1298 (time 1787147360.99 total 10.00 last 129.74) - /unlink 2519 (time 1787147370.99 total 20.01 last 122.09) - /unlink 3708 (time 1787147380.99 total 30.01 last 118.88) total: 4500 /unlink in 36.91 seconds: 121.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 566 (time 1787147402.49 total 10.00 last 56.58) - open/close 1297 (time 1787147412.50 total 20.01 last 73.02) - open/close 2014 (time 1787147422.52 total 30.04 last 71.53) - open/close 2650 (time 1787147432.53 total 40.05 last 63.52) - open/close 3362 (time 1787147442.54 total 50.06 last 71.16) - open/close 4056 (time 1787147452.56 total 60.07 last 69.28) total: 4500 open/close in 66.37 seconds: 67.81 ops/second - /unlink 1341 (time 1787147473.61 total 10.00 last 134.07) - /unlink 2564 (time 1787147483.61 total 20.00 last 122.28) - /unlink 3923 (time 1787147493.62 total 30.01 last 135.80) total: 4500 /unlink in 34.33 seconds: 131.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 739 (time 1787147511.92 total 10.00 last 73.87) - open/close 1472 (time 1787147521.92 total 20.01 last 73.30) - open/close 2209 (time 1787147531.93 total 30.01 last 73.66) - open/close 2926 (time 1787147541.94 total 40.02 last 71.65) - open/close 3558 (time 1787147551.94 total 50.03 last 63.15) - open/close 4271 (time 1787147561.96 total 60.04 last 71.20) total: 4500 open/close in 63.49 seconds: 70.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1332 (time 1787147579.75 total 10.01 last 133.12) - /unlink 2612 (time 1787147589.76 total 20.01 last 127.98) - /unlink 3822 (time 1787147599.76 total 30.02 last 120.90) total: 4500 /unlink in 35.88 seconds: 125.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 578 (time 1787147619.71 total 10.00 last 57.77) - open/close 1144 (time 1787147629.72 total 20.01 last 56.55) - open/close 1797 (time 1787147639.73 total 30.02 last 65.27) - open/close 2429 (time 1787147649.73 total 40.02 last 63.17) - open/close 3132 (time 1787147659.74 total 50.03 last 70.23) - open/close 3807 (time 1787147669.75 total 60.04 last 67.42) total: 4500 open/close in 69.76 seconds: 64.51 ops/second - /unlink 1276 (time 1787147694.28 total 10.02 last 127.36) - /unlink 2474 (time 1787147704.28 total 20.02 last 119.77) - /unlink 3674 (time 1787147714.29 total 30.02 last 119.96) total: 4500 /unlink in 37.40 seconds: 120.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 560 (time 1787147736.75 total 10.01 last 55.97) - open/close 1183 (time 1787147746.75 total 20.01 last 62.28) - open/close 1829 (time 1787147756.76 total 30.02 last 64.56) - open/close 2486 (time 1787147766.80 total 40.05 last 65.44) - open/close 3150 (time 1787147776.81 total 50.06 last 66.34) - open/close 3828 (time 1787147786.82 total 60.08 last 67.68) - open/close 4493 (time 1787147796.84 total 70.09 last 66.42) total: 4500 open/close in 70.19 seconds: 64.11 ops/second - /unlink 1374 (time 1787147811.24 total 10.00 last 137.36) - /unlink 2593 (time 1787147821.25 total 20.01 last 121.80) - /unlink 3934 (time 1787147831.25 total 30.01 last 134.10) total: 4500 /unlink in 33.89 seconds: 132.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 684 (time 1787147849.16 total 10.01 last 68.35) - open/close 1312 (time 1787147859.19 total 20.03 last 62.65) - open/close 1934 (time 1787147869.20 total 30.04 last 62.12) - open/close 2621 (time 1787147879.21 total 40.05 last 68.64) - open/close 3279 (time 1787147889.22 total 50.06 last 65.73) - open/close 3942 (time 1787147899.23 total 60.07 last 66.24) total: 4500 open/close in 68.38 seconds: 65.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1190 (time 1787147922.02 total 10.00 last 118.97) - /unlink 2348 (time 1787147932.02 total 20.01 last 115.76) - /unlink 3479 (time 1787147942.02 total 30.01 last 113.07) total: 4500 /unlink in 38.32 seconds: 117.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 655 (time 1787147966.31 total 10.01 last 65.42) - open/close 1213 (time 1787147976.32 total 20.02 last 55.75) - open/close 1797 (time 1787147986.33 total 30.03 last 58.33) - open/close 2436 (time 1787147996.37 total 40.07 last 63.65) - open/close 3096 (time 1787148006.38 total 50.08 last 65.95) - open/close 3808 (time 1787148016.39 total 60.09 last 71.12) total: 4500 open/close in 68.91 seconds: 65.30 ops/second - /unlink 1372 (time 1787148039.07 total 10.01 last 137.07) - /unlink 2667 (time 1787148049.07 total 20.02 last 129.40) - /unlink 3998 (time 1787148059.08 total 30.02 last 133.07) total: 4500 /unlink in 34.16 seconds: 131.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 697 (time 1787148076.79 total 10.01 last 69.60) - open/close 1389 (time 1787148086.80 total 20.02 last 69.14) - open/close 2042 (time 1787148096.82 total 30.04 last 65.21) - open/close 2739 (time 1787148106.83 total 40.05 last 69.65) - open/close 3365 (time 1787148116.84 total 50.05 last 62.54) - open/close 4038 (time 1787148126.84 total 60.06 last 67.28) total: 4500 open/close in 67.20 seconds: 66.96 ops/second - /unlink 1222 (time 1787148148.42 total 10.00 last 122.16) - /unlink 2389 (time 1787148158.43 total 20.01 last 116.60) - /unlink 3753 (time 1787148168.44 total 30.02 last 136.33) total: 4500 /unlink in 36.23 seconds: 124.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 764 (time 1787148188.40 total 10.01 last 76.30) - open/close 1424 (time 1787148198.41 total 20.02 last 65.94) - open/close 2105 (time 1787148208.43 total 30.04 last 68.00) - open/close 2754 (time 1787148218.44 total 40.05 last 64.82) - open/close 3435 (time 1787148228.45 total 50.06 last 68.01) - open/close 4119 (time 1787148238.47 total 60.08 last 68.27) total: 4500 open/close in 66.18 seconds: 68.00 ops/second - /unlink 1142 (time 1787148259.57 total 10.01 last 114.14) - /unlink 2318 (time 1787148269.57 total 20.01 last 117.57) - /unlink 3409 (time 1787148279.57 total 30.01 last 109.08) total: 4500 /unlink in 39.75 seconds: 113.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 607 (time 1787148303.83 total 10.00 last 60.69) - open/close 1239 (time 1787148313.84 total 20.01 last 63.14) - open/close 1872 (time 1787148323.84 total 30.02 last 63.27) - open/close 2504 (time 1787148333.85 total 40.02 last 63.17) - open/close 3199 (time 1787148343.85 total 50.02 last 69.49) - open/close 3833 (time 1787148353.86 total 60.03 last 63.33) total: 4500 open/close in 69.81 seconds: 64.46 ops/second - /unlink 1257 (time 1787148377.38 total 10.00 last 125.66) - /unlink 2482 (time 1787148387.39 total 20.00 last 122.49) - /unlink 3607 (time 1787148397.39 total 30.01 last 112.46) total: 4500 /unlink in 38.32 seconds: 117.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 633 (time 1787148420.29 total 10.00 last 63.29) - open/close 1250 (time 1787148430.31 total 20.02 last 61.60) - open/close 1938 (time 1787148440.31 total 30.02 last 68.76) - open/close 2519 (time 1787148450.32 total 40.03 last 58.06) - open/close 3195 (time 1787148460.32 total 50.03 last 67.57) - open/close 3877 (time 1787148470.33 total 60.04 last 68.13) total: 4500 open/close in 69.50 seconds: 64.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1220 (time 1787148493.71 total 10.02 last 121.74) - /unlink 2525 (time 1787148503.72 total 20.03 last 130.37) - /unlink 3806 (time 1787148513.76 total 30.06 last 127.67) total: 4500 /unlink in 37.50 seconds: 120.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 536 (time 1787148538.26 total 10.00 last 53.58) - open/close 1169 (time 1787148548.26 total 20.01 last 63.27) - open/close 1820 (time 1787148558.26 total 30.01 last 65.09) - open/close 2509 (time 1787148568.26 total 40.01 last 68.90) - open/close 3122 (time 1787148578.27 total 50.02 last 61.24) - open/close 3718 (time 1787148588.28 total 60.03 last 59.54) - open/close 4280 (time 1787148598.29 total 70.04 last 56.17) total: 4500 open/close in 73.74 seconds: 61.02 ops/second - /unlink 1181 (time 1787148616.52 total 10.01 last 118.02) - /unlink 2404 (time 1787148626.52 total 20.01 last 122.21) - /unlink 3593 (time 1787148636.53 total 30.02 last 118.81) total: 4500 /unlink in 37.51 seconds: 119.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 555 (time 1787148659.83 total 10.01 last 55.44) - open/close 1154 (time 1787148669.84 total 20.02 last 59.85) - open/close 1743 (time 1787148679.85 total 30.02 last 58.87) - open/close 2359 (time 1787148689.86 total 40.04 last 61.49) - open/close 3014 (time 1787148699.87 total 50.05 last 65.44) - open/close 3675 (time 1787148709.88 total 60.06 last 66.03) - open/close 4286 (time 1787148719.90 total 70.07 last 61.03) total: 4500 open/close in 73.05 seconds: 61.60 ops/second - /unlink 1221 (time 1787148737.50 total 10.00 last 122.09) - /unlink 2520 (time 1787148747.50 total 20.01 last 129.84) - /unlink 3685 (time 1787148757.51 total 30.01 last 116.46) total: 4500 /unlink in 36.58 seconds: 123.03 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95193 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: modules unloaded.