== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:28:43 (1787387323) Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg210-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg210-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg210-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg210-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 668 (time 1787387544.48 total 10.01 last 66.72) - open/close 1317 (time 1787387554.50 total 20.03 last 64.76) - open/close 1825 (time 1787387564.52 total 30.04 last 50.74) - open/close 2456 (time 1787387574.54 total 40.07 last 62.92) - open/close 3300 (time 1787387584.54 total 50.07 last 84.40) - open/close 3936 (time 1787387594.55 total 60.08 last 63.58) total: 4500 open/close in 68.57 seconds: 65.63 ops/second - /unlink 1398 (time 1787387617.09 total 10.01 last 139.69) - /unlink 2692 (time 1787387627.09 total 20.01 last 129.35) - /unlink 3942 (time 1787387637.10 total 30.01 last 124.98) total: 4500 /unlink in 34.18 seconds: 131.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 752 (time 1787387656.26 total 10.03 last 75.00) - open/close 1364 (time 1787387666.27 total 20.04 last 61.14) - open/close 1845 (time 1787387676.29 total 30.06 last 48.01) - open/close 2307 (time 1787387686.29 total 40.06 last 46.18) - open/close 2925 (time 1787387696.29 total 50.06 last 61.79) - open/close 3558 (time 1787387706.29 total 60.06 last 63.29) - open/close 4143 (time 1787387716.29 total 70.06 last 58.49) total: 4500 open/close in 75.58 seconds: 59.54 ops/second - /unlink 1231 (time 1787387736.99 total 10.00 last 123.04) - /unlink 2468 (time 1787387746.99 total 20.01 last 123.66) - /unlink 3785 (time 1787387756.99 total 30.01 last 131.65) total: 4500 /unlink in 34.89 seconds: 128.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 676 (time 1787387776.78 total 10.02 last 67.47) - open/close 1318 (time 1787387786.79 total 20.03 last 64.15) - open/close 1964 (time 1787387796.80 total 30.03 last 64.56) - open/close 2670 (time 1787387806.80 total 40.04 last 70.58) - open/close 3304 (time 1787387816.80 total 50.04 last 63.38) - open/close 4070 (time 1787387826.81 total 60.05 last 76.54) total: 4500 open/close in 65.32 seconds: 68.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1321 (time 1787387845.80 total 10.01 last 132.01) - /unlink 2400 (time 1787387855.80 total 20.01 last 107.85) - /unlink 3665 (time 1787387865.81 total 30.02 last 126.40) total: 4500 /unlink in 36.44 seconds: 123.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 764 (time 1787387886.32 total 10.01 last 76.36) - open/close 1493 (time 1787387896.32 total 20.01 last 72.84) - open/close 2132 (time 1787387906.33 total 30.02 last 63.85) - open/close 2690 (time 1787387916.35 total 40.04 last 55.72) - open/close 3178 (time 1787387926.38 total 50.07 last 48.64) - open/close 3945 (time 1787387936.40 total 60.09 last 76.58) total: 4500 open/close in 67.56 seconds: 66.60 ops/second - /unlink 1383 (time 1787387958.76 total 10.00 last 138.26) - /unlink 2787 (time 1787387968.76 total 20.00 last 140.38) - /unlink 4141 (time 1787387978.76 total 30.01 last 135.38) total: 4500 /unlink in 32.99 seconds: 136.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 700 (time 1787387997.10 total 10.01 last 69.95) - open/close 1283 (time 1787388007.11 total 20.01 last 58.26) - open/close 1951 (time 1787388017.12 total 30.02 last 66.74) - open/close 2808 (time 1787388027.12 total 40.03 last 85.66) - open/close 3470 (time 1787388037.12 total 50.03 last 66.18) - open/close 4087 (time 1787388047.13 total 60.04 last 61.68) total: 4500 open/close in 68.03 seconds: 66.15 ops/second - /unlink 1107 (time 1787388070.25 total 10.00 last 110.65) - /unlink 2334 (time 1787388080.26 total 20.01 last 122.65) - /unlink 3690 (time 1787388090.26 total 30.01 last 135.54) total: 4500 /unlink in 36.52 seconds: 123.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 607 (time 1787388112.49 total 10.01 last 60.63) - open/close 1365 (time 1787388122.49 total 20.01 last 75.79) - open/close 2082 (time 1787388132.49 total 30.01 last 71.69) - open/close 2820 (time 1787388142.49 total 40.02 last 73.77) - open/close 3458 (time 1787388152.52 total 50.04 last 63.65) - open/close 4244 (time 1787388162.52 total 60.04 last 78.58) total: 4500 open/close in 63.47 seconds: 70.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1307 (time 1787388180.69 total 10.00 last 130.66) - /unlink 2607 (time 1787388190.69 total 20.00 last 129.99) - /unlink 3994 (time 1787388200.69 total 30.01 last 138.68) total: 4500 /unlink in 33.97 seconds: 132.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 638 (time 1787388219.76 total 10.01 last 63.76) - open/close 1337 (time 1787388229.78 total 20.03 last 69.76) - open/close 1947 (time 1787388239.79 total 30.03 last 60.98) - open/close 2723 (time 1787388249.79 total 40.03 last 77.59) - open/close 3273 (time 1787388259.80 total 50.04 last 54.93) - open/close 3966 (time 1787388269.80 total 60.04 last 69.30) total: 4500 open/close in 68.43 seconds: 65.76 ops/second - /unlink 1148 (time 1787388293.16 total 10.00 last 114.79) - /unlink 2502 (time 1787388303.17 total 20.02 last 135.21) - /unlink 3881 (time 1787388313.18 total 30.02 last 137.89) total: 4500 /unlink in 36.31 seconds: 123.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 675 (time 1787388335.54 total 10.02 last 67.39) - open/close 1202 (time 1787388345.55 total 20.02 last 52.69) - open/close 1834 (time 1787388355.55 total 30.02 last 63.16) - open/close 2397 (time 1787388365.56 total 40.03 last 56.26) - open/close 3237 (time 1787388375.57 total 50.04 last 83.95) - open/close 3993 (time 1787388385.58 total 60.05 last 75.51) total: 4500 open/close in 67.57 seconds: 66.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1175 (time 1787388408.03 total 10.00 last 117.49) - /unlink 2473 (time 1787388418.04 total 20.00 last 129.76) - /unlink 3781 (time 1787388428.04 total 30.01 last 130.77) total: 4500 /unlink in 36.24 seconds: 124.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 640 (time 1787388449.86 total 10.01 last 63.95) - open/close 1150 (time 1787388459.86 total 20.01 last 50.98) - open/close 1765 (time 1787388469.87 total 30.02 last 61.44) - open/close 2501 (time 1787388479.87 total 40.02 last 73.57) - open/close 3146 (time 1787388489.88 total 50.03 last 64.48) - open/close 3799 (time 1787388499.88 total 60.03 last 65.29) - open/close 4471 (time 1787388509.88 total 70.03 last 67.17) total: 4500 open/close in 70.54 seconds: 63.79 ops/second - /unlink 1197 (time 1787388525.97 total 10.01 last 119.60) - /unlink 2361 (time 1787388535.97 total 20.01 last 116.39) - /unlink 3620 (time 1787388545.97 total 30.01 last 125.88) total: 4500 /unlink in 36.02 seconds: 124.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 611 (time 1787388567.41 total 10.01 last 61.04) - open/close 1274 (time 1787388577.42 total 20.01 last 66.28) - open/close 1996 (time 1787388587.42 total 30.02 last 72.16) - open/close 2625 (time 1787388597.45 total 40.05 last 62.72) - open/close 3282 (time 1787388607.46 total 50.06 last 65.64) - open/close 4001 (time 1787388617.47 total 60.07 last 71.83) total: 4500 open/close in 67.49 seconds: 66.68 ops/second - /unlink 1004 (time 1787388640.74 total 10.00 last 100.36) - /unlink 1997 (time 1787388650.75 total 20.01 last 99.27) - /unlink 2851 (time 1787388660.75 total 30.01 last 85.39) - /unlink 3943 (time 1787388670.75 total 40.01 last 109.17) total: 4500 /unlink in 44.64 seconds: 100.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 640 (time 1787388690.91 total 10.00 last 63.98) - open/close 1224 (time 1787388700.92 total 20.02 last 58.33) - open/close 1771 (time 1787388710.94 total 30.03 last 54.61) - open/close 2329 (time 1787388720.94 total 40.04 last 55.77) - open/close 2950 (time 1787388730.95 total 50.04 last 62.09) - open/close 3616 (time 1787388740.96 total 60.05 last 66.49) - open/close 4304 (time 1787388750.96 total 70.06 last 68.78) total: 4500 open/close in 73.70 seconds: 61.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1033 (time 1787388771.01 total 10.01 last 103.16) - /unlink 2052 (time 1787388781.01 total 20.02 last 101.87) - /unlink 3073 (time 1787388791.02 total 30.02 last 102.02) - /unlink 4195 (time 1787388801.03 total 40.03 last 112.09) total: 4500 /unlink in 43.67 seconds: 103.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 579 (time 1787388819.50 total 10.01 last 57.84) - open/close 1229 (time 1787388829.50 total 20.01 last 64.98) - open/close 1928 (time 1787388839.50 total 30.02 last 69.88) - open/close 2669 (time 1787388849.50 total 40.02 last 74.10) - open/close 3428 (time 1787388859.52 total 50.04 last 75.75) - open/close 4018 (time 1787388869.54 total 60.05 last 58.90) total: 4500 open/close in 67.41 seconds: 66.76 ops/second - /unlink 1473 (time 1787388891.44 total 10.00 last 147.26) - /unlink 2674 (time 1787388901.45 total 20.01 last 119.98) - /unlink 3752 (time 1787388911.45 total 30.02 last 107.74) total: 4500 /unlink in 36.48 seconds: 123.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 725 (time 1787388933.97 total 10.00 last 72.48) - open/close 1418 (time 1787388943.98 total 20.01 last 69.26) - open/close 2086 (time 1787388953.99 total 30.02 last 66.73) - open/close 2860 (time 1787388964.00 total 40.03 last 77.36) - open/close 3606 (time 1787388974.00 total 50.03 last 74.57) - open/close 4252 (time 1787388984.01 total 60.04 last 64.51) total: 4500 open/close in 63.27 seconds: 71.13 ops/second - /unlink 1119 (time 1787389002.83 total 10.00 last 111.86) - /unlink 2595 (time 1787389012.85 total 20.02 last 147.42) - /unlink 3810 (time 1787389022.85 total 30.02 last 121.44) total: 4500 /unlink in 35.32 seconds: 127.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 576 (time 1787389043.21 total 10.00 last 57.58) - open/close 1097 (time 1787389053.22 total 20.01 last 52.06) - open/close 1658 (time 1787389063.22 total 30.01 last 56.09) - open/close 2300 (time 1787389073.23 total 40.02 last 64.16) - open/close 3033 (time 1787389083.24 total 50.03 last 73.23) - open/close 3612 (time 1787389093.24 total 60.03 last 57.87) - open/close 4105 (time 1787389103.25 total 70.04 last 49.28) total: 4500 open/close in 76.25 seconds: 59.02 ops/second - /unlink 1155 (time 1787389125.05 total 10.01 last 115.33) - /unlink 2247 (time 1787389135.06 total 20.02 last 109.10) - /unlink 3381 (time 1787389145.06 total 30.02 last 113.38) - /unlink 4460 (time 1787389155.08 total 40.05 last 107.68) total: 4500 /unlink in 40.73 seconds: 110.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 619 (time 1787389171.97 total 10.01 last 61.86) - open/close 1304 (time 1787389181.97 total 20.01 last 68.45) - open/close 2038 (time 1787389191.98 total 30.02 last 73.32) - open/close 2582 (time 1787389202.00 total 40.04 last 54.34) - open/close 3224 (time 1787389212.01 total 50.05 last 64.13) - open/close 3818 (time 1787389222.01 total 60.05 last 59.37) - open/close 4397 (time 1787389232.04 total 70.08 last 57.71) total: 4500 open/close in 71.83 seconds: 62.65 ops/second - /unlink 1224 (time 1787389249.13 total 10.00 last 122.34) - /unlink 2324 (time 1787389259.14 total 20.01 last 109.98) - /unlink 3701 (time 1787389269.14 total 30.01 last 137.68) total: 4500 /unlink in 37.04 seconds: 121.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 696 (time 1787389292.11 total 10.01 last 69.51) - open/close 1302 (time 1787389302.11 total 20.01 last 60.60) - open/close 1867 (time 1787389312.11 total 30.02 last 56.49) - open/close 2444 (time 1787389322.11 total 40.02 last 57.68) - open/close 2974 (time 1787389332.12 total 50.03 last 52.95) - open/close 3476 (time 1787389342.14 total 60.05 last 50.08) - open/close 3984 (time 1787389352.15 total 70.05 last 50.78) total: 4500 open/close in 77.82 seconds: 57.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1186 (time 1787389375.73 total 10.01 last 118.49) - /unlink 2386 (time 1787389385.74 total 20.01 last 119.94) - /unlink 3610 (time 1787389395.74 total 30.02 last 122.39) total: 4500 /unlink in 38.01 seconds: 118.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 727 (time 1787389419.02 total 10.04 last 72.41) - open/close 1449 (time 1787389429.03 total 20.04 last 72.18) - open/close 2213 (time 1787389439.06 total 30.07 last 76.15) - open/close 3023 (time 1787389449.06 total 40.08 last 80.99) - open/close 3782 (time 1787389459.07 total 50.08 last 75.85) - open/close 4415 (time 1787389469.08 total 60.10 last 63.18) total: 4500 open/close in 61.18 seconds: 73.56 ops/second - /unlink 1123 (time 1787389486.11 total 10.01 last 112.24) - /unlink 2391 (time 1787389496.11 total 20.01 last 126.75) - /unlink 3735 (time 1787389506.12 total 30.01 last 134.35) total: 4500 /unlink in 37.43 seconds: 120.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 506 (time 1787389529.38 total 10.00 last 50.58) - open/close 1120 (time 1787389539.39 total 20.01 last 61.37) - open/close 1729 (time 1787389549.40 total 30.03 last 60.81) - open/close 2227 (time 1787389559.42 total 40.05 last 49.69) - open/close 2807 (time 1787389569.44 total 50.06 last 57.91) - open/close 3428 (time 1787389579.44 total 60.07 last 62.07) - open/close 4120 (time 1787389589.45 total 70.07 last 69.19) total: 4500 open/close in 75.56 seconds: 59.56 ops/second - /unlink 1125 (time 1787389611.30 total 10.01 last 112.42) - /unlink 2163 (time 1787389621.30 total 20.01 last 103.79) - /unlink 3249 (time 1787389631.31 total 30.01 last 108.55) - /unlink 4421 (time 1787389641.32 total 40.02 last 117.10) total: 4500 /unlink in 41.09 seconds: 109.52 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95003 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user 'cl1': success lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user 'cl1': success Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local oleg210-server: modules unloaded.