== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:55:58 (1789134958) Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg427-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg427-server oleg427-server: oleg427-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg427-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 649 (time 1789135197.96 total 10.02 last 64.80) - open/close 1129 (time 1789135207.99 total 20.04 last 47.86) - open/close 1672 (time 1789135218.02 total 30.07 last 54.15) - open/close 2245 (time 1789135228.03 total 40.08 last 57.27) - open/close 2841 (time 1789135238.03 total 50.08 last 59.60) - open/close 3441 (time 1789135248.03 total 60.08 last 59.99) - open/close 4043 (time 1789135258.03 total 70.08 last 60.19) total: 4500 open/close in 77.50 seconds: 58.07 ops/second - /unlink 1281 (time 1789135281.35 total 10.00 last 128.10) - /unlink 2450 (time 1789135291.36 total 20.02 last 116.70) - /unlink 3385 (time 1789135301.38 total 30.04 last 93.32) - /unlink 4369 (time 1789135311.38 total 40.04 last 98.38) total: 4500 /unlink in 41.24 seconds: 109.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 535 (time 1789135327.51 total 10.02 last 53.39) - open/close 1062 (time 1789135337.51 total 20.02 last 52.69) - open/close 1756 (time 1789135347.51 total 30.03 last 69.36) - open/close 2387 (time 1789135357.53 total 40.04 last 63.01) - open/close 3012 (time 1789135367.55 total 50.06 last 62.39) - open/close 3613 (time 1789135377.57 total 60.08 last 59.97) - open/close 4228 (time 1789135387.57 total 70.09 last 61.47) total: 4500 open/close in 74.76 seconds: 60.20 ops/second - /unlink 1266 (time 1789135407.19 total 10.00 last 126.54) - /unlink 2297 (time 1789135417.21 total 20.02 last 102.90) - /unlink 3171 (time 1789135427.23 total 30.04 last 87.22) - /unlink 3886 (time 1789135437.24 total 40.05 last 71.44) total: 4500 /unlink in 48.70 seconds: 92.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 459 (time 1789135463.06 total 10.05 last 45.66) - open/close 1067 (time 1789135473.07 total 20.06 last 60.74) - open/close 1656 (time 1789135483.09 total 30.07 last 58.83) - open/close 2177 (time 1789135493.09 total 40.08 last 52.06) - open/close 2772 (time 1789135503.10 total 50.09 last 59.47) - open/close 3392 (time 1789135513.11 total 60.10 last 61.90) - open/close 3882 (time 1789135523.13 total 70.12 last 48.90) - open/close 4488 (time 1789135533.14 total 80.12 last 60.58) total: 4500 open/close in 80.28 seconds: 56.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1117 (time 1789135550.81 total 10.00 last 111.66) - /unlink 2126 (time 1789135560.82 total 20.01 last 100.86) - /unlink 3099 (time 1789135570.82 total 30.01 last 97.28) - /unlink 4033 (time 1789135580.83 total 40.02 last 93.28) total: 4500 /unlink in 44.73 seconds: 100.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 611 (time 1789135601.40 total 10.00 last 61.08) - open/close 1216 (time 1789135611.42 total 20.02 last 60.39) - open/close 1899 (time 1789135621.42 total 30.02 last 68.29) - open/close 2372 (time 1789135631.49 total 40.09 last 46.99) - open/close 2888 (time 1789135641.50 total 50.10 last 51.54) - open/close 3499 (time 1789135651.52 total 60.13 last 60.94) - open/close 4006 (time 1789135661.53 total 70.14 last 50.65) total: 4500 open/close in 80.11 seconds: 56.17 ops/second - /unlink 896 (time 1789135686.79 total 10.00 last 89.57) - /unlink 1760 (time 1789135696.79 total 20.01 last 86.38) - /unlink 2805 (time 1789135706.79 total 30.01 last 104.47) - /unlink 3995 (time 1789135716.80 total 40.02 last 118.92) total: 4500 /unlink in 44.37 seconds: 101.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 691 (time 1789135736.97 total 10.02 last 68.97) - open/close 1351 (time 1789135746.97 total 20.02 last 65.96) - open/close 1962 (time 1789135757.00 total 30.05 last 60.96) - open/close 2514 (time 1789135767.00 total 40.05 last 55.19) - open/close 3061 (time 1789135777.01 total 50.06 last 54.65) - open/close 3566 (time 1789135787.02 total 60.07 last 50.44) - open/close 4072 (time 1789135797.04 total 70.09 last 50.49) total: 4500 open/close in 77.58 seconds: 58.00 ops/second - /unlink 1059 (time 1789135819.33 total 10.00 last 105.89) - /unlink 2125 (time 1789135829.33 total 20.01 last 106.55) - /unlink 3220 (time 1789135839.33 total 30.01 last 109.48) - /unlink 4337 (time 1789135849.33 total 40.01 last 111.69) total: 4500 /unlink in 41.76 seconds: 107.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 392 (time 1789135869.27 total 10.03 last 39.10) - open/close 857 (time 1789135879.28 total 20.03 last 46.47) - open/close 1363 (time 1789135889.28 total 30.03 last 50.60) - open/close 2032 (time 1789135899.29 total 40.05 last 66.83) - open/close 2711 (time 1789135909.29 total 50.05 last 67.90) - open/close 3313 (time 1789135919.30 total 60.06 last 60.13) - open/close 3910 (time 1789135929.31 total 70.06 last 59.67) - open/close 4488 (time 1789135939.31 total 80.07 last 57.77) total: 4500 open/close in 80.33 seconds: 56.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1193 (time 1789135955.18 total 10.01 last 119.24) - /unlink 2362 (time 1789135965.19 total 20.01 last 116.79) - /unlink 3613 (time 1789135975.20 total 30.02 last 125.09) total: 4500 /unlink in 38.61 seconds: 116.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 584 (time 1789136001.17 total 10.00 last 58.38) - open/close 1144 (time 1789136011.17 total 20.01 last 55.99) - open/close 1838 (time 1789136021.17 total 30.01 last 69.36) - open/close 2555 (time 1789136031.18 total 40.02 last 71.66) - open/close 3195 (time 1789136041.18 total 50.02 last 64.00) - open/close 3778 (time 1789136051.20 total 60.03 last 58.21) - open/close 4401 (time 1789136061.21 total 70.05 last 62.20) total: 4500 open/close in 71.53 seconds: 62.91 ops/second - /unlink 1269 (time 1789136077.97 total 10.00 last 126.86) - /unlink 2593 (time 1789136087.97 total 20.01 last 132.35) - /unlink 3797 (time 1789136097.97 total 30.01 last 120.39) total: 4500 /unlink in 37.77 seconds: 119.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 546 (time 1789136122.91 total 10.01 last 54.57) - open/close 1120 (time 1789136132.92 total 20.01 last 57.38) - open/close 1808 (time 1789136142.92 total 30.01 last 68.77) - open/close 2509 (time 1789136152.92 total 40.02 last 70.08) - open/close 3184 (time 1789136162.94 total 50.03 last 67.42) - open/close 3784 (time 1789136172.94 total 60.03 last 59.99) - open/close 4333 (time 1789136182.94 total 70.04 last 54.87) total: 4500 open/close in 72.52 seconds: 62.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1143 (time 1789136201.17 total 10.01 last 114.18) - /unlink 2297 (time 1789136211.18 total 20.02 last 115.32) - /unlink 3386 (time 1789136221.18 total 30.02 last 108.83) - /unlink 4414 (time 1789136231.19 total 40.03 last 102.75) total: 4500 /unlink in 40.87 seconds: 110.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 663 (time 1789136248.11 total 10.00 last 66.30) - open/close 1312 (time 1789136258.11 total 20.00 last 64.89) - open/close 1994 (time 1789136268.13 total 30.02 last 68.06) - open/close 2652 (time 1789136278.14 total 40.03 last 65.75) - open/close 3288 (time 1789136288.16 total 50.05 last 63.46) - open/close 3910 (time 1789136298.16 total 60.05 last 62.19) total: 4500 open/close in 69.39 seconds: 64.85 ops/second - /unlink 1227 (time 1789136322.78 total 10.00 last 122.64) - /unlink 2397 (time 1789136332.79 total 20.01 last 116.90) - /unlink 3616 (time 1789136342.80 total 30.02 last 121.80) total: 4500 /unlink in 38.30 seconds: 117.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 688 (time 1789136366.72 total 10.02 last 68.67) - open/close 1310 (time 1789136376.73 total 20.03 last 62.15) - open/close 1995 (time 1789136386.73 total 30.03 last 68.47) - open/close 2666 (time 1789136396.74 total 40.04 last 67.04) - open/close 3310 (time 1789136406.76 total 50.06 last 64.25) - open/close 3928 (time 1789136416.79 total 60.09 last 61.66) total: 4500 open/close in 69.65 seconds: 64.60 ops/second - /unlink 1178 (time 1789136441.28 total 10.01 last 117.69) - /unlink 2412 (time 1789136451.28 total 20.01 last 123.35) - /unlink 3602 (time 1789136461.29 total 30.02 last 118.98) total: 4500 /unlink in 37.78 seconds: 119.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 681 (time 1789136483.81 total 10.01 last 68.05) - open/close 1326 (time 1789136493.82 total 20.01 last 64.47) - open/close 1955 (time 1789136503.84 total 30.04 last 62.76) - open/close 2584 (time 1789136513.85 total 40.05 last 62.82) - open/close 3187 (time 1789136523.86 total 50.05 last 60.27) - open/close 3856 (time 1789136533.86 total 60.06 last 66.85) total: 4500 open/close in 70.00 seconds: 64.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1265 (time 1789136558.57 total 10.00 last 126.48) - /unlink 2549 (time 1789136568.58 total 20.01 last 128.34) - /unlink 3768 (time 1789136578.58 total 30.01 last 121.82) total: 4500 /unlink in 37.16 seconds: 121.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 704 (time 1789136600.69 total 10.01 last 70.32) - open/close 1387 (time 1789136610.70 total 20.01 last 68.28) - open/close 2042 (time 1789136620.71 total 30.02 last 65.43) - open/close 2643 (time 1789136630.71 total 40.03 last 60.06) - open/close 3213 (time 1789136640.72 total 50.04 last 56.93) - open/close 3763 (time 1789136650.74 total 60.06 last 54.91) - open/close 4462 (time 1789136660.75 total 70.06 last 69.87) total: 4500 open/close in 70.66 seconds: 63.68 ops/second - /unlink 1129 (time 1789136676.03 total 10.01 last 112.83) - /unlink 2404 (time 1789136686.03 total 20.01 last 127.42) - /unlink 3706 (time 1789136696.03 total 30.01 last 130.18) total: 4500 /unlink in 35.82 seconds: 125.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 687 (time 1789136716.22 total 10.01 last 68.65) - open/close 1324 (time 1789136726.25 total 20.03 last 63.54) - open/close 1965 (time 1789136736.25 total 30.04 last 64.08) - open/close 2672 (time 1789136746.26 total 40.05 last 70.61) - open/close 3363 (time 1789136756.26 total 50.05 last 69.10) - open/close 4000 (time 1789136766.27 total 60.05 last 63.68) total: 4500 open/close in 67.79 seconds: 66.38 ops/second - /unlink 1291 (time 1789136788.56 total 10.01 last 128.96) - /unlink 2476 (time 1789136798.56 total 20.01 last 118.46) - /unlink 3551 (time 1789136808.58 total 30.03 last 107.28) total: 4500 /unlink in 39.63 seconds: 113.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 700 (time 1789136833.23 total 10.02 last 69.89) - open/close 1376 (time 1789136843.24 total 20.03 last 67.53) - open/close 2009 (time 1789136853.24 total 30.03 last 63.30) - open/close 2672 (time 1789136863.24 total 40.04 last 66.25) - open/close 3298 (time 1789136873.25 total 50.04 last 62.58) - open/close 4033 (time 1789136883.26 total 60.05 last 73.41) total: 4500 open/close in 66.65 seconds: 67.52 ops/second - /unlink 1317 (time 1789136904.49 total 10.00 last 131.68) - /unlink 2613 (time 1789136914.50 total 20.01 last 129.48) - /unlink 3773 (time 1789136924.50 total 30.02 last 115.92) total: 4500 /unlink in 37.49 seconds: 120.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 540 (time 1789136949.64 total 10.01 last 53.97) - open/close 1200 (time 1789136959.65 total 20.02 last 65.93) - open/close 1875 (time 1789136969.67 total 30.03 last 67.39) - open/close 2508 (time 1789136979.67 total 40.03 last 63.28) - open/close 3140 (time 1789136989.68 total 50.04 last 63.18) - open/close 3756 (time 1789136999.68 total 60.04 last 61.60) - open/close 4395 (time 1789137009.70 total 70.06 last 63.74) total: 4500 open/close in 72.37 seconds: 62.18 ops/second - /unlink 1324 (time 1789137027.18 total 10.00 last 132.35) - /unlink 2465 (time 1789137037.18 total 20.01 last 114.07) - /unlink 3693 (time 1789137047.19 total 30.01 last 122.74) total: 4500 /unlink in 36.37 seconds: 123.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 683 (time 1789137069.08 total 10.01 last 68.25) - open/close 1329 (time 1789137079.09 total 20.01 last 64.57) - open/close 1928 (time 1789137089.10 total 30.03 last 59.81) - open/close 2524 (time 1789137099.11 total 40.04 last 59.54) - open/close 3194 (time 1789137109.12 total 50.04 last 66.97) - open/close 3884 (time 1789137119.12 total 60.05 last 68.94) total: 4500 open/close in 68.98 seconds: 65.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1259 (time 1789137142.83 total 10.00 last 125.86) - /unlink 2411 (time 1789137152.83 total 20.01 last 115.14) - /unlink 3589 (time 1789137162.83 total 30.01 last 117.79) total: 4500 /unlink in 37.48 seconds: 120.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 685 (time 1789137185.16 total 10.01 last 68.45) - open/close 1328 (time 1789137195.17 total 20.02 last 64.23) - open/close 1979 (time 1789137205.19 total 30.04 last 64.96) - open/close 2706 (time 1789137215.19 total 40.04 last 72.68) - open/close 3369 (time 1789137225.22 total 50.07 last 66.14) - open/close 4050 (time 1789137235.22 total 60.07 last 68.06) total: 4500 open/close in 66.89 seconds: 67.27 ops/second - /unlink 1220 (time 1789137257.09 total 10.01 last 121.88) - /unlink 2416 (time 1789137267.09 total 20.01 last 119.58) - /unlink 3603 (time 1789137277.10 total 30.02 last 118.66) total: 4500 /unlink in 37.61 seconds: 119.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 640 (time 1789137300.31 total 10.01 last 63.92) - open/close 1236 (time 1789137310.31 total 20.02 last 59.58) - open/close 1845 (time 1789137320.31 total 30.02 last 60.90) - open/close 2514 (time 1789137330.32 total 40.02 last 66.85) - open/close 3151 (time 1789137340.33 total 50.03 last 63.64) - open/close 3775 (time 1789137350.33 total 60.04 last 62.38) - open/close 4405 (time 1789137360.34 total 70.04 last 62.98) total: 4500 open/close in 71.93 seconds: 62.56 ops/second - /unlink 1024 (time 1789137378.01 total 10.01 last 102.34) - /unlink 2109 (time 1789137388.01 total 20.01 last 108.43) - /unlink 3286 (time 1789137398.02 total 30.02 last 117.60) - /unlink 4256 (time 1789137408.03 total 40.03 last 96.94) total: 4500 /unlink in 42.96 seconds: 104.75 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11449: 95330 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded.