== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:28:51 (1787833731) Stopping clients: oleg404-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg404-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg404-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg404-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg404-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg404-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg404-server oleg404-server: oleg404-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg404-server' oleg404-server: oleg404-server.virtnet: executing load_modules_local oleg404-server: Loading modules from /home/green/git/lustre-release/lustre oleg404-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg404-server: detected 4 online CPUs by sysfs oleg404-server: MODOPTS_LIBCFS= oleg404-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg404-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg404-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg404-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug -1 all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 652 (time 1787833945.96 total 10.00 last 65.19) - open/close 1313 (time 1787833955.96 total 20.00 last 66.09) - open/close 1912 (time 1787833965.96 total 30.01 last 59.87) - open/close 2618 (time 1787833975.97 total 40.01 last 70.55) - open/close 3312 (time 1787833985.97 total 50.02 last 69.39) - open/close 4039 (time 1787833995.99 total 60.03 last 72.56) total: 4500 open/close in 67.33 seconds: 66.84 ops/second - /unlink 1231 (time 1787834017.86 total 10.00 last 123.06) - /unlink 2339 (time 1787834027.88 total 20.02 last 110.62) - /unlink 3518 (time 1787834037.88 total 30.02 last 117.84) total: 4500 /unlink in 38.99 seconds: 115.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 523 (time 1787834062.14 total 10.01 last 52.27) - open/close 1093 (time 1787834072.15 total 20.01 last 56.99) - open/close 1694 (time 1787834082.16 total 30.02 last 60.02) - open/close 2345 (time 1787834092.17 total 40.03 last 65.05) - open/close 2983 (time 1787834102.18 total 50.04 last 63.72) - open/close 3653 (time 1787834112.20 total 60.06 last 66.90) - open/close 4281 (time 1787834122.20 total 70.07 last 62.74) total: 4500 open/close in 74.33 seconds: 60.54 ops/second - /unlink 951 (time 1787834143.51 total 10.01 last 95.03) - /unlink 1805 (time 1787834153.52 total 20.02 last 85.30) - /unlink 2698 (time 1787834163.52 total 30.02 last 89.27) - /unlink 3629 (time 1787834173.54 total 40.04 last 92.96) total: 4500 /unlink in 49.34 seconds: 91.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 336 (time 1787834201.19 total 10.04 last 33.48) - open/close 791 (time 1787834211.21 total 20.06 last 45.38) - open/close 1270 (time 1787834221.22 total 30.07 last 47.88) - open/close 1809 (time 1787834231.23 total 40.08 last 53.86) - open/close 2486 (time 1787834241.23 total 50.08 last 67.69) - open/close 3139 (time 1787834251.24 total 60.09 last 65.21) - open/close 3756 (time 1787834261.27 total 70.12 last 61.54) - open/close 4291 (time 1787834271.27 total 80.12 last 53.46) total: 4500 open/close in 84.26 seconds: 53.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 966 (time 1787834292.48 total 10.01 last 96.49) - /unlink 2082 (time 1787834302.49 total 20.02 last 111.56) - /unlink 2997 (time 1787834312.50 total 30.03 last 91.39) - /unlink 3916 (time 1787834322.51 total 40.04 last 91.78) total: 4500 /unlink in 45.91 seconds: 98.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 597 (time 1787834343.56 total 10.12 last 58.98) - open/close 1170 (time 1787834353.58 total 20.14 last 57.17) - open/close 1672 (time 1787834363.61 total 30.17 last 50.08) - open/close 2262 (time 1787834373.61 total 40.17 last 58.97) - open/close 2736 (time 1787834383.63 total 50.19 last 47.33) - open/close 3207 (time 1787834393.66 total 60.22 last 46.96) - open/close 3542 (time 1787834403.69 total 70.25 last 33.39) - open/close 3947 (time 1787834413.70 total 80.26 last 40.45) - open/close 4480 (time 1787834423.71 total 90.27 last 53.28) total: 4500 open/close in 90.58 seconds: 49.68 ops/second - /unlink 893 (time 1787834439.00 total 10.00 last 89.29) - /unlink 1549 (time 1787834449.01 total 20.00 last 65.59) - /unlink 2271 (time 1787834459.01 total 30.01 last 72.16) - /unlink 3045 (time 1787834469.02 total 40.01 last 77.36) - /unlink 4209 (time 1787834479.02 total 50.02 last 116.37) total: 4500 /unlink in 52.59 seconds: 85.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 586 (time 1787834497.66 total 10.01 last 58.56) - open/close 1180 (time 1787834507.67 total 20.02 last 59.31) - open/close 1707 (time 1787834517.68 total 30.04 last 52.64) - open/close 2330 (time 1787834527.70 total 40.05 last 62.20) - open/close 2939 (time 1787834537.70 total 50.06 last 60.87) - open/close 3561 (time 1787834547.71 total 60.06 last 62.16) - open/close 4192 (time 1787834557.71 total 70.06 last 63.09) total: 4500 open/close in 75.49 seconds: 59.61 ops/second - /unlink 1122 (time 1787834578.37 total 10.00 last 112.16) - /unlink 2097 (time 1787834588.37 total 20.01 last 97.47) - /unlink 3248 (time 1787834598.37 total 30.01 last 115.07) - /unlink 4375 (time 1787834608.38 total 40.01 last 112.67) total: 4500 /unlink in 41.08 seconds: 109.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 567 (time 1787834625.35 total 10.03 last 56.56) - open/close 1160 (time 1787834635.36 total 20.03 last 59.25) - open/close 1768 (time 1787834645.37 total 30.04 last 60.74) - open/close 2310 (time 1787834655.39 total 40.07 last 54.08) - open/close 2842 (time 1787834665.39 total 50.07 last 53.19) - open/close 3444 (time 1787834675.40 total 60.07 last 60.17) - open/close 4001 (time 1787834685.41 total 70.08 last 55.64) total: 4500 open/close in 79.54 seconds: 56.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1187 (time 1787834710.21 total 10.00 last 118.70) - /unlink 2278 (time 1787834720.22 total 20.01 last 108.97) - /unlink 3488 (time 1787834730.23 total 30.01 last 120.96) total: 4500 /unlink in 39.14 seconds: 114.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 664 (time 1787834754.71 total 10.01 last 66.34) - open/close 1292 (time 1787834764.73 total 20.03 last 62.67) - open/close 1824 (time 1787834774.76 total 30.05 last 53.09) - open/close 2447 (time 1787834784.76 total 40.06 last 62.27) - open/close 3035 (time 1787834794.77 total 50.07 last 58.74) - open/close 3638 (time 1787834804.77 total 60.07 last 60.28) - open/close 4216 (time 1787834814.78 total 70.08 last 57.76) total: 4500 open/close in 75.39 seconds: 59.69 ops/second - /unlink 1080 (time 1787834836.01 total 10.00 last 107.95) - /unlink 2099 (time 1787834846.01 total 20.01 last 101.85) - /unlink 3240 (time 1787834856.02 total 30.01 last 114.05) total: 4500 /unlink in 39.69 seconds: 113.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 652 (time 1787834881.11 total 10.01 last 65.14) - open/close 1269 (time 1787834891.11 total 20.02 last 61.66) - open/close 1900 (time 1787834901.12 total 30.02 last 63.08) - open/close 2532 (time 1787834911.12 total 40.02 last 63.16) - open/close 3107 (time 1787834921.14 total 50.05 last 57.37) - open/close 3624 (time 1787834931.15 total 60.05 last 51.69) - open/close 4208 (time 1787834941.16 total 70.07 last 58.30) total: 4500 open/close in 74.32 seconds: 60.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1237 (time 1787834960.34 total 10.01 last 123.63) - /unlink 2436 (time 1787834970.35 total 20.01 last 119.86) - /unlink 3658 (time 1787834980.35 total 30.01 last 122.18) total: 4500 /unlink in 36.87 seconds: 122.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 653 (time 1787835002.28 total 10.08 last 64.77) - open/close 1319 (time 1787835012.28 total 20.09 last 66.56) - open/close 1915 (time 1787835022.29 total 30.10 last 59.54) - open/close 2536 (time 1787835032.30 total 40.11 last 62.05) - open/close 3159 (time 1787835042.30 total 50.11 last 62.27) - open/close 3796 (time 1787835052.31 total 60.12 last 63.67) - open/close 4418 (time 1787835062.33 total 70.14 last 62.07) total: 4500 open/close in 71.20 seconds: 63.20 ops/second - /unlink 1090 (time 1787835078.54 total 10.00 last 108.95) - /unlink 2131 (time 1787835088.54 total 20.01 last 104.09) - /unlink 3506 (time 1787835098.55 total 30.02 last 137.36) total: 4500 /unlink in 37.73 seconds: 119.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 674 (time 1787835121.48 total 10.01 last 67.35) - open/close 1360 (time 1787835131.49 total 20.02 last 68.54) - open/close 2046 (time 1787835141.52 total 30.05 last 68.37) - open/close 2715 (time 1787835151.52 total 40.05 last 66.89) - open/close 3309 (time 1787835161.53 total 50.06 last 59.34) - open/close 3998 (time 1787835171.53 total 60.06 last 68.89) total: 4500 open/close in 67.75 seconds: 66.42 ops/second - /unlink 1049 (time 1787835194.41 total 10.02 last 104.71) - /unlink 1987 (time 1787835204.41 total 20.02 last 93.75) - /unlink 3292 (time 1787835214.42 total 30.02 last 130.49) total: 4500 /unlink in 39.26 seconds: 114.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 635 (time 1787835238.62 total 10.01 last 63.45) - open/close 1337 (time 1787835248.63 total 20.01 last 70.16) - open/close 2064 (time 1787835258.63 total 30.02 last 72.66) - open/close 2700 (time 1787835268.64 total 40.03 last 63.53) - open/close 3323 (time 1787835278.67 total 50.06 last 62.11) - open/close 4007 (time 1787835288.68 total 60.06 last 68.39) total: 4500 open/close in 66.25 seconds: 67.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1203 (time 1787835310.52 total 10.02 last 120.11) - /unlink 2241 (time 1787835320.52 total 20.02 last 103.80) - /unlink 3350 (time 1787835330.54 total 30.04 last 110.65) total: 4500 /unlink in 40.02 seconds: 112.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 670 (time 1787835355.71 total 10.00 last 66.99) - open/close 1308 (time 1787835365.72 total 20.01 last 63.72) - open/close 1784 (time 1787835375.73 total 30.02 last 47.57) - open/close 2412 (time 1787835385.74 total 40.03 last 62.74) - open/close 3014 (time 1787835395.74 total 50.03 last 60.19) - open/close 3711 (time 1787835405.77 total 60.06 last 69.49) - open/close 4378 (time 1787835415.78 total 70.07 last 66.65) total: 4500 open/close in 71.65 seconds: 62.81 ops/second - /unlink 1309 (time 1787835432.35 total 10.00 last 130.88) - /unlink 2505 (time 1787835442.35 total 20.00 last 119.57) - /unlink 3676 (time 1787835452.35 total 30.01 last 117.09) total: 4500 /unlink in 36.81 seconds: 122.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 651 (time 1787835473.46 total 10.01 last 65.07) - open/close 1316 (time 1787835483.47 total 20.01 last 66.44) - open/close 2038 (time 1787835493.50 total 30.04 last 72.00) - open/close 2710 (time 1787835503.52 total 40.06 last 67.07) - open/close 3367 (time 1787835513.54 total 50.08 last 65.57) - open/close 3995 (time 1787835523.56 total 60.10 last 62.66) total: 4500 open/close in 69.38 seconds: 64.86 ops/second - /unlink 1041 (time 1787835548.03 total 10.00 last 104.06) - /unlink 2135 (time 1787835558.04 total 20.01 last 109.31) - /unlink 3417 (time 1787835568.06 total 30.03 last 127.99) total: 4500 /unlink in 38.66 seconds: 116.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 594 (time 1787835592.00 total 10.05 last 59.13) - open/close 1172 (time 1787835602.01 total 20.06 last 57.73) - open/close 1751 (time 1787835612.02 total 30.07 last 57.83) - open/close 2385 (time 1787835622.03 total 40.08 last 63.30) - open/close 3066 (time 1787835632.04 total 50.09 last 68.05) - open/close 3776 (time 1787835642.04 total 60.09 last 70.98) - open/close 4459 (time 1787835652.05 total 70.10 last 68.29) total: 4500 open/close in 70.56 seconds: 63.77 ops/second - /unlink 1313 (time 1787835667.13 total 10.00 last 131.24) - /unlink 2484 (time 1787835677.14 total 20.01 last 117.01) - /unlink 3607 (time 1787835687.14 total 30.01 last 112.29) total: 4500 /unlink in 37.73 seconds: 119.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 708 (time 1787835709.62 total 10.01 last 70.71) - open/close 1410 (time 1787835719.63 total 20.02 last 70.12) - open/close 2111 (time 1787835729.64 total 30.03 last 70.05) - open/close 2788 (time 1787835739.65 total 40.04 last 67.63) - open/close 3465 (time 1787835749.66 total 50.05 last 67.62) - open/close 4111 (time 1787835759.66 total 60.05 last 64.60) total: 4500 open/close in 65.83 seconds: 68.36 ops/second - /unlink 1119 (time 1787835781.62 total 10.00 last 111.86) - /unlink 2104 (time 1787835791.63 total 20.01 last 98.41) - /unlink 3010 (time 1787835801.64 total 30.02 last 90.53) - /unlink 4205 (time 1787835811.64 total 40.02 last 119.50) total: 4500 /unlink in 42.54 seconds: 105.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 657 (time 1787835829.92 total 10.01 last 65.66) - open/close 1274 (time 1787835839.93 total 20.01 last 61.67) - open/close 1942 (time 1787835849.94 total 30.02 last 66.74) - open/close 2602 (time 1787835859.96 total 40.04 last 65.86) - open/close 3296 (time 1787835869.97 total 50.05 last 69.35) - open/close 3987 (time 1787835879.98 total 60.06 last 69.04) total: 4500 open/close in 67.17 seconds: 67.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1366 (time 1787835901.59 total 10.01 last 136.53) - /unlink 2615 (time 1787835911.59 total 20.01 last 124.90) - /unlink 3850 (time 1787835921.59 total 30.01 last 123.46) total: 4500 /unlink in 37.16 seconds: 121.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 566 (time 1787835944.07 total 10.00 last 56.58) - open/close 1165 (time 1787835954.08 total 20.01 last 59.84) - open/close 1673 (time 1787835964.12 total 30.05 last 50.63) - open/close 2273 (time 1787835974.13 total 40.06 last 59.94) - open/close 2788 (time 1787835984.13 total 50.07 last 51.46) - open/close 3408 (time 1787835994.15 total 60.08 last 61.90) - open/close 4102 (time 1787836004.16 total 70.09 last 69.37) total: 4500 open/close in 76.31 seconds: 58.97 ops/second - /unlink 1177 (time 1787836025.43 total 10.01 last 117.53) - /unlink 2273 (time 1787836035.44 total 20.02 last 109.50) - /unlink 3481 (time 1787836045.45 total 30.03 last 120.72) total: 4500 /unlink in 38.53 seconds: 116.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 695 (time 1787836068.50 total 10.00 last 69.48) - open/close 1393 (time 1787836078.52 total 20.02 last 69.65) - open/close 2095 (time 1787836088.52 total 30.03 last 70.17) - open/close 2669 (time 1787836098.53 total 40.03 last 57.38) - open/close 3355 (time 1787836108.53 total 50.03 last 68.59) - open/close 3929 (time 1787836118.56 total 60.07 last 57.20) total: 4500 open/close in 69.76 seconds: 64.50 ops/second - /unlink 1272 (time 1787836144.76 total 10.00 last 127.14) - /unlink 2471 (time 1787836154.76 total 20.01 last 119.88) - /unlink 3841 (time 1787836164.77 total 30.02 last 136.80) total: 4500 /unlink in 35.75 seconds: 125.87 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95430 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg404-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg404-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg404-server stop mds service on oleg404-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg404-server stop mds service on oleg404-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg404-server unloading modules via unload_modules_local on: 'oleg404-server' oleg404-server: oleg404-server.virtnet: executing unload_modules_local oleg404-server: modules unloaded.