== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:33:51 (1761240831) Stopping clients: oleg613-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg613-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg613-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg613-server oleg613-server: oleg613-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing load_modules_local oleg613-server: Loading modules from /home/green/git/lustre-release/lustre oleg613-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg613-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg613-server: detected 4 online CPUs by sysfs oleg613-server: MODOPTS_LIBCFS= oleg613-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg613-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg613-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg613-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg613-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug -1 all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 700 (time 1761241017.50 total 10.00 last 69.98) - open/close 1433 (time 1761241027.51 total 20.01 last 73.24) - open/close 2121 (time 1761241037.52 total 30.03 last 68.70) - open/close 2835 (time 1761241047.54 total 40.04 last 71.31) - open/close 3512 (time 1761241057.55 total 50.05 last 67.63) - open/close 4272 (time 1761241067.55 total 60.05 last 75.99) total: 4500 open/close in 63.15 seconds: 71.25 ops/second - /unlink 1339 (time 1761241084.38 total 10.00 last 133.90) - /unlink 2674 (time 1761241094.39 total 20.01 last 133.40) - /unlink 3970 (time 1761241104.40 total 30.02 last 129.44) total: 4500 /unlink in 33.95 seconds: 132.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 651 (time 1761241122.73 total 10.02 last 64.96) - open/close 1369 (time 1761241132.74 total 20.03 last 71.74) - open/close 2149 (time 1761241142.75 total 30.03 last 77.97) - open/close 2853 (time 1761241152.75 total 40.04 last 70.38) - open/close 3540 (time 1761241162.75 total 50.04 last 68.68) - open/close 4271 (time 1761241172.75 total 60.04 last 73.09) total: 4500 open/close in 62.75 seconds: 71.71 ops/second - /unlink 1586 (time 1761241188.28 total 10.01 last 158.48) - /unlink 3070 (time 1761241198.29 total 20.01 last 148.37) total: 4500 /unlink in 29.95 seconds: 150.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 861 (time 1761241221.41 total 10.00 last 86.07) - open/close 1641 (time 1761241231.41 total 20.01 last 77.94) - open/close 2427 (time 1761241241.43 total 30.03 last 78.49) - open/close 3213 (time 1761241251.44 total 40.04 last 78.52) - open/close 3990 (time 1761241261.44 total 50.04 last 77.69) total: 4500 open/close in 56.73 seconds: 79.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1425 (time 1761241281.49 total 10.00 last 142.48) - /unlink 2856 (time 1761241291.50 total 20.01 last 143.02) - /unlink 4274 (time 1761241301.50 total 30.01 last 141.74) total: 4500 /unlink in 31.77 seconds: 141.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 704 (time 1761241316.81 total 10.01 last 70.34) - open/close 1442 (time 1761241326.81 total 20.01 last 73.78) - open/close 2158 (time 1761241336.82 total 30.02 last 71.51) - open/close 2927 (time 1761241346.84 total 40.04 last 76.79) - open/close 3665 (time 1761241356.84 total 50.04 last 73.78) - open/close 4412 (time 1761241366.85 total 60.05 last 74.63) total: 4500 open/close in 61.10 seconds: 73.65 ops/second - /unlink 1487 (time 1761241381.02 total 10.00 last 148.65) - /unlink 2915 (time 1761241391.03 total 20.01 last 142.65) total: 4500 /unlink in 29.97 seconds: 150.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 806 (time 1761241414.51 total 10.00 last 80.59) - open/close 1607 (time 1761241424.51 total 20.00 last 80.08) - open/close 2448 (time 1761241434.52 total 30.02 last 84.00) - open/close 3275 (time 1761241444.54 total 40.03 last 82.60) - open/close 4066 (time 1761241454.55 total 50.04 last 79.03) total: 4500 open/close in 55.58 seconds: 80.96 ops/second - /unlink 1423 (time 1761241473.17 total 10.01 last 142.18) - /unlink 2749 (time 1761241483.17 total 20.01 last 132.58) - /unlink 4119 (time 1761241493.18 total 30.02 last 136.82) total: 4500 /unlink in 32.80 seconds: 137.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 757 (time 1761241509.66 total 10.01 last 75.62) - open/close 1568 (time 1761241519.67 total 20.01 last 81.07) - open/close 2379 (time 1761241529.68 total 30.02 last 81.03) - open/close 3328 (time 1761241539.68 total 40.02 last 94.88) - open/close 4187 (time 1761241549.69 total 50.04 last 85.80) total: 4500 open/close in 54.08 seconds: 83.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1422 (time 1761241567.39 total 10.00 last 142.14) - /unlink 2775 (time 1761241577.39 total 20.00 last 135.29) - /unlink 3974 (time 1761241587.39 total 30.01 last 119.88) total: 4500 /unlink in 34.07 seconds: 132.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 723 (time 1761241606.11 total 10.00 last 72.28) - open/close 1518 (time 1761241616.11 total 20.01 last 79.47) - open/close 2308 (time 1761241626.12 total 30.01 last 78.96) - open/close 3087 (time 1761241636.13 total 40.02 last 77.84) - open/close 3780 (time 1761241646.13 total 50.02 last 69.28) - open/close 4479 (time 1761241656.13 total 60.03 last 69.88) total: 4500 open/close in 60.31 seconds: 74.61 ops/second - /unlink 1377 (time 1761241669.74 total 10.00 last 137.65) - /unlink 2797 (time 1761241679.75 total 20.01 last 141.95) - /unlink 4297 (time 1761241689.75 total 30.01 last 149.93) total: 4500 /unlink in 31.54 seconds: 142.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 741 (time 1761241704.60 total 10.00 last 74.07) - open/close 1447 (time 1761241714.61 total 20.01 last 70.52) - open/close 2203 (time 1761241724.61 total 30.02 last 75.55) - open/close 2987 (time 1761241734.62 total 40.03 last 78.34) - open/close 3727 (time 1761241744.63 total 50.04 last 73.94) - open/close 4423 (time 1761241754.64 total 60.05 last 69.51) total: 4500 open/close in 61.11 seconds: 73.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1313 (time 1761241769.64 total 10.01 last 131.17) - /unlink 2715 (time 1761241779.64 total 20.01 last 140.20) - /unlink 4012 (time 1761241789.64 total 30.01 last 129.64) total: 4500 /unlink in 33.76 seconds: 133.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 758 (time 1761241807.33 total 10.01 last 75.72) - open/close 1542 (time 1761241817.33 total 20.01 last 78.37) - open/close 2341 (time 1761241827.33 total 30.01 last 79.90) - open/close 3144 (time 1761241837.34 total 40.02 last 80.26) - open/close 3884 (time 1761241847.35 total 50.03 last 73.94) total: 4500 open/close in 58.25 seconds: 77.25 ops/second - /unlink 1379 (time 1761241870.12 total 10.00 last 137.87) - /unlink 2705 (time 1761241880.13 total 20.01 last 132.53) - /unlink 3972 (time 1761241890.13 total 30.01 last 126.65) total: 4500 /unlink in 33.89 seconds: 132.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 733 (time 1761241908.39 total 10.00 last 73.27) - open/close 1396 (time 1761241918.40 total 20.01 last 66.29) - open/close 2120 (time 1761241928.40 total 30.01 last 72.35) - open/close 2750 (time 1761241938.42 total 40.03 last 62.88) - open/close 3431 (time 1761241948.43 total 50.04 last 68.05) - open/close 3966 (time 1761241958.44 total 60.05 last 53.46) total: 4500 open/close in 68.50 seconds: 65.69 ops/second - /unlink 1213 (time 1761241981.65 total 10.01 last 121.22) - /unlink 2489 (time 1761241991.66 total 20.01 last 127.52) - /unlink 3774 (time 1761242001.67 total 30.02 last 128.43) total: 4500 /unlink in 35.55 seconds: 126.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 715 (time 1761242021.75 total 10.00 last 71.47) - open/close 1430 (time 1761242031.76 total 20.02 last 71.40) - open/close 2153 (time 1761242041.78 total 30.03 last 72.21) - open/close 2876 (time 1761242051.78 total 40.03 last 72.29) - open/close 3696 (time 1761242061.79 total 50.04 last 81.91) - open/close 4436 (time 1761242071.80 total 60.05 last 73.95) total: 4500 open/close in 61.12 seconds: 73.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1193 (time 1761242087.39 total 10.00 last 119.28) - /unlink 2283 (time 1761242097.39 total 20.00 last 109.00) - /unlink 3576 (time 1761242107.40 total 30.01 last 129.19) total: 4500 /unlink in 37.09 seconds: 121.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 774 (time 1761242129.12 total 10.01 last 77.35) - open/close 1423 (time 1761242139.12 total 20.01 last 64.88) - open/close 2186 (time 1761242149.13 total 30.01 last 76.29) - open/close 2931 (time 1761242159.13 total 40.02 last 74.43) - open/close 3736 (time 1761242169.14 total 50.03 last 80.43) total: 4500 open/close in 59.58 seconds: 75.52 ops/second - /unlink 1500 (time 1761242192.09 total 10.00 last 149.94) - /unlink 2817 (time 1761242202.10 total 20.01 last 131.65) - /unlink 4143 (time 1761242212.11 total 30.02 last 132.41) total: 4500 /unlink in 32.67 seconds: 137.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 710 (time 1761242229.71 total 10.01 last 70.94) - open/close 1471 (time 1761242239.72 total 20.02 last 76.00) - open/close 2229 (time 1761242249.72 total 30.02 last 75.78) - open/close 2951 (time 1761242259.73 total 40.03 last 72.14) - open/close 3660 (time 1761242269.74 total 50.04 last 70.85) - open/close 4385 (time 1761242279.74 total 60.05 last 72.46) total: 4500 open/close in 61.47 seconds: 73.21 ops/second - /unlink 1207 (time 1761242295.65 total 10.00 last 120.65) - /unlink 2381 (time 1761242305.66 total 20.01 last 117.35) - /unlink 3702 (time 1761242315.66 total 30.01 last 132.06) total: 4500 /unlink in 36.71 seconds: 122.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 692 (time 1761242336.55 total 10.01 last 69.16) - open/close 1431 (time 1761242346.56 total 20.02 last 73.79) - open/close 2134 (time 1761242356.58 total 30.04 last 70.18) - open/close 2832 (time 1761242366.58 total 40.04 last 69.79) - open/close 3441 (time 1761242376.59 total 50.05 last 60.84) - open/close 4127 (time 1761242386.59 total 60.05 last 68.59) total: 4500 open/close in 65.11 seconds: 69.11 ops/second - /unlink 1376 (time 1761242405.34 total 10.01 last 137.42) - /unlink 2594 (time 1761242415.35 total 20.02 last 121.69) - /unlink 3927 (time 1761242425.35 total 30.02 last 133.29) total: 4500 /unlink in 34.37 seconds: 130.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 635 (time 1761242444.93 total 10.01 last 63.42) - open/close 1213 (time 1761242454.93 total 20.01 last 57.78) - open/close 1864 (time 1761242464.93 total 30.02 last 65.07) - open/close 2625 (time 1761242474.94 total 40.03 last 76.04) - open/close 3274 (time 1761242484.94 total 50.03 last 64.90) - open/close 3978 (time 1761242494.96 total 60.04 last 70.30) total: 4500 open/close in 67.71 seconds: 66.46 ops/second - /unlink 1423 (time 1761242517.07 total 10.00 last 142.23) - /unlink 2787 (time 1761242527.08 total 20.01 last 136.34) - /unlink 4328 (time 1761242537.08 total 30.01 last 154.09) total: 4500 /unlink in 31.27 seconds: 143.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 752 (time 1761242552.09 total 10.00 last 75.16) - open/close 1546 (time 1761242562.10 total 20.01 last 79.33) - open/close 2304 (time 1761242572.10 total 30.02 last 75.78) - open/close 3058 (time 1761242582.11 total 40.02 last 75.34) - open/close 3820 (time 1761242592.13 total 50.04 last 76.04) - open/close 4385 (time 1761242602.13 total 60.05 last 56.50) total: 4500 open/close in 62.43 seconds: 72.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1201 (time 1761242620.06 total 10.00 last 120.08) - /unlink 2457 (time 1761242630.07 total 20.00 last 125.57) - /unlink 3755 (time 1761242640.07 total 30.01 last 129.76) total: 4500 /unlink in 35.43 seconds: 127.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 852 (time 1761242658.39 total 10.01 last 85.11) - open/close 1526 (time 1761242668.40 total 20.02 last 67.35) - open/close 2218 (time 1761242678.40 total 30.03 last 69.15) - open/close 3002 (time 1761242688.42 total 40.04 last 78.25) - open/close 3800 (time 1761242698.43 total 50.05 last 79.72) total: 4500 open/close in 59.36 seconds: 75.81 ops/second - /unlink 1421 (time 1761242721.10 total 10.00 last 142.10) - /unlink 2835 (time 1761242731.11 total 20.00 last 141.37) - /unlink 4348 (time 1761242741.11 total 30.01 last 151.17) total: 4500 /unlink in 31.11 seconds: 144.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 748 (time 1761242755.77 total 10.02 last 74.65) - open/close 1471 (time 1761242765.78 total 20.02 last 72.28) - open/close 2271 (time 1761242775.79 total 30.03 last 79.91) - open/close 2933 (time 1761242785.80 total 40.05 last 66.10) - open/close 3711 (time 1761242795.81 total 50.05 last 77.77) - open/close 4437 (time 1761242805.82 total 60.07 last 72.46) total: 4500 open/close in 61.08 seconds: 73.67 ops/second - /unlink 1306 (time 1761242821.05 total 10.01 last 130.52) - /unlink 2697 (time 1761242831.05 total 20.01 last 139.07) - /unlink 4082 (time 1761242841.05 total 30.01 last 138.47) total: 4500 /unlink in 32.91 seconds: 136.75 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11102: 93803 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg613-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg613-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg613-server stop mds service on oleg613-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg613-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg613-server' oleg613-server: oleg613-server.virtnet: executing unload_modules_local oleg613-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg613-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.