== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:02:59 (1787832179) Stopping clients: oleg202-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg202-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg202-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg202-server oleg202-server: oleg202-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing load_modules_local oleg202-server: Loading modules from /home/green/git/lustre-release/lustre oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg202-server: detected 4 online CPUs by sysfs oleg202-server: MODOPTS_LIBCFS= oleg202-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg202-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg202-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg202-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 562 (time 1787832391.14 total 10.04 last 55.95) - open/close 1005 (time 1787832401.16 total 20.06 last 44.25) - open/close 1699 (time 1787832411.18 total 30.08 last 69.20) - open/close 2344 (time 1787832421.20 total 40.10 last 64.40) - open/close 3091 (time 1787832431.20 total 50.10 last 74.68) - open/close 3968 (time 1787832441.21 total 60.11 last 87.65) total: 4500 open/close in 66.92 seconds: 67.24 ops/second - /unlink 1484 (time 1787832462.67 total 10.00 last 148.38) - /unlink 2915 (time 1787832472.67 total 20.01 last 143.03) - /unlink 4378 (time 1787832482.69 total 30.02 last 146.05) total: 4500 /unlink in 30.99 seconds: 145.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 797 (time 1787832498.32 total 10.01 last 79.59) - open/close 1483 (time 1787832508.32 total 20.02 last 68.59) - open/close 2170 (time 1787832518.33 total 30.02 last 68.68) - open/close 2847 (time 1787832528.36 total 40.05 last 67.46) - open/close 3722 (time 1787832538.38 total 50.07 last 87.36) - open/close 4488 (time 1787832548.39 total 60.08 last 76.55) total: 4500 open/close in 60.25 seconds: 74.69 ops/second - /unlink 1659 (time 1787832562.82 total 10.02 last 165.64) - /unlink 2677 (time 1787832572.83 total 20.02 last 101.77) - /unlink 3736 (time 1787832582.83 total 30.02 last 105.86) total: 4500 /unlink in 39.02 seconds: 115.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 650 (time 1787832609.86 total 10.02 last 64.88) - open/close 1389 (time 1787832619.86 total 20.02 last 73.86) - open/close 2146 (time 1787832629.87 total 30.03 last 75.62) - open/close 3010 (time 1787832639.88 total 40.04 last 86.34) - open/close 3786 (time 1787832649.88 total 50.04 last 77.58) total: 4500 open/close in 59.46 seconds: 75.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1435 (time 1787832674.11 total 10.01 last 143.40) - /unlink 3034 (time 1787832684.12 total 20.01 last 159.82) total: 4500 /unlink in 29.85 seconds: 150.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 863 (time 1787832710.13 total 10.02 last 86.16) - open/close 1520 (time 1787832720.14 total 20.03 last 65.61) - open/close 2366 (time 1787832730.15 total 30.04 last 84.53) - open/close 3170 (time 1787832740.17 total 40.05 last 80.28) - open/close 3954 (time 1787832750.17 total 50.06 last 78.35) total: 4500 open/close in 57.48 seconds: 78.29 ops/second - /unlink 1600 (time 1787832772.98 total 10.01 last 159.87) - /unlink 3088 (time 1787832782.98 total 20.01 last 148.79) total: 4500 /unlink in 29.21 seconds: 154.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 780 (time 1787832807.42 total 10.02 last 77.88) - open/close 1503 (time 1787832817.43 total 20.03 last 72.19) - open/close 2126 (time 1787832827.44 total 30.04 last 62.25) - open/close 2763 (time 1787832837.46 total 40.06 last 63.55) - open/close 3614 (time 1787832847.48 total 50.08 last 84.96) - open/close 4496 (time 1787832857.49 total 60.09 last 88.13) total: 4500 open/close in 60.12 seconds: 74.86 ops/second - /unlink 1264 (time 1787832871.38 total 10.00 last 126.36) - /unlink 2290 (time 1787832881.39 total 20.01 last 102.51) - /unlink 3527 (time 1787832891.39 total 30.01 last 123.68) total: 4500 /unlink in 36.85 seconds: 122.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 729 (time 1787832913.53 total 10.01 last 72.83) - open/close 1579 (time 1787832923.57 total 20.04 last 84.72) - open/close 2153 (time 1787832933.60 total 30.07 last 57.23) - open/close 2702 (time 1787832943.60 total 40.07 last 54.89) - open/close 3364 (time 1787832953.64 total 50.11 last 65.93) - open/close 4085 (time 1787832963.64 total 60.12 last 72.06) total: 4500 open/close in 65.07 seconds: 69.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1337 (time 1787832983.87 total 10.00 last 133.64) - /unlink 3028 (time 1787832993.88 total 20.01 last 169.03) total: 4500 /unlink in 29.35 seconds: 153.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 903 (time 1787833018.49 total 10.01 last 90.20) - open/close 1676 (time 1787833028.49 total 20.01 last 77.29) - open/close 2405 (time 1787833038.50 total 30.02 last 72.86) - open/close 3084 (time 1787833048.50 total 40.02 last 67.89) - open/close 3852 (time 1787833058.50 total 50.02 last 76.78) total: 4500 open/close in 58.85 seconds: 76.47 ops/second - /unlink 1446 (time 1787833082.45 total 10.00 last 144.54) - /unlink 2804 (time 1787833092.45 total 20.01 last 135.74) - /unlink 4063 (time 1787833102.45 total 30.01 last 125.89) total: 4500 /unlink in 32.86 seconds: 136.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 756 (time 1787833121.72 total 10.01 last 75.55) - open/close 1434 (time 1787833131.74 total 20.03 last 67.66) - open/close 2134 (time 1787833141.75 total 30.04 last 69.92) - open/close 2956 (time 1787833151.76 total 40.04 last 82.16) - open/close 3768 (time 1787833161.77 total 50.06 last 81.09) - open/close 4408 (time 1787833171.78 total 60.06 last 63.95) total: 4500 open/close in 61.72 seconds: 72.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1521 (time 1787833189.53 total 10.00 last 152.04) - /unlink 2829 (time 1787833199.53 total 20.00 last 130.79) - /unlink 4042 (time 1787833209.53 total 30.01 last 121.29) total: 4500 /unlink in 33.64 seconds: 133.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 653 (time 1787833229.42 total 10.01 last 65.22) - open/close 1429 (time 1787833239.42 total 20.01 last 77.59) - open/close 2168 (time 1787833249.44 total 30.03 last 73.75) - open/close 2774 (time 1787833259.45 total 40.04 last 60.58) - open/close 3543 (time 1787833269.46 total 50.05 last 76.83) - open/close 4353 (time 1787833279.46 total 60.05 last 80.98) total: 4500 open/close in 62.54 seconds: 71.96 ops/second - /unlink 1399 (time 1787833296.66 total 10.01 last 139.77) - /unlink 3215 (time 1787833306.66 total 20.02 last 181.48) total: 4500 /unlink in 29.12 seconds: 154.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 475 (time 1787833331.13 total 10.02 last 47.39) - open/close 1365 (time 1787833341.13 total 20.03 last 88.97) - open/close 2206 (time 1787833351.14 total 30.04 last 84.01) - open/close 2922 (time 1787833361.15 total 40.04 last 71.56) - open/close 3686 (time 1787833371.15 total 50.04 last 76.39) - open/close 4416 (time 1787833381.16 total 60.05 last 72.94) total: 4500 open/close in 61.81 seconds: 72.81 ops/second - /unlink 1388 (time 1787833398.21 total 10.00 last 138.74) - /unlink 2954 (time 1787833408.21 total 20.01 last 156.58) total: 4500 /unlink in 28.74 seconds: 156.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 794 (time 1787833431.94 total 10.01 last 79.34) - open/close 1733 (time 1787833441.94 total 20.01 last 93.85) - open/close 2550 (time 1787833451.95 total 30.02 last 81.63) - open/close 3351 (time 1787833461.95 total 40.02 last 80.08) - open/close 4161 (time 1787833471.95 total 50.02 last 80.99) total: 4500 open/close in 54.32 seconds: 82.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1566 (time 1787833490.62 total 10.01 last 156.44) - /unlink 3030 (time 1787833500.62 total 20.01 last 146.39) total: 4500 /unlink in 28.32 seconds: 158.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 786 (time 1787833524.30 total 10.01 last 78.52) - open/close 1471 (time 1787833534.31 total 20.02 last 68.44) - open/close 2106 (time 1787833544.31 total 30.02 last 63.46) - open/close 2834 (time 1787833554.32 total 40.04 last 72.72) - open/close 3494 (time 1787833564.33 total 50.04 last 65.98) - open/close 4287 (time 1787833574.33 total 60.04 last 79.26) total: 4500 open/close in 62.46 seconds: 72.05 ops/second - /unlink 1610 (time 1787833591.90 total 10.02 last 160.74) - /unlink 3245 (time 1787833601.91 total 20.02 last 163.48) total: 4500 /unlink in 27.21 seconds: 165.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 833 (time 1787833623.87 total 10.01 last 83.26) - open/close 1689 (time 1787833633.87 total 20.01 last 85.59) - open/close 2597 (time 1787833643.87 total 30.01 last 90.75) - open/close 3404 (time 1787833653.88 total 40.02 last 80.66) - open/close 4115 (time 1787833663.89 total 50.03 last 71.02) total: 4500 open/close in 55.21 seconds: 81.50 ops/second - /unlink 1589 (time 1787833684.73 total 10.00 last 158.87) - /unlink 3085 (time 1787833694.73 total 20.00 last 149.58) total: 4500 /unlink in 28.50 seconds: 157.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 922 (time 1787833718.31 total 10.00 last 92.20) - open/close 1708 (time 1787833728.32 total 20.01 last 78.53) - open/close 2493 (time 1787833738.32 total 30.01 last 78.48) - open/close 3418 (time 1787833748.33 total 40.02 last 92.46) - open/close 4300 (time 1787833758.33 total 50.02 last 88.19) total: 4500 open/close in 52.29 seconds: 86.06 ops/second - /unlink 1709 (time 1787833774.10 total 10.00 last 170.81) - /unlink 3352 (time 1787833784.10 total 20.01 last 164.22) total: 4500 /unlink in 27.15 seconds: 165.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 904 (time 1787833806.32 total 10.01 last 90.35) - open/close 1793 (time 1787833816.34 total 20.02 last 88.75) - open/close 2630 (time 1787833826.37 total 30.05 last 83.47) - open/close 3390 (time 1787833836.38 total 40.06 last 75.94) - open/close 4326 (time 1787833846.39 total 50.07 last 93.48) total: 4500 open/close in 51.96 seconds: 86.60 ops/second - /unlink 1812 (time 1787833862.80 total 10.00 last 181.15) - /unlink 3500 (time 1787833872.81 total 20.01 last 168.72) total: 4500 /unlink in 26.75 seconds: 168.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 832 (time 1787833893.86 total 10.01 last 83.14) - open/close 1669 (time 1787833903.86 total 20.01 last 83.68) - open/close 2578 (time 1787833913.87 total 30.02 last 90.78) - open/close 3408 (time 1787833923.88 total 40.03 last 82.96) - open/close 4346 (time 1787833933.88 total 50.03 last 93.75) total: 4500 open/close in 51.80 seconds: 86.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1323 (time 1787833949.75 total 10.00 last 132.28) - /unlink 2930 (time 1787833959.76 total 20.01 last 160.64) - /unlink 4471 (time 1787833969.76 total 30.01 last 154.08) total: 4500 /unlink in 30.22 seconds: 148.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 817 (time 1787833985.08 total 10.00 last 81.68) - open/close 1628 (time 1787833995.09 total 20.01 last 81.03) - open/close 2503 (time 1787834005.10 total 30.02 last 87.44) - open/close 3407 (time 1787834015.10 total 40.02 last 90.35) - open/close 4392 (time 1787834025.11 total 50.03 last 98.41) total: 4500 open/close in 51.22 seconds: 87.85 ops/second - /unlink 1768 (time 1787834040.75 total 10.00 last 176.80) - /unlink 3408 (time 1787834050.75 total 20.00 last 163.95) total: 4500 /unlink in 26.76 seconds: 168.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 920 (time 1787834071.84 total 10.01 last 91.94) - open/close 1808 (time 1787834081.85 total 20.01 last 88.77) - open/close 2612 (time 1787834091.85 total 30.01 last 80.40) - open/close 3451 (time 1787834101.85 total 40.01 last 83.87) - open/close 4324 (time 1787834111.86 total 50.03 last 87.20) total: 4500 open/close in 52.28 seconds: 86.08 ops/second - /unlink 1593 (time 1787834128.29 total 10.01 last 159.21) - /unlink 3068 (time 1787834138.29 total 20.01 last 147.50) - /unlink 4475 (time 1787834148.29 total 30.01 last 140.69) total: 4500 /unlink in 30.23 seconds: 148.85 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95314 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg202-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server unloading modules via unload_modules_local on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing unload_modules_local oleg202-server: modules unloaded.