== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:09:59 (1787141399) Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg242-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg242-server oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg242-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 853 (time 1787141613.00 total 10.00 last 85.26) - open/close 1620 (time 1787141623.01 total 20.01 last 76.62) - open/close 2627 (time 1787141633.01 total 30.02 last 100.61) - open/close 3406 (time 1787141643.02 total 40.03 last 77.87) - open/close 4229 (time 1787141653.02 total 50.03 last 82.27) total: 4500 open/close in 53.00 seconds: 84.91 ops/second - /unlink 1490 (time 1787141670.25 total 10.00 last 149.00) - /unlink 2918 (time 1787141680.25 total 20.00 last 142.74) - /unlink 4404 (time 1787141690.26 total 30.01 last 148.53) total: 4500 /unlink in 30.93 seconds: 145.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 731 (time 1787141706.47 total 10.00 last 73.09) - open/close 1564 (time 1787141716.48 total 20.01 last 83.24) - open/close 2405 (time 1787141726.49 total 30.01 last 84.04) - open/close 3270 (time 1787141736.49 total 40.02 last 86.43) - open/close 4104 (time 1787141746.51 total 50.03 last 83.31) total: 4500 open/close in 54.54 seconds: 82.51 ops/second - /unlink 1582 (time 1787141764.79 total 10.00 last 158.20) - /unlink 3102 (time 1787141774.79 total 20.00 last 151.95) total: 4500 /unlink in 29.09 seconds: 154.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 999 (time 1787141798.12 total 10.00 last 99.87) - open/close 1765 (time 1787141808.12 total 20.01 last 76.58) - open/close 2476 (time 1787141818.13 total 30.01 last 71.06) - open/close 2988 (time 1787141828.14 total 40.02 last 51.14) - open/close 3853 (time 1787141838.14 total 50.02 last 86.49) total: 4500 open/close in 56.68 seconds: 79.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1829 (time 1787141859.78 total 10.02 last 182.58) - /unlink 3366 (time 1787141869.78 total 20.03 last 153.56) total: 4500 /unlink in 29.14 seconds: 154.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 818 (time 1787141894.10 total 10.01 last 81.72) - open/close 1407 (time 1787141904.10 total 20.01 last 58.89) - open/close 2149 (time 1787141914.11 total 30.02 last 74.14) - open/close 2864 (time 1787141924.13 total 40.04 last 71.40) - open/close 3601 (time 1787141934.16 total 50.06 last 73.48) - open/close 4368 (time 1787141944.17 total 60.08 last 76.57) total: 4500 open/close in 61.80 seconds: 72.82 ops/second - /unlink 1533 (time 1787141959.96 total 10.00 last 153.27) - /unlink 2904 (time 1787141969.97 total 20.01 last 137.05) - /unlink 4211 (time 1787141979.97 total 30.01 last 130.66) total: 4500 /unlink in 32.39 seconds: 138.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 796 (time 1787141997.72 total 10.02 last 79.47) - open/close 1600 (time 1787142007.75 total 20.04 last 80.20) - open/close 2581 (time 1787142017.75 total 30.05 last 98.05) - open/close 3368 (time 1787142027.76 total 40.05 last 78.67) - open/close 4219 (time 1787142037.76 total 50.06 last 85.02) total: 4500 open/close in 53.48 seconds: 84.14 ops/second - /unlink 1669 (time 1787142056.03 total 10.00 last 166.85) - /unlink 3140 (time 1787142066.04 total 20.02 last 146.92) total: 4500 /unlink in 29.76 seconds: 151.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 659 (time 1787142091.04 total 10.01 last 65.83) - open/close 1416 (time 1787142101.04 total 20.02 last 75.66) - open/close 2212 (time 1787142111.05 total 30.02 last 79.56) - open/close 2946 (time 1787142121.05 total 40.02 last 73.40) - open/close 3680 (time 1787142131.07 total 50.05 last 73.23) - open/close 4368 (time 1787142141.08 total 60.06 last 68.73) total: 4500 open/close in 61.75 seconds: 72.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1384 (time 1787142158.14 total 10.00 last 138.35) - /unlink 2693 (time 1787142168.16 total 20.02 last 130.72) - /unlink 3966 (time 1787142178.16 total 30.02 last 127.30) total: 4500 /unlink in 33.78 seconds: 133.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 689 (time 1787142197.33 total 10.02 last 68.75) - open/close 1350 (time 1787142207.35 total 20.04 last 65.99) - open/close 2120 (time 1787142217.36 total 30.05 last 76.94) - open/close 2933 (time 1787142227.37 total 40.06 last 81.19) - open/close 3606 (time 1787142237.37 total 50.06 last 67.29) - open/close 4385 (time 1787142247.39 total 60.08 last 77.79) total: 4500 open/close in 61.91 seconds: 72.69 ops/second - /unlink 1339 (time 1787142264.83 total 10.01 last 133.74) - /unlink 2540 (time 1787142274.84 total 20.02 last 120.06) - /unlink 3951 (time 1787142284.84 total 30.02 last 141.04) total: 4500 /unlink in 34.08 seconds: 132.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 676 (time 1787142305.06 total 10.01 last 67.51) - open/close 1415 (time 1787142315.07 total 20.02 last 73.83) - open/close 2079 (time 1787142325.08 total 30.03 last 66.36) - open/close 2901 (time 1787142335.08 total 40.03 last 82.18) - open/close 3562 (time 1787142345.09 total 50.05 last 66.01) - open/close 4387 (time 1787142355.10 total 60.05 last 82.47) total: 4500 open/close in 61.45 seconds: 73.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1585 (time 1787142371.18 total 10.00 last 158.45) - /unlink 3260 (time 1787142381.19 total 20.01 last 167.41) total: 4500 /unlink in 28.20 seconds: 159.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 938 (time 1787142404.10 total 10.01 last 93.73) - open/close 1762 (time 1787142414.10 total 20.01 last 82.37) - open/close 2556 (time 1787142424.11 total 30.02 last 79.36) - open/close 3355 (time 1787142434.11 total 40.02 last 79.88) - open/close 3937 (time 1787142444.13 total 50.04 last 58.09) total: 4500 open/close in 57.55 seconds: 78.20 ops/second - /unlink 1417 (time 1787142467.76 total 10.01 last 141.61) - /unlink 3022 (time 1787142477.76 total 20.01 last 160.45) - /unlink 4389 (time 1787142487.77 total 30.01 last 136.65) total: 4500 /unlink in 31.28 seconds: 143.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 945 (time 1787142504.79 total 10.02 last 94.28) - open/close 1795 (time 1787142514.81 total 20.04 last 84.83) - open/close 2631 (time 1787142524.82 total 30.05 last 83.54) - open/close 3293 (time 1787142534.82 total 40.05 last 66.17) - open/close 4163 (time 1787142544.82 total 50.05 last 87.00) total: 4500 open/close in 53.89 seconds: 83.50 ops/second - /unlink 1079 (time 1787142563.39 total 10.00 last 107.87) - /unlink 2401 (time 1787142573.40 total 20.01 last 132.15) - /unlink 3629 (time 1787142583.40 total 30.01 last 122.75) total: 4500 /unlink in 37.51 seconds: 119.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 720 (time 1787142606.20 total 10.01 last 71.94) - open/close 1467 (time 1787142616.20 total 20.02 last 74.64) - open/close 2109 (time 1787142626.22 total 30.03 last 64.12) - open/close 2707 (time 1787142636.22 total 40.04 last 59.76) - open/close 3432 (time 1787142646.24 total 50.05 last 72.41) - open/close 4225 (time 1787142656.26 total 60.07 last 79.12) total: 4500 open/close in 63.49 seconds: 70.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1250 (time 1787142674.19 total 10.02 last 124.75) - /unlink 2495 (time 1787142684.19 total 20.02 last 124.46) - /unlink 3983 (time 1787142694.20 total 30.03 last 148.68) total: 4500 /unlink in 33.24 seconds: 135.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 787 (time 1787142712.70 total 10.02 last 78.51) - open/close 1648 (time 1787142722.70 total 20.03 last 86.09) - open/close 2486 (time 1787142732.70 total 30.03 last 83.75) - open/close 3290 (time 1787142742.72 total 40.05 last 80.27) - open/close 4076 (time 1787142752.73 total 50.05 last 78.56) total: 4500 open/close in 55.21 seconds: 81.50 ops/second - /unlink 1369 (time 1787142772.72 total 10.01 last 136.81) - /unlink 2861 (time 1787142782.72 total 20.01 last 149.19) - /unlink 4354 (time 1787142792.72 total 30.01 last 149.20) total: 4500 /unlink in 31.01 seconds: 145.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 875 (time 1787142808.36 total 10.01 last 87.42) - open/close 1591 (time 1787142818.37 total 20.02 last 71.55) - open/close 2333 (time 1787142828.38 total 30.03 last 74.11) - open/close 3187 (time 1787142838.38 total 40.03 last 85.38) - open/close 3784 (time 1787142848.39 total 50.04 last 59.66) total: 4500 open/close in 59.57 seconds: 75.54 ops/second - /unlink 1318 (time 1787142872.16 total 10.00 last 131.74) - /unlink 2681 (time 1787142882.18 total 20.02 last 136.11) - /unlink 3730 (time 1787142892.18 total 30.02 last 104.90) total: 4500 /unlink in 36.03 seconds: 124.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 753 (time 1787142913.89 total 10.01 last 75.25) - open/close 1409 (time 1787142923.89 total 20.01 last 65.60) - open/close 2124 (time 1787142933.89 total 30.01 last 71.49) - open/close 2895 (time 1787142943.90 total 40.02 last 77.03) - open/close 3779 (time 1787142953.90 total 50.02 last 88.37) - open/close 4416 (time 1787142963.92 total 60.04 last 63.56) total: 4500 open/close in 61.25 seconds: 73.47 ops/second - /unlink 1133 (time 1787142981.86 total 10.00 last 113.26) - /unlink 2686 (time 1787142991.88 total 20.02 last 155.10) - /unlink 4065 (time 1787143001.88 total 30.02 last 137.84) total: 4500 /unlink in 33.82 seconds: 133.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 701 (time 1787143022.28 total 10.00 last 70.07) - open/close 1404 (time 1787143032.29 total 20.02 last 70.21) - open/close 2116 (time 1787143042.31 total 30.03 last 71.07) - open/close 3002 (time 1787143052.32 total 40.04 last 88.52) - open/close 3817 (time 1787143062.33 total 50.05 last 81.44) total: 4500 open/close in 60.03 seconds: 74.97 ops/second - /unlink 1391 (time 1787143088.52 total 10.00 last 139.04) - /unlink 3040 (time 1787143098.52 total 20.01 last 164.87) total: 4500 /unlink in 29.81 seconds: 150.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 765 (time 1787143122.91 total 10.00 last 76.49) - open/close 1482 (time 1787143132.94 total 20.03 last 71.50) - open/close 2234 (time 1787143142.95 total 30.04 last 75.13) - open/close 2965 (time 1787143152.95 total 40.04 last 73.10) - open/close 3459 (time 1787143162.95 total 50.04 last 49.38) - open/close 4319 (time 1787143172.95 total 60.04 last 85.98) total: 4500 open/close in 62.22 seconds: 72.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1263 (time 1787143189.62 total 10.01 last 126.22) - /unlink 2573 (time 1787143199.62 total 20.01 last 130.96) - /unlink 3925 (time 1787143209.63 total 30.01 last 135.14) total: 4500 /unlink in 34.13 seconds: 131.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 745 (time 1787143228.90 total 10.00 last 74.49) - open/close 1482 (time 1787143238.91 total 20.01 last 73.65) - open/close 2254 (time 1787143248.91 total 30.01 last 77.17) - open/close 3033 (time 1787143258.92 total 40.02 last 77.87) - open/close 3817 (time 1787143268.92 total 50.02 last 78.40) total: 4500 open/close in 59.31 seconds: 75.87 ops/second - /unlink 1499 (time 1787143295.32 total 10.00 last 149.84) - /unlink 2856 (time 1787143305.32 total 20.01 last 135.68) - /unlink 4232 (time 1787143315.32 total 30.01 last 137.58) total: 4500 /unlink in 32.30 seconds: 139.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 844 (time 1787143332.49 total 10.01 last 84.28) - open/close 1737 (time 1787143342.51 total 20.03 last 89.12) - open/close 2426 (time 1787143352.54 total 30.07 last 68.66) - open/close 3089 (time 1787143362.55 total 40.08 last 66.25) - open/close 3882 (time 1787143372.55 total 50.08 last 79.28) total: 4500 open/close in 57.07 seconds: 78.85 ops/second - /unlink 1109 (time 1787143394.31 total 10.00 last 110.87) - /unlink 2177 (time 1787143404.31 total 20.01 last 106.74) - /unlink 3325 (time 1787143414.32 total 30.01 last 114.77) total: 4500 /unlink in 39.91 seconds: 112.77 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95176 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: modules unloaded.