== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 06:03:18 (1787565798) Stopping clients: oleg654-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg654-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg654-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg654-server oleg654-server: oleg654-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg654-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg654-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg654-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 950 (time 1787565977.36 total 10.01 last 94.90) - open/close 1784 (time 1787565987.36 total 20.01 last 83.39) - open/close 2689 (time 1787565997.37 total 30.02 last 90.41) - open/close 3495 (time 1787566007.38 total 40.02 last 80.58) - open/close 4227 (time 1787566017.38 total 50.03 last 73.14) total: 4500 open/close in 53.96 seconds: 83.40 ops/second - /unlink 1539 (time 1787566035.40 total 10.01 last 153.74) - /unlink 3195 (time 1787566045.40 total 20.01 last 165.59) total: 4500 /unlink in 27.89 seconds: 161.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 906 (time 1787566067.56 total 10.01 last 90.51) - open/close 1830 (time 1787566077.57 total 20.02 last 92.31) - open/close 2767 (time 1787566087.57 total 30.03 last 93.63) - open/close 3579 (time 1787566097.58 total 40.03 last 81.18) - open/close 4499 (time 1787566107.58 total 50.03 last 91.98) total: 4500 open/close in 50.03 seconds: 89.95 ops/second - /unlink 1706 (time 1787566121.85 total 10.01 last 170.46) - /unlink 3263 (time 1787566131.86 total 20.01 last 155.59) total: 4500 /unlink in 29.31 seconds: 153.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 917 (time 1787566155.49 total 10.01 last 91.64) - open/close 1783 (time 1787566165.50 total 20.01 last 86.54) - open/close 2703 (time 1787566175.51 total 30.02 last 91.95) - open/close 3630 (time 1787566185.52 total 40.03 last 92.56) total: 4500 open/close in 49.95 seconds: 90.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1735 (time 1787566209.60 total 10.01 last 173.36) - /unlink 3428 (time 1787566219.60 total 20.01 last 169.23) total: 4500 /unlink in 26.71 seconds: 168.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 893 (time 1787566240.56 total 10.02 last 89.16) - open/close 1744 (time 1787566250.56 total 20.02 last 85.07) - open/close 2607 (time 1787566260.57 total 30.03 last 86.24) - open/close 3450 (time 1787566270.57 total 40.03 last 84.29) - open/close 4313 (time 1787566280.58 total 50.03 last 86.23) total: 4500 open/close in 52.05 seconds: 86.46 ops/second - /unlink 1770 (time 1787566296.37 total 10.00 last 176.96) - /unlink 3270 (time 1787566306.37 total 20.00 last 149.99) total: 4500 /unlink in 27.78 seconds: 161.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 887 (time 1787566327.65 total 10.00 last 88.67) - open/close 1795 (time 1787566337.66 total 20.01 last 90.71) - open/close 2682 (time 1787566347.67 total 30.02 last 88.68) - open/close 3599 (time 1787566357.67 total 40.02 last 91.64) - open/close 4434 (time 1787566367.67 total 50.02 last 83.48) total: 4500 open/close in 50.79 seconds: 88.61 ops/second - /unlink 1826 (time 1787566382.79 total 10.00 last 182.53) - /unlink 3576 (time 1787566392.79 total 20.00 last 175.00) total: 4500 /unlink in 25.40 seconds: 177.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 758 (time 1787566412.09 total 10.01 last 75.73) - open/close 1549 (time 1787566422.10 total 20.02 last 79.02) - open/close 2149 (time 1787566432.11 total 30.03 last 59.94) - open/close 2763 (time 1787566442.17 total 40.09 last 61.04) - open/close 3577 (time 1787566452.18 total 50.10 last 81.31) - open/close 4356 (time 1787566462.19 total 60.12 last 77.78) total: 4500 open/close in 61.92 seconds: 72.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1577 (time 1787566478.47 total 10.01 last 157.56) - /unlink 3063 (time 1787566488.47 total 20.02 last 148.48) total: 4500 /unlink in 29.47 seconds: 152.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 962 (time 1787566511.63 total 10.01 last 96.09) - open/close 1806 (time 1787566521.63 total 20.01 last 84.38) - open/close 2708 (time 1787566531.63 total 30.02 last 90.18) - open/close 3585 (time 1787566541.64 total 40.02 last 87.67) total: 4500 open/close in 49.72 seconds: 90.50 ops/second - /unlink 1676 (time 1787566565.30 total 10.00 last 167.57) - /unlink 3304 (time 1787566575.31 total 20.01 last 162.66) total: 4500 /unlink in 27.11 seconds: 166.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 813 (time 1787566597.32 total 10.00 last 81.26) - open/close 1693 (time 1787566607.33 total 20.02 last 87.89) - open/close 2514 (time 1787566617.34 total 30.02 last 82.08) - open/close 3229 (time 1787566627.34 total 40.02 last 71.49) - open/close 4133 (time 1787566637.34 total 50.02 last 90.39) total: 4500 open/close in 54.11 seconds: 83.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1591 (time 1787566654.91 total 10.00 last 159.09) - /unlink 3197 (time 1787566664.91 total 20.00 last 160.58) total: 4500 /unlink in 27.99 seconds: 160.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 872 (time 1787566686.35 total 10.02 last 87.02) - open/close 1725 (time 1787566696.36 total 20.03 last 85.25) - open/close 2533 (time 1787566706.36 total 30.03 last 80.75) - open/close 3457 (time 1787566716.37 total 40.03 last 92.38) - open/close 4400 (time 1787566726.37 total 50.04 last 94.23) total: 4500 open/close in 51.14 seconds: 87.99 ops/second - /unlink 1597 (time 1787566741.94 total 10.00 last 159.66) - /unlink 3158 (time 1787566751.94 total 20.01 last 156.04) total: 4500 /unlink in 28.54 seconds: 157.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 802 (time 1787566775.58 total 10.02 last 80.06) - open/close 1626 (time 1787566785.59 total 20.03 last 82.34) - open/close 2388 (time 1787566795.59 total 30.03 last 76.19) - open/close 2959 (time 1787566805.60 total 40.03 last 57.06) - open/close 3845 (time 1787566815.60 total 50.03 last 88.59) total: 4500 open/close in 57.18 seconds: 78.69 ops/second - /unlink 1557 (time 1787566836.98 total 10.00 last 155.67) - /unlink 3076 (time 1787566847.00 total 20.02 last 151.68) total: 4500 /unlink in 28.36 seconds: 158.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 848 (time 1787566869.35 total 10.00 last 84.78) - open/close 1755 (time 1787566879.36 total 20.01 last 90.66) - open/close 2602 (time 1787566889.37 total 30.02 last 84.62) - open/close 3530 (time 1787566899.37 total 40.02 last 92.77) - open/close 4306 (time 1787566909.37 total 50.02 last 77.59) total: 4500 open/close in 52.40 seconds: 85.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1637 (time 1787566925.23 total 10.00 last 163.66) - /unlink 3180 (time 1787566935.23 total 20.01 last 154.23) total: 4500 /unlink in 29.31 seconds: 153.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 870 (time 1787566960.41 total 10.02 last 86.86) - open/close 1839 (time 1787566970.41 total 20.02 last 96.89) - open/close 2696 (time 1787566980.42 total 30.02 last 85.64) - open/close 3547 (time 1787566990.42 total 40.03 last 85.04) - open/close 4331 (time 1787567000.45 total 50.05 last 78.23) total: 4500 open/close in 52.07 seconds: 86.43 ops/second - /unlink 1621 (time 1787567017.89 total 10.00 last 162.09) - /unlink 3276 (time 1787567027.89 total 20.00 last 165.47) total: 4500 /unlink in 27.09 seconds: 166.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 756 (time 1787567048.96 total 10.01 last 75.51) - open/close 1640 (time 1787567058.97 total 20.02 last 88.33) - open/close 2545 (time 1787567069.00 total 30.05 last 90.27) - open/close 3387 (time 1787567079.00 total 40.05 last 84.17) - open/close 4214 (time 1787567089.00 total 50.05 last 82.67) total: 4500 open/close in 53.16 seconds: 84.64 ops/second - /unlink 1380 (time 1787567106.72 total 10.00 last 137.95) - /unlink 2974 (time 1787567116.73 total 20.01 last 159.30) total: 4500 /unlink in 29.35 seconds: 153.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 904 (time 1787567140.65 total 10.04 last 90.06) - open/close 1744 (time 1787567150.65 total 20.04 last 83.97) - open/close 2686 (time 1787567160.66 total 30.05 last 94.14) - open/close 3557 (time 1787567170.66 total 40.06 last 87.03) - open/close 4360 (time 1787567180.68 total 50.07 last 80.20) total: 4500 open/close in 51.77 seconds: 86.92 ops/second - /unlink 1471 (time 1787567197.60 total 10.00 last 147.10) - /unlink 2939 (time 1787567207.60 total 20.00 last 146.75) total: 4500 /unlink in 29.92 seconds: 150.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 825 (time 1787567231.39 total 10.01 last 82.43) - open/close 1638 (time 1787567241.41 total 20.02 last 81.21) - open/close 2430 (time 1787567251.41 total 30.02 last 79.20) - open/close 3201 (time 1787567261.41 total 40.03 last 77.05) - open/close 4083 (time 1787567271.42 total 50.03 last 88.16) total: 4500 open/close in 54.62 seconds: 82.39 ops/second - /unlink 1630 (time 1787567289.58 total 10.00 last 162.95) - /unlink 3346 (time 1787567299.58 total 20.01 last 171.52) total: 4500 /unlink in 26.75 seconds: 168.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 780 (time 1787567319.86 total 10.00 last 77.99) - open/close 1563 (time 1787567329.87 total 20.02 last 78.15) - open/close 2345 (time 1787567339.88 total 30.03 last 78.16) - open/close 2961 (time 1787567349.88 total 40.03 last 61.57) - open/close 3792 (time 1787567359.88 total 50.03 last 83.10) total: 4500 open/close in 58.68 seconds: 76.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1478 (time 1787567383.42 total 10.00 last 147.74) - /unlink 2831 (time 1787567393.42 total 20.01 last 135.28) - /unlink 4203 (time 1787567403.42 total 30.01 last 137.18) total: 4500 /unlink in 32.24 seconds: 139.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 799 (time 1787567420.64 total 10.00 last 79.88) - open/close 1563 (time 1787567430.64 total 20.01 last 76.36) - open/close 2331 (time 1787567440.65 total 30.01 last 76.75) - open/close 3116 (time 1787567450.66 total 40.02 last 78.44) - open/close 4010 (time 1787567460.66 total 50.03 last 89.37) total: 4500 open/close in 56.13 seconds: 80.17 ops/second - /unlink 1603 (time 1787567480.93 total 10.00 last 160.25) - /unlink 3114 (time 1787567490.93 total 20.01 last 151.04) total: 4500 /unlink in 29.82 seconds: 150.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 757 (time 1787567515.67 total 10.01 last 75.64) - open/close 1542 (time 1787567525.67 total 20.01 last 78.49) - open/close 2369 (time 1787567535.67 total 30.01 last 82.68) - open/close 3239 (time 1787567545.68 total 40.02 last 86.94) - open/close 4078 (time 1787567555.69 total 50.03 last 83.83) total: 4500 open/close in 54.97 seconds: 81.86 ops/second - /unlink 1606 (time 1787567575.12 total 10.00 last 160.58) - /unlink 3187 (time 1787567585.13 total 20.00 last 158.05) total: 4500 /unlink in 28.46 seconds: 158.14 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95193 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg654-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg654-server unloading modules via unload_modules_local on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing unload_modules_local oleg654-server: modules unloaded.