== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:46:36 (1785469596) Stopping clients: oleg615-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg615-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg615-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg615-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg615-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg615-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg615-server oleg615-server: oleg615-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing load_modules_local oleg615-server: Loading modules from /home/green/git/lustre-release/lustre oleg615-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg615-server: detected 4 online CPUs by sysfs oleg615-server: MODOPTS_LIBCFS= oleg615-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg615-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg615-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg615-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug -1 all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 727 (time 1785469797.37 total 10.01 last 72.66) - open/close 1578 (time 1785469807.38 total 20.02 last 85.00) - open/close 2490 (time 1785469817.38 total 30.02 last 91.20) - open/close 3348 (time 1785469827.39 total 40.03 last 85.72) - open/close 4029 (time 1785469837.41 total 50.05 last 67.95) total: 4500 open/close in 56.20 seconds: 80.07 ops/second - /unlink 1517 (time 1785469858.11 total 10.00 last 151.66) - /unlink 2770 (time 1785469868.12 total 20.01 last 125.15) - /unlink 4108 (time 1785469878.13 total 30.02 last 133.71) total: 4500 /unlink in 32.89 seconds: 136.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 835 (time 1785469896.08 total 10.00 last 83.49) - open/close 1649 (time 1785469906.08 total 20.00 last 81.40) - open/close 2409 (time 1785469916.09 total 30.01 last 75.93) - open/close 3210 (time 1785469926.09 total 40.02 last 80.07) - open/close 4033 (time 1785469936.10 total 50.03 last 82.19) total: 4500 open/close in 56.00 seconds: 80.36 ops/second - /unlink 1434 (time 1785469956.26 total 10.01 last 143.30) - /unlink 2876 (time 1785469966.27 total 20.01 last 144.16) - /unlink 4204 (time 1785469976.27 total 30.02 last 132.71) total: 4500 /unlink in 32.03 seconds: 140.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 842 (time 1785469993.49 total 10.00 last 84.18) - open/close 1746 (time 1785470003.50 total 20.01 last 90.33) - open/close 2575 (time 1785470013.50 total 30.01 last 82.90) - open/close 3402 (time 1785470023.51 total 40.02 last 82.58) - open/close 4230 (time 1785470033.51 total 50.03 last 82.79) total: 4500 open/close in 52.96 seconds: 84.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1539 (time 1785470050.14 total 10.00 last 153.86) - /unlink 3091 (time 1785470060.15 total 20.01 last 155.13) total: 4500 /unlink in 29.14 seconds: 154.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 859 (time 1785470083.52 total 10.01 last 85.77) - open/close 1746 (time 1785470093.53 total 20.02 last 88.65) - open/close 2673 (time 1785470103.53 total 30.02 last 92.69) - open/close 3515 (time 1785470113.53 total 40.02 last 84.19) - open/close 4370 (time 1785470123.53 total 50.02 last 85.48) total: 4500 open/close in 51.22 seconds: 87.86 ops/second - /unlink 1592 (time 1785470138.85 total 10.01 last 159.11) - /unlink 3148 (time 1785470148.86 total 20.01 last 155.52) total: 4500 /unlink in 28.27 seconds: 159.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 879 (time 1785470171.39 total 10.02 last 87.76) - open/close 1768 (time 1785470181.39 total 20.02 last 88.84) - open/close 2598 (time 1785470191.40 total 30.03 last 82.94) - open/close 3444 (time 1785470201.42 total 40.04 last 84.47) - open/close 4292 (time 1785470211.42 total 50.05 last 84.79) total: 4500 open/close in 52.19 seconds: 86.22 ops/second - /unlink 1567 (time 1785470227.50 total 10.00 last 156.68) - /unlink 2815 (time 1785470237.50 total 20.00 last 124.76) - /unlink 4179 (time 1785470247.51 total 30.01 last 136.30) total: 4500 /unlink in 32.54 seconds: 138.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 913 (time 1785470264.53 total 10.01 last 91.18) - open/close 1756 (time 1785470274.53 total 20.02 last 84.27) - open/close 2617 (time 1785470284.54 total 30.03 last 86.01) - open/close 3572 (time 1785470294.56 total 40.05 last 95.30) - open/close 4427 (time 1785470304.58 total 50.06 last 85.37) total: 4500 open/close in 50.88 seconds: 88.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1610 (time 1785470319.25 total 10.00 last 161.00) - /unlink 3148 (time 1785470329.26 total 20.01 last 153.69) total: 4500 /unlink in 28.93 seconds: 155.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 835 (time 1785470352.36 total 10.01 last 83.43) - open/close 1660 (time 1785470362.36 total 20.01 last 82.48) - open/close 2542 (time 1785470372.37 total 30.01 last 88.20) - open/close 3332 (time 1785470382.37 total 40.02 last 78.96) - open/close 3997 (time 1785470392.38 total 50.03 last 66.41) total: 4500 open/close in 58.33 seconds: 77.15 ops/second - /unlink 1463 (time 1785470415.58 total 10.00 last 146.29) - /unlink 3081 (time 1785470425.58 total 20.00 last 161.78) total: 4500 /unlink in 28.98 seconds: 155.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 861 (time 1785470449.02 total 10.01 last 86.03) - open/close 1579 (time 1785470459.03 total 20.02 last 71.71) - open/close 2265 (time 1785470469.04 total 30.03 last 68.55) - open/close 3052 (time 1785470479.05 total 40.04 last 78.57) - open/close 3890 (time 1785470489.06 total 50.05 last 83.71) total: 4500 open/close in 57.35 seconds: 78.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1618 (time 1785470510.27 total 10.00 last 161.74) - /unlink 3121 (time 1785470520.28 total 20.02 last 150.12) total: 4500 /unlink in 29.40 seconds: 153.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 806 (time 1785470544.99 total 10.02 last 80.47) - open/close 1694 (time 1785470554.99 total 20.02 last 88.78) - open/close 2458 (time 1785470564.99 total 30.02 last 76.38) - open/close 3338 (time 1785470575.00 total 40.02 last 87.98) - open/close 4212 (time 1785470585.00 total 50.03 last 87.34) total: 4500 open/close in 53.73 seconds: 83.76 ops/second - /unlink 1520 (time 1785470602.67 total 10.01 last 151.89) - /unlink 3210 (time 1785470612.67 total 20.01 last 168.98) total: 4500 /unlink in 28.26 seconds: 159.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 910 (time 1785470635.67 total 10.01 last 90.89) - open/close 1816 (time 1785470645.67 total 20.01 last 90.59) - open/close 2702 (time 1785470655.67 total 30.02 last 88.57) - open/close 3673 (time 1785470665.68 total 40.02 last 97.09) total: 4500 open/close in 49.39 seconds: 91.12 ops/second - /unlink 1436 (time 1785470688.83 total 10.01 last 143.49) - /unlink 2606 (time 1785470698.86 total 20.03 last 116.68) - /unlink 3851 (time 1785470708.86 total 30.04 last 124.45) total: 4500 /unlink in 34.11 seconds: 131.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 833 (time 1785470727.38 total 10.01 last 83.20) - open/close 1770 (time 1785470737.38 total 20.01 last 93.69) - open/close 2691 (time 1785470747.38 total 30.01 last 92.08) - open/close 3544 (time 1785470757.39 total 40.03 last 85.21) - open/close 4382 (time 1785470767.40 total 50.03 last 83.72) total: 4500 open/close in 51.35 seconds: 87.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1438 (time 1785470783.04 total 10.00 last 143.77) - /unlink 2850 (time 1785470793.04 total 20.00 last 141.17) total: 4500 /unlink in 29.55 seconds: 152.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 848 (time 1785470816.89 total 10.01 last 84.73) - open/close 1742 (time 1785470826.89 total 20.01 last 89.36) - open/close 2631 (time 1785470836.90 total 30.02 last 88.82) - open/close 3556 (time 1785470846.91 total 40.03 last 92.41) - open/close 4495 (time 1785470856.92 total 50.04 last 93.85) total: 4500 open/close in 50.07 seconds: 89.88 ops/second - /unlink 1643 (time 1785470870.44 total 10.00 last 164.29) - /unlink 3225 (time 1785470880.45 total 20.01 last 158.01) total: 4500 /unlink in 27.81 seconds: 161.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 917 (time 1785470902.54 total 10.00 last 91.67) - open/close 1628 (time 1785470912.54 total 20.01 last 71.08) - open/close 2540 (time 1785470922.55 total 30.02 last 91.07) - open/close 3486 (time 1785470932.56 total 40.03 last 94.54) - open/close 4351 (time 1785470942.56 total 50.03 last 86.47) total: 4500 open/close in 51.48 seconds: 87.41 ops/second - /unlink 1688 (time 1785470957.87 total 10.00 last 168.80) - /unlink 3254 (time 1785470967.87 total 20.00 last 156.59) total: 4500 /unlink in 28.30 seconds: 159.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 782 (time 1785470990.86 total 10.02 last 78.05) - open/close 1570 (time 1785471000.87 total 20.03 last 78.75) - open/close 2362 (time 1785471010.88 total 30.03 last 79.13) - open/close 3126 (time 1785471020.88 total 40.04 last 76.39) - open/close 3998 (time 1785471030.89 total 50.05 last 87.11) total: 4500 open/close in 55.75 seconds: 80.72 ops/second - /unlink 1669 (time 1785471051.10 total 10.01 last 166.80) - /unlink 3148 (time 1785471061.10 total 20.01 last 147.88) total: 4500 /unlink in 28.57 seconds: 157.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 921 (time 1785471083.76 total 10.01 last 91.99) - open/close 1824 (time 1785471093.77 total 20.02 last 90.25) - open/close 2751 (time 1785471103.79 total 30.04 last 92.51) - open/close 3712 (time 1785471113.79 total 40.04 last 96.08) - open/close 4422 (time 1785471123.80 total 50.05 last 70.95) total: 4500 open/close in 50.93 seconds: 88.36 ops/second - /unlink 1704 (time 1785471138.89 total 10.00 last 170.37) - /unlink 3342 (time 1785471148.89 total 20.01 last 163.73) total: 4500 /unlink in 27.54 seconds: 163.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 931 (time 1785471169.81 total 10.01 last 92.99) - open/close 1803 (time 1785471179.81 total 20.01 last 87.18) - open/close 2656 (time 1785471189.81 total 30.02 last 85.26) - open/close 3542 (time 1785471199.82 total 40.02 last 88.58) - open/close 4371 (time 1785471209.82 total 50.03 last 82.83) total: 4500 open/close in 51.45 seconds: 87.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1751 (time 1785471224.45 total 10.00 last 175.03) - /unlink 3231 (time 1785471234.45 total 20.01 last 147.92) total: 4500 /unlink in 27.34 seconds: 164.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 972 (time 1785471254.89 total 10.01 last 97.12) - open/close 2006 (time 1785471264.89 total 20.01 last 103.40) - open/close 2920 (time 1785471274.89 total 30.01 last 91.40) - open/close 3786 (time 1785471284.90 total 40.01 last 86.56) total: 4500 open/close in 47.59 seconds: 94.55 ops/second - /unlink 1636 (time 1785471306.27 total 10.00 last 163.56) - /unlink 3181 (time 1785471316.27 total 20.00 last 154.47) total: 4500 /unlink in 28.35 seconds: 158.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 793 (time 1785471339.12 total 10.01 last 79.22) - open/close 1528 (time 1785471349.13 total 20.03 last 73.37) - open/close 2142 (time 1785471359.14 total 30.03 last 61.37) - open/close 2844 (time 1785471369.15 total 40.04 last 70.13) - open/close 3685 (time 1785471379.16 total 50.05 last 84.01) - open/close 4252 (time 1785471389.16 total 60.06 last 56.68) total: 4500 open/close in 63.53 seconds: 70.83 ops/second - /unlink 1487 (time 1785471407.64 total 10.02 last 148.38) - /unlink 3038 (time 1785471417.65 total 20.03 last 155.01) - /unlink 4455 (time 1785471427.65 total 30.03 last 141.69) total: 4500 /unlink in 30.60 seconds: 147.06 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95210 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg615-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg615-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg615-server stop mds service on oleg615-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg615-server stop mds service on oleg615-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg615-server unloading modules via unload_modules_local on: 'oleg615-server' oleg615-server: oleg615-server.virtnet: executing unload_modules_local oleg615-server: modules unloaded.