== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:22:07 (1788445327) Stopping clients: oleg659-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg659-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg659-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg659-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg659-server oleg659-server: oleg659-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing load_modules_local oleg659-server: Loading modules from /home/green/git/lustre-release/lustre oleg659-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg659-server: detected 4 online CPUs by sysfs oleg659-server: MODOPTS_LIBCFS= oleg659-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg659-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg659-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg659-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 811 (time 1788445529.41 total 10.00 last 81.08) - open/close 1666 (time 1788445539.43 total 20.02 last 85.32) - open/close 2408 (time 1788445549.43 total 30.03 last 74.18) - open/close 3272 (time 1788445559.44 total 40.03 last 86.35) - open/close 4103 (time 1788445569.47 total 50.06 last 82.86) total: 4500 open/close in 55.71 seconds: 80.78 ops/second - /unlink 1448 (time 1788445589.37 total 10.01 last 144.72) - /unlink 3086 (time 1788445599.37 total 20.01 last 163.79) total: 4500 /unlink in 29.07 seconds: 154.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 843 (time 1788445624.04 total 10.01 last 84.25) - open/close 1602 (time 1788445634.05 total 20.01 last 75.88) - open/close 2311 (time 1788445644.06 total 30.02 last 70.83) - open/close 3146 (time 1788445654.06 total 40.02 last 83.48) - open/close 3981 (time 1788445664.06 total 50.02 last 83.48) total: 4500 open/close in 56.49 seconds: 79.66 ops/second - /unlink 1688 (time 1788445685.07 total 10.00 last 168.74) - /unlink 3199 (time 1788445695.07 total 20.01 last 151.02) total: 4500 /unlink in 29.12 seconds: 154.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 851 (time 1788445718.80 total 10.01 last 85.05) - open/close 1720 (time 1788445728.80 total 20.01 last 86.87) - open/close 2586 (time 1788445738.80 total 30.01 last 86.57) - open/close 3374 (time 1788445748.82 total 40.03 last 78.69) - open/close 4274 (time 1788445758.86 total 50.06 last 89.67) total: 4500 open/close in 52.83 seconds: 85.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1691 (time 1788445775.53 total 10.00 last 169.09) - /unlink 2873 (time 1788445785.54 total 20.01 last 118.10) - /unlink 4138 (time 1788445795.54 total 30.01 last 126.48) total: 4500 /unlink in 32.61 seconds: 138.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 913 (time 1788445811.78 total 10.00 last 91.29) - open/close 1820 (time 1788445821.79 total 20.01 last 90.62) - open/close 2787 (time 1788445831.80 total 30.02 last 96.55) - open/close 3669 (time 1788445841.81 total 40.03 last 88.11) total: 4500 open/close in 49.27 seconds: 91.33 ops/second - /unlink 1635 (time 1788445864.68 total 10.01 last 163.41) - /unlink 3170 (time 1788445874.68 total 20.01 last 153.50) total: 4500 /unlink in 28.51 seconds: 157.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 743 (time 1788445897.87 total 10.01 last 74.22) - open/close 1581 (time 1788445907.89 total 20.03 last 83.67) - open/close 2365 (time 1788445917.89 total 30.03 last 78.36) - open/close 3180 (time 1788445927.90 total 40.05 last 81.40) - open/close 4025 (time 1788445937.91 total 50.06 last 84.41) total: 4500 open/close in 56.05 seconds: 80.28 ops/second - /unlink 1555 (time 1788445957.69 total 10.00 last 155.46) - /unlink 2774 (time 1788445967.70 total 20.02 last 121.67) - /unlink 4025 (time 1788445977.71 total 30.02 last 125.06) total: 4500 /unlink in 33.58 seconds: 133.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 807 (time 1788445994.98 total 10.00 last 80.69) - open/close 1665 (time 1788446004.98 total 20.00 last 85.80) - open/close 2560 (time 1788446015.01 total 30.03 last 89.25) - open/close 3374 (time 1788446025.02 total 40.04 last 81.33) - open/close 4191 (time 1788446035.02 total 50.04 last 81.70) total: 4500 open/close in 53.57 seconds: 84.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1608 (time 1788446052.76 total 10.00 last 160.78) - /unlink 3110 (time 1788446062.76 total 20.00 last 150.19) total: 4500 /unlink in 28.56 seconds: 157.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 931 (time 1788446085.89 total 10.00 last 93.08) - open/close 1804 (time 1788446095.89 total 20.00 last 87.29) - open/close 2727 (time 1788446105.90 total 30.01 last 92.24) - open/close 3658 (time 1788446115.90 total 40.01 last 93.08) total: 4500 open/close in 49.83 seconds: 90.31 ops/second - /unlink 1637 (time 1788446139.33 total 10.02 last 163.43) - /unlink 3223 (time 1788446149.33 total 20.02 last 158.60) total: 4500 /unlink in 28.54 seconds: 157.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 928 (time 1788446172.82 total 10.02 last 92.61) - open/close 1706 (time 1788446182.82 total 20.03 last 77.75) - open/close 2594 (time 1788446192.82 total 30.03 last 88.80) - open/close 3497 (time 1788446202.83 total 40.03 last 90.28) - open/close 4257 (time 1788446212.84 total 50.04 last 75.89) total: 4500 open/close in 52.83 seconds: 85.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1371 (time 1788446231.21 total 10.00 last 137.09) - /unlink 2844 (time 1788446241.22 total 20.01 last 147.18) - /unlink 4348 (time 1788446251.22 total 30.01 last 150.40) total: 4500 /unlink in 30.95 seconds: 145.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 870 (time 1788446266.32 total 10.01 last 86.88) - open/close 1752 (time 1788446276.32 total 20.02 last 88.14) - open/close 2633 (time 1788446286.33 total 30.03 last 88.01) - open/close 3492 (time 1788446296.34 total 40.03 last 85.87) - open/close 4314 (time 1788446306.34 total 50.04 last 82.16) total: 4500 open/close in 51.96 seconds: 86.60 ops/second - /unlink 1605 (time 1788446322.52 total 10.00 last 160.46) - /unlink 3129 (time 1788446332.52 total 20.00 last 152.40) total: 4500 /unlink in 28.53 seconds: 157.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 902 (time 1788446355.01 total 10.01 last 90.15) - open/close 1782 (time 1788446365.01 total 20.01 last 87.98) - open/close 2702 (time 1788446375.02 total 30.02 last 91.92) - open/close 3556 (time 1788446385.03 total 40.02 last 85.34) - open/close 4436 (time 1788446395.03 total 50.02 last 88.00) total: 4500 open/close in 50.76 seconds: 88.66 ops/second - /unlink 1638 (time 1788446410.20 total 10.00 last 163.76) - /unlink 3077 (time 1788446420.20 total 20.00 last 143.89) total: 4500 /unlink in 29.10 seconds: 154.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 816 (time 1788446443.68 total 10.01 last 81.51) - open/close 1704 (time 1788446453.68 total 20.02 last 88.75) - open/close 2610 (time 1788446463.69 total 30.02 last 90.55) - open/close 3518 (time 1788446473.69 total 40.03 last 90.76) - open/close 4423 (time 1788446483.70 total 50.03 last 90.46) total: 4500 open/close in 50.89 seconds: 88.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1651 (time 1788446498.13 total 10.01 last 165.02) - /unlink 3232 (time 1788446508.13 total 20.01 last 158.07) total: 4500 /unlink in 28.22 seconds: 159.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 891 (time 1788446531.43 total 10.01 last 89.05) - open/close 1772 (time 1788446541.43 total 20.01 last 88.04) - open/close 2694 (time 1788446551.44 total 30.02 last 92.15) - open/close 3542 (time 1788446561.45 total 40.03 last 84.73) - open/close 4414 (time 1788446571.45 total 50.03 last 87.17) total: 4500 open/close in 50.93 seconds: 88.36 ops/second - /unlink 1630 (time 1788446586.09 total 10.00 last 162.96) - /unlink 3089 (time 1788446596.09 total 20.00 last 145.90) total: 4500 /unlink in 29.39 seconds: 153.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 858 (time 1788446620.04 total 10.03 last 85.53) - open/close 1727 (time 1788446630.04 total 20.03 last 86.89) - open/close 2595 (time 1788446640.05 total 30.04 last 86.74) - open/close 3449 (time 1788446650.05 total 40.04 last 85.36) - open/close 4253 (time 1788446660.05 total 50.05 last 80.38) total: 4500 open/close in 53.26 seconds: 84.49 ops/second - /unlink 1596 (time 1788446677.45 total 10.00 last 159.60) - /unlink 2955 (time 1788446687.45 total 20.00 last 135.87) - /unlink 4330 (time 1788446697.45 total 30.00 last 137.47) total: 4500 /unlink in 31.55 seconds: 142.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 736 (time 1788446713.67 total 10.00 last 73.59) - open/close 1512 (time 1788446723.68 total 20.02 last 77.49) - open/close 2337 (time 1788446733.69 total 30.02 last 82.44) - open/close 3109 (time 1788446743.69 total 40.03 last 77.18) - open/close 3980 (time 1788446753.70 total 50.04 last 87.02) total: 4500 open/close in 56.37 seconds: 79.83 ops/second - /unlink 1640 (time 1788446774.21 total 10.00 last 163.93) - /unlink 3217 (time 1788446784.21 total 20.00 last 157.69) total: 4500 /unlink in 28.26 seconds: 159.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 865 (time 1788446806.09 total 10.00 last 86.46) - open/close 1724 (time 1788446816.10 total 20.01 last 85.85) - open/close 2553 (time 1788446826.10 total 30.02 last 82.86) - open/close 3376 (time 1788446836.11 total 40.03 last 82.20) - open/close 4195 (time 1788446846.12 total 50.03 last 81.84) total: 4500 open/close in 53.20 seconds: 84.59 ops/second - /unlink 1585 (time 1788446862.94 total 10.00 last 158.49) - /unlink 3142 (time 1788446872.94 total 20.00 last 155.70) total: 4500 /unlink in 28.93 seconds: 155.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 859 (time 1788446896.30 total 10.01 last 85.82) - open/close 1698 (time 1788446906.30 total 20.01 last 83.85) - open/close 2578 (time 1788446916.31 total 30.02 last 87.92) - open/close 3438 (time 1788446926.32 total 40.03 last 85.95) - open/close 4318 (time 1788446936.32 total 50.03 last 88.00) total: 4500 open/close in 52.12 seconds: 86.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1646 (time 1788446953.11 total 10.00 last 164.56) - /unlink 3171 (time 1788446963.11 total 20.01 last 152.45) total: 4500 /unlink in 28.99 seconds: 155.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 856 (time 1788446985.91 total 10.00 last 85.59) - open/close 1708 (time 1788446995.91 total 20.00 last 85.20) - open/close 2442 (time 1788447005.91 total 30.00 last 73.39) - open/close 3218 (time 1788447015.92 total 40.01 last 77.52) - open/close 4005 (time 1788447025.92 total 50.02 last 78.69) total: 4500 open/close in 56.63 seconds: 79.46 ops/second - /unlink 1335 (time 1788447046.29 total 10.00 last 133.47) - /unlink 2843 (time 1788447056.30 total 20.01 last 150.74) - /unlink 4408 (time 1788447066.30 total 30.01 last 156.44) total: 4500 /unlink in 30.76 seconds: 146.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 913 (time 1788447081.10 total 10.01 last 91.24) - open/close 1824 (time 1788447091.11 total 20.01 last 91.03) - open/close 2707 (time 1788447101.11 total 30.02 last 88.29) - open/close 3589 (time 1788447111.11 total 40.02 last 88.17) - open/close 4444 (time 1788447121.12 total 50.02 last 85.46) total: 4500 open/close in 50.71 seconds: 88.74 ops/second - /unlink 1667 (time 1788447135.86 total 10.01 last 166.60) - /unlink 3268 (time 1788447145.86 total 20.01 last 160.06) total: 4500 /unlink in 27.93 seconds: 161.13 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95144 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg659-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg659-server unloading modules via unload_modules_local on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing unload_modules_local oleg659-server: modules unloaded.