== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:14:12 (1780341252) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg459-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg459-server oleg459-server: oleg459-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg459-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg459-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 688 (time 1780341449.35 total 10.01 last 68.76) - open/close 1445 (time 1780341459.38 total 20.03 last 75.52) - open/close 2203 (time 1780341469.38 total 30.04 last 75.74) - open/close 3023 (time 1780341479.39 total 40.04 last 81.94) - open/close 3758 (time 1780341489.40 total 50.05 last 73.45) total: 4500 open/close in 59.99 seconds: 75.02 ops/second - /unlink 1453 (time 1780341513.48 total 10.00 last 145.28) - /unlink 2791 (time 1780341523.48 total 20.01 last 133.69) - /unlink 4187 (time 1780341533.49 total 30.01 last 139.56) total: 4500 /unlink in 32.39 seconds: 138.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 715 (time 1780341550.79 total 10.01 last 71.44) - open/close 1482 (time 1780341560.80 total 20.02 last 76.57) - open/close 2276 (time 1780341570.81 total 30.03 last 79.37) - open/close 3076 (time 1780341580.81 total 40.03 last 79.96) - open/close 3770 (time 1780341590.81 total 50.03 last 69.39) total: 4500 open/close in 59.80 seconds: 75.25 ops/second - /unlink 1197 (time 1780341614.94 total 10.00 last 119.66) - /unlink 2710 (time 1780341624.94 total 20.00 last 151.29) - /unlink 4064 (time 1780341634.95 total 30.01 last 135.30) total: 4500 /unlink in 33.13 seconds: 135.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 772 (time 1780341653.44 total 10.02 last 77.05) - open/close 1536 (time 1780341663.44 total 20.03 last 76.36) - open/close 2326 (time 1780341673.44 total 30.03 last 78.98) - open/close 3116 (time 1780341683.45 total 40.03 last 78.97) - open/close 3926 (time 1780341693.46 total 50.04 last 80.92) total: 4500 open/close in 57.21 seconds: 78.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1411 (time 1780341714.26 total 10.00 last 141.05) - /unlink 2877 (time 1780341724.26 total 20.01 last 146.51) - /unlink 4395 (time 1780341734.26 total 30.01 last 151.78) total: 4500 /unlink in 30.82 seconds: 146.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 807 (time 1780341748.73 total 10.01 last 80.63) - open/close 1590 (time 1780341758.74 total 20.02 last 78.20) - open/close 2367 (time 1780341768.74 total 30.02 last 77.69) - open/close 3198 (time 1780341778.74 total 40.02 last 83.09) - open/close 4066 (time 1780341788.74 total 50.02 last 86.80) total: 4500 open/close in 55.06 seconds: 81.74 ops/second - /unlink 1637 (time 1780341806.77 total 10.01 last 163.60) - /unlink 3186 (time 1780341816.77 total 20.01 last 154.85) total: 4500 /unlink in 28.58 seconds: 157.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 872 (time 1780341838.94 total 10.01 last 87.14) - open/close 1672 (time 1780341848.95 total 20.01 last 79.96) - open/close 2552 (time 1780341858.96 total 30.03 last 87.86) - open/close 3540 (time 1780341868.97 total 40.03 last 98.74) - open/close 4446 (time 1780341878.97 total 50.03 last 90.60) total: 4500 open/close in 50.69 seconds: 88.77 ops/second - /unlink 1654 (time 1780341892.84 total 10.00 last 165.39) - /unlink 3400 (time 1780341902.84 total 20.00 last 174.59) total: 4500 /unlink in 26.14 seconds: 172.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 903 (time 1780341921.64 total 10.01 last 90.20) - open/close 1818 (time 1780341931.64 total 20.01 last 91.50) - open/close 2713 (time 1780341941.64 total 30.02 last 89.45) - open/close 3602 (time 1780341951.65 total 40.02 last 88.86) total: 4500 open/close in 49.40 seconds: 91.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1779 (time 1780341973.76 total 10.01 last 177.80) - /unlink 3514 (time 1780341983.76 total 20.01 last 173.45) total: 4500 /unlink in 25.79 seconds: 174.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 928 (time 1780342002.54 total 10.01 last 92.68) - open/close 1850 (time 1780342012.55 total 20.02 last 92.15) - open/close 2759 (time 1780342022.55 total 30.02 last 90.89) - open/close 3741 (time 1780342032.57 total 40.04 last 98.03) total: 4500 open/close in 47.27 seconds: 95.20 ops/second - /unlink 1730 (time 1780342052.44 total 10.00 last 172.98) - /unlink 3480 (time 1780342062.44 total 20.00 last 174.96) total: 4500 /unlink in 25.59 seconds: 175.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 893 (time 1780342081.16 total 10.01 last 89.23) - open/close 1911 (time 1780342091.17 total 20.02 last 101.72) - open/close 2818 (time 1780342101.17 total 30.02 last 90.66) - open/close 3719 (time 1780342111.18 total 40.03 last 90.06) total: 4500 open/close in 48.74 seconds: 92.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1671 (time 1780342133.02 total 10.00 last 167.04) - /unlink 3518 (time 1780342143.02 total 20.01 last 184.64) total: 4500 /unlink in 25.57 seconds: 176.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 955 (time 1780342161.38 total 10.00 last 95.49) - open/close 1956 (time 1780342171.39 total 20.01 last 100.03) - open/close 2872 (time 1780342181.40 total 30.02 last 91.54) - open/close 3807 (time 1780342191.41 total 40.03 last 93.40) total: 4500 open/close in 48.63 seconds: 92.54 ops/second - /unlink 1644 (time 1780342213.13 total 10.00 last 164.39) - /unlink 3477 (time 1780342223.13 total 20.00 last 183.28) total: 4500 /unlink in 25.95 seconds: 173.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 942 (time 1780342241.63 total 10.01 last 94.09) - open/close 1814 (time 1780342251.63 total 20.02 last 87.15) - open/close 2759 (time 1780342261.63 total 30.02 last 94.49) - open/close 3643 (time 1780342271.64 total 40.03 last 88.34) total: 4500 open/close in 48.58 seconds: 92.63 ops/second - /unlink 1890 (time 1780342292.75 total 10.00 last 188.95) - /unlink 3718 (time 1780342302.75 total 20.01 last 182.72) total: 4500 /unlink in 24.52 seconds: 183.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 968 (time 1780342320.67 total 10.00 last 96.78) - open/close 2028 (time 1780342330.67 total 20.00 last 105.98) - open/close 3038 (time 1780342340.67 total 30.01 last 100.93) - open/close 4139 (time 1780342350.68 total 40.01 last 110.08) total: 4500 open/close in 43.16 seconds: 104.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1851 (time 1780342366.60 total 10.01 last 184.95) - /unlink 3630 (time 1780342376.60 total 20.01 last 177.90) total: 4500 /unlink in 24.85 seconds: 181.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 862 (time 1780342394.53 total 10.00 last 86.20) - open/close 1752 (time 1780342404.54 total 20.00 last 88.98) - open/close 2684 (time 1780342414.54 total 30.01 last 93.14) - open/close 3574 (time 1780342424.54 total 40.01 last 88.98) total: 4500 open/close in 48.94 seconds: 91.94 ops/second - /unlink 1821 (time 1780342445.99 total 10.00 last 182.09) - /unlink 3760 (time 1780342455.99 total 20.01 last 193.81) total: 4500 /unlink in 24.03 seconds: 187.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 949 (time 1780342472.67 total 10.01 last 94.85) - open/close 1895 (time 1780342482.69 total 20.03 last 94.40) - open/close 2802 (time 1780342492.70 total 30.04 last 90.61) - open/close 3726 (time 1780342502.70 total 40.04 last 92.37) total: 4500 open/close in 47.78 seconds: 94.17 ops/second - /unlink 1688 (time 1780342522.97 total 10.00 last 168.73) - /unlink 3456 (time 1780342532.98 total 20.01 last 176.71) total: 4500 /unlink in 26.30 seconds: 171.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1040 (time 1780342552.50 total 10.01 last 103.94) - open/close 2056 (time 1780342562.51 total 20.01 last 101.57) - open/close 3034 (time 1780342572.51 total 30.01 last 97.77) - open/close 4017 (time 1780342582.52 total 40.02 last 98.21) total: 4500 open/close in 45.02 seconds: 99.96 ops/second - /unlink 1702 (time 1780342600.65 total 10.00 last 170.18) - /unlink 3387 (time 1780342610.66 total 20.01 last 168.31) total: 4500 /unlink in 26.04 seconds: 172.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 996 (time 1780342629.49 total 10.00 last 99.56) - open/close 1987 (time 1780342639.50 total 20.01 last 99.04) - open/close 2902 (time 1780342649.51 total 30.02 last 91.41) - open/close 3938 (time 1780342659.51 total 40.02 last 103.55) total: 4500 open/close in 46.16 seconds: 97.49 ops/second - /unlink 1764 (time 1780342678.26 total 10.01 last 176.15) - /unlink 3526 (time 1780342688.27 total 20.02 last 176.15) total: 4500 /unlink in 25.22 seconds: 178.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 954 (time 1780342706.82 total 10.00 last 95.37) - open/close 1986 (time 1780342716.82 total 20.00 last 103.19) - open/close 2970 (time 1780342726.83 total 30.01 last 98.33) - open/close 3958 (time 1780342736.84 total 40.02 last 98.72) total: 4500 open/close in 45.33 seconds: 99.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1979 (time 1780342754.67 total 10.00 last 197.85) - /unlink 3931 (time 1780342764.67 total 20.01 last 195.12) total: 4500 /unlink in 22.84 seconds: 197.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 909 (time 1780342780.38 total 10.00 last 90.88) - open/close 1830 (time 1780342790.39 total 20.01 last 92.02) - open/close 2815 (time 1780342800.40 total 30.02 last 98.44) - open/close 3808 (time 1780342810.40 total 40.02 last 99.28) total: 4500 open/close in 47.62 seconds: 94.49 ops/second - /unlink 1766 (time 1780342830.69 total 10.00 last 176.58) - /unlink 2887 (time 1780342840.69 total 20.00 last 112.08) - /unlink 3620 (time 1780342850.69 total 30.00 last 73.30) total: 4500 /unlink in 37.79 seconds: 119.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 555 (time 1780342877.28 total 10.01 last 55.46) - open/close 1122 (time 1780342887.30 total 20.02 last 56.62) - open/close 1627 (time 1780342897.31 total 30.03 last 50.45) - open/close 2227 (time 1780342907.31 total 40.03 last 59.98) - open/close 2789 (time 1780342917.31 total 50.04 last 56.19) - open/close 3434 (time 1780342927.31 total 60.04 last 64.49) - open/close 4099 (time 1780342937.32 total 70.04 last 66.49) total: 4500 open/close in 75.95 seconds: 59.25 ops/second - /unlink 1450 (time 1780342957.58 total 10.00 last 144.93) - /unlink 2821 (time 1780342967.58 total 20.01 last 137.05) - /unlink 4292 (time 1780342977.58 total 30.01 last 147.08) total: 4500 /unlink in 31.63 seconds: 142.27 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94195 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server unloading modules via unload_modules_local on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local oleg459-server: modules unloaded.