== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:46:53 (1780339613) Stopping clients: oleg424-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg424-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg424-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg424-server oleg424-server: oleg424-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg424-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg424-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 687 (time 1780339827.67 total 10.01 last 68.64) - open/close 1378 (time 1780339837.68 total 20.02 last 69.01) - open/close 2089 (time 1780339847.69 total 30.03 last 71.08) - open/close 2675 (time 1780339857.71 total 40.04 last 58.49) - open/close 3338 (time 1780339867.73 total 50.06 last 66.17) - open/close 3987 (time 1780339877.74 total 60.08 last 64.80) - open/close 4469 (time 1780339887.76 total 70.10 last 48.12) total: 4500 open/close in 70.98 seconds: 63.40 ops/second - /unlink 1041 (time 1780339905.06 total 10.01 last 104.00) - /unlink 2178 (time 1780339915.07 total 20.02 last 113.56) - /unlink 3200 (time 1780339925.07 total 30.02 last 102.20) total: 4500 /unlink in 39.97 seconds: 112.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 700 (time 1780339950.20 total 10.03 last 69.79) - open/close 1232 (time 1780339960.24 total 20.07 last 52.98) - open/close 1786 (time 1780339970.25 total 30.08 last 55.34) - open/close 2452 (time 1780339980.26 total 40.09 last 66.54) - open/close 3124 (time 1780339990.27 total 50.10 last 67.10) - open/close 3768 (time 1780340000.27 total 60.10 last 64.40) - open/close 4376 (time 1780340010.28 total 70.11 last 60.76) total: 4500 open/close in 72.37 seconds: 62.18 ops/second - /unlink 1049 (time 1780340027.45 total 10.00 last 104.86) - /unlink 2149 (time 1780340037.46 total 20.01 last 109.96) - /unlink 3234 (time 1780340047.46 total 30.01 last 108.46) - /unlink 4347 (time 1780340057.47 total 40.02 last 111.15) total: 4500 /unlink in 41.66 seconds: 108.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 597 (time 1780340074.60 total 10.02 last 59.60) - open/close 1223 (time 1780340084.62 total 20.04 last 62.46) - open/close 1885 (time 1780340094.62 total 30.04 last 66.19) - open/close 2516 (time 1780340104.65 total 40.07 last 62.94) - open/close 3152 (time 1780340114.65 total 50.07 last 63.58) - open/close 3857 (time 1780340124.65 total 60.07 last 70.49) total: 4500 open/close in 69.70 seconds: 64.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1111 (time 1780340150.20 total 10.00 last 111.09) - /unlink 2252 (time 1780340160.21 total 20.01 last 113.97) - /unlink 3382 (time 1780340170.21 total 30.01 last 112.98) total: 4500 /unlink in 39.66 seconds: 113.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 601 (time 1780340195.07 total 10.01 last 60.04) - open/close 1073 (time 1780340205.07 total 20.01 last 47.19) - open/close 1603 (time 1780340215.09 total 30.02 last 52.94) - open/close 2091 (time 1780340225.10 total 40.03 last 48.75) - open/close 2645 (time 1780340235.10 total 50.04 last 55.35) - open/close 3191 (time 1780340245.11 total 60.05 last 54.57) - open/close 3885 (time 1780340255.11 total 70.05 last 69.37) total: 4500 open/close in 78.78 seconds: 57.12 ops/second - /unlink 1190 (time 1780340279.41 total 10.01 last 118.92) - /unlink 2283 (time 1780340289.42 total 20.01 last 109.27) - /unlink 3397 (time 1780340299.43 total 30.03 last 111.21) - /unlink 4360 (time 1780340309.45 total 40.04 last 96.14) total: 4500 /unlink in 42.35 seconds: 106.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 415 (time 1780340328.45 total 10.02 last 41.44) - open/close 732 (time 1780340338.46 total 20.03 last 31.66) - open/close 1156 (time 1780340348.48 total 30.05 last 42.32) - open/close 1832 (time 1780340358.49 total 40.06 last 67.52) - open/close 2465 (time 1780340368.50 total 50.06 last 63.28) - open/close 3115 (time 1780340378.50 total 60.07 last 64.96) - open/close 3756 (time 1780340388.51 total 70.07 last 64.09) - open/close 4420 (time 1780340398.51 total 80.07 last 66.40) total: 4500 open/close in 81.21 seconds: 55.41 ops/second - /unlink 1029 (time 1780340415.41 total 10.00 last 102.88) - /unlink 2185 (time 1780340425.41 total 20.00 last 115.56) - /unlink 3392 (time 1780340435.42 total 30.01 last 120.62) - /unlink 4482 (time 1780340445.42 total 40.01 last 108.99) total: 4500 /unlink in 40.18 seconds: 112.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 584 (time 1780340461.09 total 10.02 last 58.30) - open/close 1112 (time 1780340471.10 total 20.03 last 52.75) - open/close 1732 (time 1780340481.10 total 30.03 last 61.95) - open/close 2281 (time 1780340491.11 total 40.04 last 54.86) - open/close 2896 (time 1780340501.11 total 50.05 last 61.48) - open/close 3362 (time 1780340511.13 total 60.06 last 46.54) - open/close 3910 (time 1780340521.15 total 70.08 last 54.69) - open/close 4445 (time 1780340531.15 total 80.08 last 53.49) total: 4500 open/close in 81.18 seconds: 55.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 950 (time 1780340547.21 total 10.00 last 94.96) - /unlink 2033 (time 1780340557.23 total 20.02 last 108.14) - /unlink 3175 (time 1780340567.23 total 30.02 last 114.17) - /unlink 4195 (time 1780340577.23 total 40.03 last 101.96) total: 4500 /unlink in 43.39 seconds: 103.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 577 (time 1780340596.22 total 10.01 last 57.66) - open/close 1145 (time 1780340606.23 total 20.02 last 56.74) - open/close 1758 (time 1780340616.27 total 30.06 last 61.06) - open/close 2374 (time 1780340626.27 total 40.06 last 61.58) - open/close 3049 (time 1780340636.28 total 50.07 last 67.42) - open/close 3665 (time 1780340646.29 total 60.07 last 61.58) - open/close 4267 (time 1780340656.30 total 70.08 last 60.14) total: 4500 open/close in 73.76 seconds: 61.01 ops/second - /unlink 1169 (time 1780340675.10 total 10.01 last 116.80) - /unlink 2168 (time 1780340685.11 total 20.01 last 99.88) - /unlink 3169 (time 1780340695.11 total 30.02 last 100.04) - /unlink 4192 (time 1780340705.11 total 40.02 last 102.30) total: 4500 /unlink in 43.39 seconds: 103.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 596 (time 1780340724.33 total 10.01 last 59.57) - open/close 1252 (time 1780340734.34 total 20.02 last 65.53) - open/close 1903 (time 1780340744.35 total 30.03 last 65.00) - open/close 2482 (time 1780340754.35 total 40.03 last 57.89) - open/close 3059 (time 1780340764.36 total 50.04 last 57.69) - open/close 3603 (time 1780340774.36 total 60.04 last 54.37) - open/close 4137 (time 1780340784.37 total 70.05 last 53.36) total: 4500 open/close in 75.84 seconds: 59.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1223 (time 1780340805.59 total 10.02 last 122.09) - /unlink 2319 (time 1780340815.60 total 20.02 last 109.56) - /unlink 3486 (time 1780340825.60 total 30.02 last 116.68) total: 4500 /unlink in 38.84 seconds: 115.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 555 (time 1780340850.29 total 10.01 last 55.47) - open/close 1104 (time 1780340860.30 total 20.02 last 54.83) - open/close 1577 (time 1780340870.31 total 30.03 last 47.25) - open/close 2200 (time 1780340880.32 total 40.04 last 62.24) - open/close 2766 (time 1780340890.34 total 50.05 last 56.50) - open/close 3338 (time 1780340900.35 total 60.07 last 57.14) - open/close 3954 (time 1780340910.36 total 70.08 last 61.49) total: 4500 open/close in 79.11 seconds: 56.88 ops/second - /unlink 957 (time 1780340934.83 total 10.01 last 95.61) - /unlink 1819 (time 1780340944.84 total 20.02 last 86.11) - /unlink 2750 (time 1780340954.84 total 30.02 last 93.09) - /unlink 3771 (time 1780340964.84 total 40.02 last 102.06) total: 4500 /unlink in 46.97 seconds: 95.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 550 (time 1780340987.87 total 10.00 last 54.98) - open/close 1086 (time 1780340997.88 total 20.01 last 53.56) - open/close 1652 (time 1780341007.91 total 30.04 last 56.41) - open/close 2222 (time 1780341017.91 total 40.05 last 56.98) - open/close 2801 (time 1780341027.92 total 50.06 last 57.82) - open/close 3384 (time 1780341037.94 total 60.07 last 58.23) - open/close 3978 (time 1780341047.95 total 70.09 last 59.31) total: 4500 open/close in 79.25 seconds: 56.78 ops/second - /unlink 1010 (time 1780341072.80 total 10.01 last 100.93) - /unlink 1987 (time 1780341082.80 total 20.01 last 97.67) - /unlink 2807 (time 1780341092.81 total 30.02 last 81.89) - /unlink 3452 (time 1780341102.81 total 40.02 last 64.49) - /unlink 4277 (time 1780341112.83 total 50.04 last 82.39) total: 4500 /unlink in 52.71 seconds: 85.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 655 (time 1780341131.40 total 10.02 last 65.39) - open/close 1178 (time 1780341141.43 total 20.05 last 52.14) - open/close 1631 (time 1780341151.46 total 30.07 last 45.19) - open/close 2057 (time 1780341161.46 total 40.08 last 42.59) - open/close 2588 (time 1780341171.47 total 50.09 last 53.04) - open/close 3097 (time 1780341181.48 total 60.10 last 50.83) - open/close 3591 (time 1780341191.51 total 70.13 last 49.26) - open/close 4221 (time 1780341201.52 total 80.13 last 62.97) total: 4500 open/close in 84.65 seconds: 53.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 977 (time 1780341221.12 total 10.01 last 97.62) - /unlink 2096 (time 1780341231.13 total 20.01 last 111.84) - /unlink 2993 (time 1780341241.13 total 30.02 last 89.67) - /unlink 4104 (time 1780341251.13 total 40.02 last 111.08) total: 4500 /unlink in 44.33 seconds: 101.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 533 (time 1780341270.50 total 10.01 last 53.25) - open/close 1088 (time 1780341280.51 total 20.02 last 55.43) - open/close 1690 (time 1780341290.51 total 30.02 last 60.19) - open/close 2276 (time 1780341300.52 total 40.03 last 58.54) - open/close 2829 (time 1780341310.53 total 50.04 last 55.27) - open/close 3417 (time 1780341320.54 total 60.05 last 58.74) - open/close 3977 (time 1780341330.54 total 70.05 last 56.00) total: 4500 open/close in 77.97 seconds: 57.71 ops/second - /unlink 1081 (time 1780341353.81 total 10.00 last 108.06) - /unlink 2180 (time 1780341363.82 total 20.01 last 109.85) - /unlink 3262 (time 1780341373.82 total 30.01 last 108.15) - /unlink 4293 (time 1780341383.82 total 40.01 last 103.09) total: 4500 /unlink in 42.17 seconds: 106.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 587 (time 1780341400.61 total 10.00 last 58.69) - open/close 1188 (time 1780341410.62 total 20.01 last 60.04) - open/close 1706 (time 1780341420.62 total 30.01 last 51.79) - open/close 2266 (time 1780341430.62 total 40.02 last 55.97) - open/close 2836 (time 1780341440.64 total 50.03 last 56.92) - open/close 3440 (time 1780341450.64 total 60.03 last 60.38) - open/close 4059 (time 1780341460.65 total 70.05 last 61.83) total: 4500 open/close in 78.10 seconds: 57.62 ops/second - /unlink 1062 (time 1780341483.93 total 10.01 last 106.14) - /unlink 2123 (time 1780341493.95 total 20.02 last 105.96) - /unlink 3123 (time 1780341503.95 total 30.02 last 99.97) - /unlink 4255 (time 1780341513.95 total 40.02 last 113.18) total: 4500 /unlink in 42.42 seconds: 106.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 517 (time 1780341531.57 total 10.01 last 51.66) - open/close 1123 (time 1780341541.58 total 20.01 last 60.59) - open/close 1670 (time 1780341551.58 total 30.01 last 54.68) - open/close 2218 (time 1780341561.59 total 40.02 last 54.75) - open/close 2866 (time 1780341571.59 total 50.03 last 64.76) - open/close 3489 (time 1780341581.60 total 60.03 last 62.29) - open/close 4107 (time 1780341591.61 total 70.05 last 61.70) total: 4500 open/close in 76.62 seconds: 58.73 ops/second - /unlink 1132 (time 1780341612.55 total 10.01 last 113.08) - /unlink 2312 (time 1780341622.56 total 20.02 last 117.94) - /unlink 3460 (time 1780341632.57 total 30.03 last 114.64) total: 4500 /unlink in 39.88 seconds: 112.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 603 (time 1780341656.58 total 10.01 last 60.21) - open/close 1213 (time 1780341666.58 total 20.02 last 60.97) - open/close 1821 (time 1780341676.59 total 30.03 last 60.74) - open/close 2411 (time 1780341686.60 total 40.04 last 58.95) - open/close 3051 (time 1780341696.61 total 50.04 last 63.96) - open/close 3699 (time 1780341706.62 total 60.06 last 64.72) - open/close 4348 (time 1780341716.63 total 70.07 last 64.82) total: 4500 open/close in 72.27 seconds: 62.26 ops/second - /unlink 1235 (time 1780341732.70 total 10.00 last 123.44) - /unlink 2628 (time 1780341742.70 total 20.01 last 139.26) - /unlink 3980 (time 1780341752.70 total 30.01 last 135.19) total: 4500 /unlink in 34.16 seconds: 131.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 676 (time 1780341771.32 total 10.02 last 67.48) - open/close 1418 (time 1780341781.32 total 20.02 last 74.20) - open/close 2172 (time 1780341791.33 total 30.02 last 75.35) - open/close 2944 (time 1780341801.34 total 40.03 last 77.12) - open/close 3692 (time 1780341811.34 total 50.04 last 74.77) - open/close 4423 (time 1780341821.34 total 60.04 last 73.09) total: 4500 open/close in 61.02 seconds: 73.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1456 (time 1780341835.94 total 10.01 last 145.52) - /unlink 2820 (time 1780341845.95 total 20.01 last 136.36) - /unlink 4337 (time 1780341855.95 total 30.01 last 151.65) total: 4500 /unlink in 31.24 seconds: 144.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 847 (time 1780341870.16 total 10.00 last 84.67) - open/close 1648 (time 1780341880.18 total 20.02 last 79.94) - open/close 2365 (time 1780341890.19 total 30.03 last 71.63) - open/close 3185 (time 1780341900.19 total 40.04 last 81.95) - open/close 4035 (time 1780341910.20 total 50.04 last 84.99) total: 4500 open/close in 55.79 seconds: 80.66 ops/second - /unlink 1533 (time 1780341929.43 total 10.01 last 153.22) - /unlink 3134 (time 1780341939.44 total 20.01 last 160.04) total: 4500 /unlink in 28.84 seconds: 156.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 843 (time 1780341961.58 total 10.02 last 84.16) - open/close 1640 (time 1780341971.59 total 20.02 last 79.69) - open/close 2469 (time 1780341981.59 total 30.02 last 82.86) - open/close 3304 (time 1780341991.60 total 40.03 last 83.43) - open/close 4192 (time 1780342001.62 total 50.05 last 88.65) total: 4500 open/close in 53.92 seconds: 83.46 ops/second - /unlink 1571 (time 1780342018.47 total 10.00 last 157.09) - /unlink 3169 (time 1780342028.47 total 20.00 last 159.79) total: 4500 /unlink in 27.87 seconds: 161.48 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94035 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: modules unloaded.