== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:30:26 (1789266626) Stopping clients: oleg410-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg410-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg410-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg410-server oleg410-server: oleg410-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: kmemleak disabled oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg410-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg410-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 692 (time 1789266816.20 total 10.03 last 68.98) - open/close 1360 (time 1789266826.21 total 20.04 last 66.76) - open/close 1995 (time 1789266836.22 total 30.05 last 63.45) - open/close 2674 (time 1789266846.22 total 40.05 last 67.86) - open/close 3346 (time 1789266856.23 total 50.06 last 67.15) - open/close 3980 (time 1789266866.25 total 60.08 last 63.28) total: 4500 open/close in 68.19 seconds: 65.99 ops/second - /unlink 1249 (time 1789266888.24 total 10.00 last 124.84) - /unlink 2478 (time 1789266898.25 total 20.02 last 122.76) - /unlink 3758 (time 1789266908.26 total 30.03 last 127.87) total: 4500 /unlink in 35.38 seconds: 127.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 701 (time 1789266928.75 total 10.01 last 70.02) - open/close 1330 (time 1789266938.76 total 20.01 last 62.88) - open/close 1931 (time 1789266948.76 total 30.02 last 60.06) - open/close 2615 (time 1789266958.77 total 40.03 last 68.38) - open/close 3235 (time 1789266968.77 total 50.03 last 61.95) - open/close 3766 (time 1789266978.78 total 60.04 last 53.09) - open/close 4159 (time 1789266988.80 total 70.05 last 39.23) total: 4500 open/close in 75.47 seconds: 59.62 ops/second - /unlink 1225 (time 1789267008.50 total 10.00 last 122.45) - /unlink 2417 (time 1789267018.50 total 20.01 last 119.17) - /unlink 3580 (time 1789267028.51 total 30.02 last 116.20) total: 4500 /unlink in 38.08 seconds: 118.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 667 (time 1789267052.77 total 10.00 last 66.67) - open/close 1333 (time 1789267062.79 total 20.03 last 66.44) - open/close 1971 (time 1789267072.79 total 30.03 last 63.78) - open/close 2621 (time 1789267082.80 total 40.04 last 64.95) - open/close 3229 (time 1789267092.80 total 50.04 last 60.78) - open/close 3877 (time 1789267102.81 total 60.05 last 64.76) total: 4500 open/close in 69.28 seconds: 64.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1243 (time 1789267126.58 total 10.01 last 124.17) - /unlink 2384 (time 1789267136.59 total 20.02 last 114.05) - /unlink 3488 (time 1789267146.59 total 30.02 last 110.33) - /unlink 4302 (time 1789267156.60 total 40.03 last 81.33) total: 4500 /unlink in 41.91 seconds: 107.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 591 (time 1789267174.61 total 10.01 last 59.02) - open/close 1158 (time 1789267184.63 total 20.03 last 56.62) - open/close 1768 (time 1789267194.65 total 30.04 last 60.89) - open/close 2244 (time 1789267204.66 total 40.06 last 47.53) - open/close 2730 (time 1789267214.66 total 50.06 last 48.59) - open/close 3316 (time 1789267224.66 total 60.06 last 58.59) - open/close 3932 (time 1789267234.68 total 70.08 last 61.50) total: 4500 open/close in 80.05 seconds: 56.22 ops/second - /unlink 1063 (time 1789267259.38 total 10.01 last 106.23) - /unlink 1981 (time 1789267269.39 total 20.02 last 91.71) - /unlink 2784 (time 1789267279.40 total 30.02 last 80.28) - /unlink 3810 (time 1789267289.40 total 40.03 last 102.52) total: 4500 /unlink in 45.43 seconds: 99.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 577 (time 1789267309.19 total 10.00 last 57.68) - open/close 1179 (time 1789267319.19 total 20.00 last 60.19) - open/close 1855 (time 1789267329.21 total 30.02 last 67.47) - open/close 2477 (time 1789267339.24 total 40.05 last 62.06) - open/close 3112 (time 1789267349.25 total 50.06 last 63.41) - open/close 3778 (time 1789267359.28 total 60.09 last 66.40) - open/close 4448 (time 1789267369.28 total 70.10 last 66.97) total: 4500 open/close in 70.91 seconds: 63.46 ops/second - /unlink 1192 (time 1789267385.40 total 10.00 last 119.18) - /unlink 2391 (time 1789267395.40 total 20.01 last 119.85) - /unlink 3621 (time 1789267405.41 total 30.02 last 122.86) total: 4500 /unlink in 37.77 seconds: 119.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 607 (time 1789267429.62 total 10.01 last 60.66) - open/close 1212 (time 1789267439.62 total 20.01 last 60.49) - open/close 1858 (time 1789267449.63 total 30.02 last 64.54) - open/close 2490 (time 1789267459.64 total 40.02 last 63.16) - open/close 3074 (time 1789267469.65 total 50.04 last 58.32) - open/close 3688 (time 1789267479.66 total 60.04 last 61.36) - open/close 4309 (time 1789267489.67 total 70.05 last 62.04) total: 4500 open/close in 72.79 seconds: 61.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1373 (time 1789267507.77 total 10.00 last 137.26) - /unlink 2508 (time 1789267517.79 total 20.02 last 113.33) - /unlink 3575 (time 1789267527.79 total 30.02 last 106.68) total: 4500 /unlink in 37.44 seconds: 120.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 552 (time 1789267552.92 total 10.00 last 55.19) - open/close 1114 (time 1789267562.92 total 20.01 last 56.18) - open/close 1696 (time 1789267572.94 total 30.02 last 58.09) - open/close 2228 (time 1789267582.95 total 40.04 last 53.14) - open/close 2692 (time 1789267592.96 total 50.05 last 46.36) - open/close 3194 (time 1789267602.96 total 60.05 last 50.20) - open/close 3781 (time 1789267612.96 total 70.05 last 58.68) - open/close 4373 (time 1789267622.97 total 80.06 last 59.16) total: 4500 open/close in 81.66 seconds: 55.11 ops/second - /unlink 1219 (time 1789267640.11 total 10.00 last 121.84) - /unlink 2384 (time 1789267650.12 total 20.02 last 116.35) - /unlink 3548 (time 1789267660.13 total 30.03 last 116.22) total: 4500 /unlink in 37.98 seconds: 118.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 586 (time 1789267683.78 total 10.01 last 58.55) - open/close 1173 (time 1789267693.79 total 20.01 last 58.68) - open/close 1905 (time 1789267703.79 total 30.01 last 73.19) - open/close 2554 (time 1789267713.80 total 40.02 last 64.83) - open/close 3221 (time 1789267723.81 total 50.04 last 66.62) - open/close 3913 (time 1789267733.82 total 60.05 last 69.12) total: 4500 open/close in 68.46 seconds: 65.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1169 (time 1789267757.21 total 10.00 last 116.86) - /unlink 2297 (time 1789267767.22 total 20.01 last 112.77) - /unlink 3466 (time 1789267777.22 total 30.01 last 116.89) total: 4500 /unlink in 39.43 seconds: 114.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 615 (time 1789267802.44 total 10.01 last 61.41) - open/close 1223 (time 1789267812.46 total 20.04 last 60.66) - open/close 1844 (time 1789267822.47 total 30.04 last 62.07) - open/close 2455 (time 1789267832.49 total 40.06 last 60.99) - open/close 3089 (time 1789267842.50 total 50.07 last 63.35) - open/close 3576 (time 1789267852.51 total 60.08 last 48.64) - open/close 4120 (time 1789267862.53 total 70.10 last 54.29) total: 4500 open/close in 75.78 seconds: 59.38 ops/second - /unlink 1075 (time 1789267884.51 total 10.01 last 107.41) - /unlink 2244 (time 1789267894.51 total 20.01 last 116.86) - /unlink 3336 (time 1789267904.51 total 30.02 last 109.15) - /unlink 4471 (time 1789267914.52 total 40.02 last 113.46) total: 4500 /unlink in 40.24 seconds: 111.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 488 (time 1789267929.98 total 10.02 last 48.72) - open/close 843 (time 1789267939.99 total 20.03 last 35.47) - open/close 1349 (time 1789267949.99 total 30.03 last 50.58) - open/close 1865 (time 1789267959.99 total 40.03 last 51.59) - open/close 2425 (time 1789267970.00 total 50.04 last 55.95) - open/close 3042 (time 1789267980.00 total 60.04 last 61.67) - open/close 3666 (time 1789267990.00 total 70.04 last 62.40) - open/close 4250 (time 1789268000.01 total 80.05 last 58.36) total: 4500 open/close in 84.00 seconds: 53.57 ops/second - /unlink 1161 (time 1789268019.30 total 10.00 last 116.07) - /unlink 2284 (time 1789268029.30 total 20.01 last 112.27) - /unlink 3460 (time 1789268039.31 total 30.01 last 117.53) total: 4500 /unlink in 39.67 seconds: 113.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 606 (time 1789268064.39 total 10.01 last 60.55) - open/close 1227 (time 1789268074.40 total 20.02 last 62.03) - open/close 1874 (time 1789268084.41 total 30.02 last 64.67) - open/close 2474 (time 1789268094.41 total 40.02 last 60.00) - open/close 3085 (time 1789268104.42 total 50.03 last 61.04) - open/close 3692 (time 1789268114.43 total 60.05 last 60.61) - open/close 4310 (time 1789268124.44 total 70.06 last 61.74) total: 4500 open/close in 72.68 seconds: 61.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1054 (time 1789268143.14 total 10.01 last 105.30) - /unlink 2232 (time 1789268153.14 total 20.01 last 117.79) - /unlink 3464 (time 1789268163.14 total 30.01 last 123.14) total: 4500 /unlink in 38.65 seconds: 116.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 629 (time 1789268186.67 total 10.01 last 62.87) - open/close 1235 (time 1789268196.67 total 20.01 last 60.56) - open/close 1834 (time 1789268206.67 total 30.01 last 59.89) - open/close 2374 (time 1789268216.70 total 40.04 last 53.84) - open/close 2929 (time 1789268226.71 total 50.05 last 55.47) - open/close 3503 (time 1789268236.71 total 60.05 last 57.38) - open/close 4108 (time 1789268246.74 total 70.08 last 60.34) total: 4500 open/close in 76.21 seconds: 59.05 ops/second - /unlink 1084 (time 1789268268.79 total 10.00 last 108.38) - /unlink 2113 (time 1789268278.79 total 20.00 last 102.88) - /unlink 3263 (time 1789268288.80 total 30.01 last 114.89) - /unlink 4319 (time 1789268298.80 total 40.01 last 105.59) total: 4500 /unlink in 41.80 seconds: 107.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 528 (time 1789268315.67 total 10.01 last 52.76) - open/close 1111 (time 1789268325.67 total 20.01 last 58.26) - open/close 1688 (time 1789268335.71 total 30.05 last 57.52) - open/close 2295 (time 1789268345.71 total 40.05 last 60.65) - open/close 2957 (time 1789268355.73 total 50.07 last 66.06) - open/close 3612 (time 1789268365.74 total 60.08 last 65.45) - open/close 4275 (time 1789268375.75 total 70.09 last 66.24) total: 4500 open/close in 73.36 seconds: 61.34 ops/second - /unlink 1220 (time 1789268395.05 total 10.00 last 121.99) - /unlink 2325 (time 1789268405.05 total 20.00 last 110.48) - /unlink 3502 (time 1789268415.06 total 30.01 last 117.64) total: 4500 /unlink in 38.30 seconds: 117.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 590 (time 1789268439.24 total 10.02 last 58.86) - open/close 1250 (time 1789268449.24 total 20.03 last 65.97) - open/close 1822 (time 1789268459.24 total 30.03 last 57.19) - open/close 2421 (time 1789268469.27 total 40.05 last 59.76) - open/close 3033 (time 1789268479.29 total 50.07 last 61.08) - open/close 3669 (time 1789268489.29 total 60.07 last 63.59) - open/close 4339 (time 1789268499.29 total 70.08 last 66.95) total: 4500 open/close in 72.42 seconds: 62.14 ops/second - /unlink 1177 (time 1789268516.33 total 10.01 last 117.61) - /unlink 2332 (time 1789268526.33 total 20.01 last 115.50) - /unlink 3600 (time 1789268536.33 total 30.01 last 126.78) total: 4500 /unlink in 37.32 seconds: 120.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 552 (time 1789268558.20 total 10.01 last 55.15) - open/close 1129 (time 1789268568.20 total 20.01 last 57.69) - open/close 1780 (time 1789268578.21 total 30.02 last 65.02) - open/close 2445 (time 1789268588.22 total 40.03 last 66.46) - open/close 3165 (time 1789268598.22 total 50.03 last 71.99) - open/close 3819 (time 1789268608.22 total 60.04 last 65.36) total: 4500 open/close in 70.04 seconds: 64.25 ops/second - /unlink 1126 (time 1789268632.63 total 10.00 last 112.56) - /unlink 2101 (time 1789268642.63 total 20.00 last 97.49) - /unlink 3257 (time 1789268652.63 total 30.01 last 115.57) total: 4500 /unlink in 39.44 seconds: 114.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 513 (time 1789268678.06 total 10.01 last 51.27) - open/close 1162 (time 1789268688.07 total 20.02 last 64.83) - open/close 1664 (time 1789268698.08 total 30.03 last 50.11) - open/close 2304 (time 1789268708.09 total 40.04 last 63.97) - open/close 2932 (time 1789268718.09 total 50.04 last 62.78) - open/close 3531 (time 1789268728.10 total 60.05 last 59.86) - open/close 4123 (time 1789268738.11 total 70.07 last 59.10) total: 4500 open/close in 77.59 seconds: 58.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 925 (time 1789268762.28 total 10.01 last 92.37) - /unlink 2079 (time 1789268772.28 total 20.02 last 115.38) - /unlink 3363 (time 1789268782.29 total 30.03 last 128.27) - /unlink 4456 (time 1789268792.30 total 40.03 last 109.24) total: 4500 /unlink in 40.43 seconds: 111.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 670 (time 1789268807.48 total 10.01 last 66.96) - open/close 1313 (time 1789268817.51 total 20.03 last 64.16) - open/close 1986 (time 1789268827.51 total 30.03 last 67.28) - open/close 2604 (time 1789268837.52 total 40.04 last 61.75) - open/close 3222 (time 1789268847.54 total 50.06 last 61.68) - open/close 3848 (time 1789268857.54 total 60.06 last 62.56) - open/close 4497 (time 1789268867.57 total 70.10 last 64.70) total: 4500 open/close in 70.14 seconds: 64.16 ops/second - /unlink 1106 (time 1789268883.11 total 10.01 last 110.51) - /unlink 2174 (time 1789268893.12 total 20.02 last 106.69) - /unlink 3276 (time 1789268903.13 total 30.02 last 110.17) - /unlink 4429 (time 1789268913.13 total 40.02 last 115.26) total: 4500 /unlink in 41.14 seconds: 109.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 639 (time 1789268930.12 total 10.00 last 63.90) - open/close 1245 (time 1789268940.13 total 20.01 last 60.55) - open/close 1867 (time 1789268950.13 total 30.01 last 62.17) - open/close 2425 (time 1789268960.13 total 40.01 last 55.79) - open/close 3018 (time 1789268970.14 total 50.02 last 59.27) - open/close 3536 (time 1789268980.14 total 60.02 last 51.78) - open/close 4062 (time 1789268990.14 total 70.03 last 52.59) total: 4500 open/close in 77.31 seconds: 58.20 ops/second - /unlink 1176 (time 1789269013.13 total 10.01 last 117.50) - /unlink 2338 (time 1789269023.13 total 20.01 last 116.16) - /unlink 3426 (time 1789269033.14 total 30.02 last 108.73) total: 4500 /unlink in 38.94 seconds: 115.56 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95062 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg410-server unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: kmemleak disabled modules unloaded.