== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:10:31 (1787872231) Stopping clients: oleg320-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg320-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg320-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg320-server oleg320-server: oleg320-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing load_modules_local oleg320-server: Loading modules from /home/green/git/lustre-release/lustre oleg320-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg320-server: detected 4 online CPUs by sysfs oleg320-server: MODOPTS_LIBCFS= oleg320-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg320-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg320-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg320-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug -1 all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 667 (time 1787872438.26 total 10.01 last 66.62) - open/close 1354 (time 1787872448.27 total 20.03 last 68.58) - open/close 2019 (time 1787872458.27 total 30.03 last 66.49) - open/close 2692 (time 1787872468.28 total 40.03 last 67.28) - open/close 3353 (time 1787872478.28 total 50.04 last 66.08) - open/close 4032 (time 1787872488.30 total 60.06 last 67.77) total: 4500 open/close in 66.94 seconds: 67.22 ops/second - /unlink 1270 (time 1787872509.57 total 10.01 last 126.91) - /unlink 2421 (time 1787872519.57 total 20.01 last 115.10) - /unlink 3678 (time 1787872529.57 total 30.01 last 125.66) total: 4500 /unlink in 36.32 seconds: 123.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 690 (time 1787872551.06 total 10.00 last 68.99) - open/close 1331 (time 1787872561.07 total 20.01 last 64.06) - open/close 1932 (time 1787872571.07 total 30.01 last 60.07) - open/close 2504 (time 1787872581.09 total 40.02 last 57.13) - open/close 3140 (time 1787872591.10 total 50.04 last 63.53) - open/close 3754 (time 1787872601.10 total 60.04 last 61.36) - open/close 4372 (time 1787872611.11 total 70.05 last 61.78) total: 4500 open/close in 71.98 seconds: 62.52 ops/second - /unlink 1173 (time 1787872627.96 total 10.02 last 117.11) - /unlink 2375 (time 1787872637.97 total 20.02 last 120.10) - /unlink 3671 (time 1787872647.98 total 30.03 last 129.50) total: 4500 /unlink in 36.65 seconds: 122.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 724 (time 1787872669.54 total 10.01 last 72.35) - open/close 1369 (time 1787872679.56 total 20.02 last 64.39) - open/close 2037 (time 1787872689.58 total 30.04 last 66.68) - open/close 2672 (time 1787872699.59 total 40.06 last 63.41) - open/close 3272 (time 1787872709.60 total 50.07 last 59.94) - open/close 3951 (time 1787872719.62 total 60.08 last 67.81) total: 4500 open/close in 69.49 seconds: 64.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1269 (time 1787872744.22 total 10.00 last 126.86) - /unlink 2640 (time 1787872754.22 total 20.01 last 137.02) - /unlink 4064 (time 1787872764.23 total 30.01 last 142.38) total: 4500 /unlink in 33.15 seconds: 135.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 749 (time 1787872782.41 total 10.00 last 74.89) - open/close 1411 (time 1787872792.41 total 20.00 last 66.18) - open/close 2136 (time 1787872802.42 total 30.01 last 72.44) - open/close 2867 (time 1787872812.42 total 40.02 last 73.08) - open/close 3582 (time 1787872822.44 total 50.04 last 71.36) - open/close 4175 (time 1787872832.44 total 60.04 last 59.27) total: 4500 open/close in 65.00 seconds: 69.23 ops/second - /unlink 1327 (time 1787872852.56 total 10.00 last 132.66) - /unlink 2660 (time 1787872862.56 total 20.01 last 133.24) - /unlink 4013 (time 1787872872.56 total 30.01 last 135.26) total: 4500 /unlink in 33.54 seconds: 134.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 733 (time 1787872890.99 total 10.00 last 73.27) - open/close 1443 (time 1787872901.00 total 20.01 last 70.97) - open/close 2161 (time 1787872911.00 total 30.01 last 71.80) - open/close 2916 (time 1787872921.00 total 40.01 last 75.46) - open/close 3649 (time 1787872931.01 total 50.02 last 73.24) - open/close 4396 (time 1787872941.01 total 60.02 last 74.69) total: 4500 open/close in 61.30 seconds: 73.41 ops/second - /unlink 1364 (time 1787872956.17 total 10.00 last 136.38) - /unlink 2781 (time 1787872966.17 total 20.01 last 141.60) - /unlink 4100 (time 1787872976.17 total 30.01 last 131.90) total: 4500 /unlink in 32.76 seconds: 137.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 655 (time 1787872993.10 total 10.01 last 65.46) - open/close 1389 (time 1787873003.11 total 20.02 last 73.32) - open/close 2179 (time 1787873013.13 total 30.03 last 78.88) - open/close 2944 (time 1787873023.13 total 40.03 last 76.47) - open/close 3625 (time 1787873033.14 total 50.04 last 68.06) - open/close 4306 (time 1787873043.14 total 60.05 last 68.06) total: 4500 open/close in 62.58 seconds: 71.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1175 (time 1787873061.09 total 10.00 last 117.48) - /unlink 2357 (time 1787873071.09 total 20.01 last 118.14) - /unlink 3549 (time 1787873081.10 total 30.02 last 119.06) total: 4500 /unlink in 37.67 seconds: 119.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 723 (time 1787873103.24 total 10.01 last 72.20) - open/close 1461 (time 1787873113.24 total 20.02 last 73.79) - open/close 2234 (time 1787873123.24 total 30.02 last 77.27) - open/close 2986 (time 1787873133.25 total 40.02 last 75.17) - open/close 3744 (time 1787873143.26 total 50.04 last 75.68) - open/close 4403 (time 1787873153.27 total 60.05 last 65.82) total: 4500 open/close in 61.30 seconds: 73.41 ops/second - /unlink 1445 (time 1787873168.52 total 10.01 last 144.39) - /unlink 2678 (time 1787873178.53 total 20.02 last 123.20) - /unlink 3870 (time 1787873188.54 total 30.02 last 119.12) total: 4500 /unlink in 36.09 seconds: 124.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 582 (time 1787873211.03 total 10.00 last 58.18) - open/close 1251 (time 1787873221.04 total 20.00 last 66.89) - open/close 1991 (time 1787873231.04 total 30.01 last 73.98) - open/close 2696 (time 1787873241.04 total 40.01 last 70.46) - open/close 3367 (time 1787873251.05 total 50.02 last 67.04) - open/close 4113 (time 1787873261.06 total 60.03 last 74.55) total: 4500 open/close in 66.20 seconds: 67.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1339 (time 1787873282.20 total 10.00 last 133.87) - /unlink 2747 (time 1787873292.21 total 20.01 last 140.69) - /unlink 4048 (time 1787873302.21 total 30.01 last 130.08) total: 4500 /unlink in 34.16 seconds: 131.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 685 (time 1787873320.87 total 10.01 last 68.43) - open/close 1407 (time 1787873330.89 total 20.04 last 72.02) - open/close 2075 (time 1787873340.89 total 30.04 last 66.78) - open/close 2753 (time 1787873350.90 total 40.04 last 67.79) - open/close 3405 (time 1787873360.91 total 50.06 last 65.10) - open/close 4001 (time 1787873370.93 total 60.07 last 59.52) total: 4500 open/close in 69.21 seconds: 65.02 ops/second - /unlink 1128 (time 1787873395.42 total 10.01 last 112.74) - /unlink 2355 (time 1787873405.43 total 20.01 last 122.66) - /unlink 3642 (time 1787873415.44 total 30.02 last 128.58) total: 4500 /unlink in 36.67 seconds: 122.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 738 (time 1787873436.99 total 10.01 last 73.72) - open/close 1437 (time 1787873446.99 total 20.01 last 69.88) - open/close 2087 (time 1787873457.00 total 30.02 last 64.94) - open/close 2818 (time 1787873467.01 total 40.03 last 73.04) - open/close 3528 (time 1787873477.02 total 50.04 last 70.93) - open/close 4238 (time 1787873487.03 total 60.05 last 70.93) total: 4500 open/close in 63.75 seconds: 70.58 ops/second - /unlink 1335 (time 1787873505.53 total 10.01 last 133.37) - /unlink 2575 (time 1787873515.54 total 20.02 last 123.88) - /unlink 3962 (time 1787873525.54 total 30.02 last 138.69) total: 4500 /unlink in 33.78 seconds: 133.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 685 (time 1787873543.84 total 10.00 last 68.47) - open/close 1372 (time 1787873553.85 total 20.02 last 68.61) - open/close 2121 (time 1787873563.86 total 30.03 last 74.84) - open/close 2830 (time 1787873573.87 total 40.03 last 70.84) - open/close 3486 (time 1787873583.90 total 50.06 last 65.42) - open/close 4186 (time 1787873593.90 total 60.07 last 69.98) total: 4500 open/close in 65.52 seconds: 68.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1211 (time 1787873613.96 total 10.00 last 121.09) - /unlink 2448 (time 1787873623.96 total 20.00 last 123.69) - /unlink 3738 (time 1787873633.96 total 30.01 last 128.89) total: 4500 /unlink in 35.98 seconds: 125.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 651 (time 1787873654.25 total 10.00 last 65.08) - open/close 1343 (time 1787873664.25 total 20.01 last 69.18) - open/close 2017 (time 1787873674.27 total 30.02 last 67.29) - open/close 2760 (time 1787873684.28 total 40.03 last 74.25) - open/close 3465 (time 1787873694.28 total 50.03 last 70.48) - open/close 4200 (time 1787873704.28 total 60.04 last 73.47) total: 4500 open/close in 64.50 seconds: 69.77 ops/second - /unlink 1237 (time 1787873723.18 total 10.00 last 123.67) - /unlink 2571 (time 1787873733.19 total 20.01 last 133.35) - /unlink 3896 (time 1787873743.19 total 30.01 last 132.46) total: 4500 /unlink in 35.35 seconds: 127.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 664 (time 1787873763.31 total 10.01 last 66.34) - open/close 1406 (time 1787873773.32 total 20.02 last 74.12) - open/close 2146 (time 1787873783.33 total 30.03 last 73.94) - open/close 2918 (time 1787873793.33 total 40.03 last 77.17) - open/close 3607 (time 1787873803.34 total 50.04 last 68.83) - open/close 4289 (time 1787873813.35 total 60.05 last 68.13) total: 4500 open/close in 64.24 seconds: 70.05 ops/second - /unlink 1327 (time 1787873832.38 total 10.00 last 132.69) - /unlink 2601 (time 1787873842.38 total 20.00 last 127.40) - /unlink 3837 (time 1787873852.38 total 30.00 last 123.59) total: 4500 /unlink in 35.77 seconds: 125.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 626 (time 1787873873.13 total 10.00 last 62.58) - open/close 1291 (time 1787873883.15 total 20.02 last 66.37) - open/close 1921 (time 1787873893.16 total 30.02 last 63.00) - open/close 2627 (time 1787873903.19 total 40.05 last 70.38) - open/close 3276 (time 1787873913.19 total 50.06 last 64.88) - open/close 3968 (time 1787873923.19 total 60.06 last 69.19) total: 4500 open/close in 68.08 seconds: 66.10 ops/second - /unlink 1475 (time 1787873946.02 total 10.00 last 147.44) - /unlink 2789 (time 1787873956.03 total 20.02 last 131.23) - /unlink 4081 (time 1787873966.05 total 30.03 last 129.01) total: 4500 /unlink in 33.52 seconds: 134.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 623 (time 1787873984.84 total 10.01 last 62.26) - open/close 1273 (time 1787873994.85 total 20.02 last 64.94) - open/close 1960 (time 1787874004.85 total 30.02 last 68.66) - open/close 2739 (time 1787874014.86 total 40.03 last 77.84) - open/close 3480 (time 1787874024.87 total 50.04 last 74.00) - open/close 4224 (time 1787874034.92 total 60.09 last 74.05) total: 4500 open/close in 63.82 seconds: 70.52 ops/second - /unlink 1239 (time 1787874053.31 total 10.00 last 123.88) - /unlink 2504 (time 1787874063.32 total 20.01 last 126.45) - /unlink 3747 (time 1787874073.32 total 30.01 last 124.30) total: 4500 /unlink in 36.16 seconds: 124.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 702 (time 1787874093.42 total 10.01 last 70.13) - open/close 1349 (time 1787874103.45 total 20.03 last 64.54) - open/close 2082 (time 1787874113.45 total 30.04 last 73.27) - open/close 2754 (time 1787874123.48 total 40.06 last 67.05) - open/close 3421 (time 1787874133.49 total 50.07 last 66.62) - open/close 4093 (time 1787874143.49 total 60.08 last 67.17) total: 4500 open/close in 66.50 seconds: 67.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1258 (time 1787874164.01 total 10.01 last 125.71) - /unlink 2600 (time 1787874174.02 total 20.01 last 134.19) - /unlink 3980 (time 1787874184.02 total 30.01 last 137.96) total: 4500 /unlink in 33.77 seconds: 133.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 704 (time 1787874202.69 total 10.01 last 70.31) - open/close 1428 (time 1787874212.70 total 20.02 last 72.37) - open/close 2118 (time 1787874222.70 total 30.02 last 68.99) - open/close 2798 (time 1787874232.71 total 40.03 last 67.94) - open/close 3495 (time 1787874242.71 total 50.03 last 69.67) - open/close 4201 (time 1787874252.72 total 60.04 last 70.52) total: 4500 open/close in 64.20 seconds: 70.09 ops/second - /unlink 1253 (time 1787874272.41 total 10.01 last 125.23) - /unlink 2381 (time 1787874282.42 total 20.01 last 112.72) - /unlink 3655 (time 1787874292.42 total 30.02 last 127.36) total: 4500 /unlink in 36.81 seconds: 122.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 675 (time 1787874313.37 total 10.01 last 67.46) - open/close 1405 (time 1787874323.39 total 20.02 last 72.87) - open/close 2139 (time 1787874333.39 total 30.03 last 73.38) - open/close 2830 (time 1787874343.40 total 40.04 last 69.04) - open/close 3557 (time 1787874353.42 total 50.05 last 72.59) - open/close 4293 (time 1787874363.44 total 60.07 last 73.46) total: 4500 open/close in 63.04 seconds: 71.38 ops/second - /unlink 1314 (time 1787874380.48 total 10.01 last 131.32) - /unlink 2668 (time 1787874390.48 total 20.01 last 135.36) - /unlink 3893 (time 1787874400.49 total 30.02 last 122.39) total: 4500 /unlink in 34.76 seconds: 129.46 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11375: 95291 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg320-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg320-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg320-server stop mds service on oleg320-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg320-server unloading modules via unload_modules_local on: 'oleg320-server' oleg320-server: oleg320-server.virtnet: executing unload_modules_local oleg320-server: modules unloaded.