== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:17:03 (1788445023) Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg347-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg347-server oleg347-server: oleg347-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg347-server: detected 4 online CPUs by sysfs oleg347-server: MODOPTS_LIBCFS= oleg347-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 626 (time 1788445239.94 total 10.00 last 62.58) - open/close 1329 (time 1788445249.95 total 20.02 last 70.21) - open/close 1963 (time 1788445259.97 total 30.03 last 63.31) - open/close 2649 (time 1788445269.97 total 40.03 last 68.57) - open/close 3281 (time 1788445279.98 total 50.04 last 63.18) - open/close 3887 (time 1788445289.99 total 60.05 last 60.53) - open/close 4481 (time 1788445300.00 total 70.06 last 59.35) total: 4500 open/close in 70.27 seconds: 64.04 ops/second - /unlink 1227 (time 1788445314.69 total 10.00 last 122.68) - /unlink 2406 (time 1788445324.69 total 20.00 last 117.88) - /unlink 3555 (time 1788445334.70 total 30.01 last 114.86) total: 4500 /unlink in 38.25 seconds: 117.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 647 (time 1788445358.21 total 10.01 last 64.64) - open/close 1224 (time 1788445368.23 total 20.02 last 57.62) - open/close 1745 (time 1788445378.23 total 30.03 last 52.08) - open/close 2468 (time 1788445388.25 total 40.04 last 72.21) - open/close 3162 (time 1788445398.25 total 50.04 last 69.40) - open/close 3853 (time 1788445408.25 total 60.04 last 69.09) - open/close 4389 (time 1788445418.28 total 70.07 last 53.45) total: 4500 open/close in 71.90 seconds: 62.59 ops/second - /unlink 1232 (time 1788445435.01 total 10.00 last 123.16) - /unlink 2407 (time 1788445445.01 total 20.01 last 117.46) - /unlink 3552 (time 1788445455.02 total 30.01 last 114.47) total: 4500 /unlink in 37.94 seconds: 118.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 546 (time 1788445478.26 total 10.01 last 54.56) - open/close 1053 (time 1788445488.28 total 20.03 last 50.60) - open/close 1573 (time 1788445498.29 total 30.03 last 51.96) - open/close 2267 (time 1788445508.29 total 40.04 last 69.34) - open/close 2955 (time 1788445518.30 total 50.05 last 68.76) - open/close 3621 (time 1788445528.33 total 60.08 last 66.40) - open/close 4330 (time 1788445538.35 total 70.10 last 70.74) total: 4500 open/close in 72.36 seconds: 62.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1294 (time 1788445555.95 total 10.00 last 129.38) - /unlink 2475 (time 1788445565.97 total 20.02 last 117.85) - /unlink 3523 (time 1788445575.98 total 30.03 last 104.74) - /unlink 4402 (time 1788445585.98 total 40.03 last 87.87) total: 4500 /unlink in 41.04 seconds: 109.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 426 (time 1788445605.61 total 10.01 last 42.57) - open/close 934 (time 1788445615.62 total 20.01 last 50.78) - open/close 1558 (time 1788445625.62 total 30.02 last 62.36) - open/close 2169 (time 1788445635.62 total 40.02 last 61.09) - open/close 2804 (time 1788445645.65 total 50.04 last 63.36) - open/close 3420 (time 1788445655.66 total 60.05 last 61.51) - open/close 4029 (time 1788445665.67 total 70.06 last 60.85) total: 4500 open/close in 76.64 seconds: 58.72 ops/second - /unlink 1135 (time 1788445687.90 total 10.01 last 113.44) - /unlink 2246 (time 1788445697.90 total 20.01 last 111.07) - /unlink 3414 (time 1788445707.90 total 30.01 last 116.76) total: 4500 /unlink in 38.70 seconds: 116.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 596 (time 1788445732.36 total 10.01 last 59.57) - open/close 1195 (time 1788445742.36 total 20.01 last 59.89) - open/close 1939 (time 1788445752.38 total 30.03 last 74.25) - open/close 2627 (time 1788445762.39 total 40.03 last 68.75) - open/close 3322 (time 1788445772.41 total 50.05 last 69.38) - open/close 3931 (time 1788445782.43 total 60.07 last 60.76) total: 4500 open/close in 68.30 seconds: 65.88 ops/second - /unlink 1130 (time 1788445806.37 total 10.00 last 112.95) - /unlink 2306 (time 1788445816.37 total 20.01 last 117.58) - /unlink 3440 (time 1788445826.38 total 30.01 last 113.31) total: 4500 /unlink in 39.14 seconds: 114.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 628 (time 1788445851.15 total 10.01 last 62.73) - open/close 1279 (time 1788445861.15 total 20.01 last 65.09) - open/close 1963 (time 1788445871.16 total 30.01 last 68.39) - open/close 2674 (time 1788445881.16 total 40.01 last 71.09) - open/close 3411 (time 1788445891.17 total 50.02 last 73.62) - open/close 4107 (time 1788445901.17 total 60.03 last 69.58) total: 4500 open/close in 67.34 seconds: 66.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1011 (time 1788445924.91 total 10.00 last 101.07) - /unlink 2201 (time 1788445934.92 total 20.01 last 118.94) - /unlink 3324 (time 1788445944.92 total 30.01 last 112.23) total: 4500 /unlink in 39.43 seconds: 114.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 666 (time 1788445969.07 total 10.02 last 66.49) - open/close 1336 (time 1788445979.09 total 20.04 last 66.85) - open/close 2018 (time 1788445989.10 total 30.05 last 68.12) - open/close 2702 (time 1788445999.11 total 40.06 last 68.31) - open/close 3243 (time 1788446009.13 total 50.08 last 54.02) - open/close 3786 (time 1788446019.16 total 60.11 last 54.13) - open/close 4410 (time 1788446029.18 total 70.13 last 62.28) total: 4500 open/close in 71.45 seconds: 62.98 ops/second - /unlink 991 (time 1788446045.46 total 10.02 last 98.95) - /unlink 2043 (time 1788446055.46 total 20.02 last 105.18) - /unlink 3098 (time 1788446065.46 total 30.02 last 105.47) - /unlink 4256 (time 1788446075.47 total 40.02 last 115.75) total: 4500 /unlink in 42.89 seconds: 104.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 446 (time 1788446095.17 total 10.02 last 44.49) - open/close 1042 (time 1788446105.18 total 20.04 last 59.52) - open/close 1695 (time 1788446115.18 total 30.04 last 65.29) - open/close 2344 (time 1788446125.19 total 40.05 last 64.84) - open/close 3018 (time 1788446135.20 total 50.06 last 67.34) - open/close 3694 (time 1788446145.22 total 60.07 last 67.49) - open/close 4333 (time 1788446155.22 total 70.08 last 63.87) total: 4500 open/close in 72.86 seconds: 61.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1109 (time 1788446173.28 total 10.01 last 110.83) - /unlink 2213 (time 1788446183.28 total 20.01 last 110.40) - /unlink 3317 (time 1788446193.29 total 30.01 last 110.36) total: 4500 /unlink in 39.71 seconds: 113.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 613 (time 1788446218.93 total 10.00 last 61.30) - open/close 1245 (time 1788446228.95 total 20.02 last 63.09) - open/close 1922 (time 1788446238.96 total 30.03 last 67.61) - open/close 2592 (time 1788446248.98 total 40.05 last 66.88) - open/close 3230 (time 1788446258.98 total 50.05 last 63.79) - open/close 3870 (time 1788446269.01 total 60.08 last 63.82) - open/close 4498 (time 1788446279.01 total 70.08 last 62.79) total: 4500 open/close in 70.10 seconds: 64.19 ops/second - /unlink 1223 (time 1788446293.73 total 10.00 last 122.24) - /unlink 2450 (time 1788446303.73 total 20.01 last 122.67) - /unlink 3684 (time 1788446313.73 total 30.01 last 123.40) total: 4500 /unlink in 36.74 seconds: 122.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 713 (time 1788446335.90 total 10.02 last 71.19) - open/close 1399 (time 1788446345.90 total 20.02 last 68.56) - open/close 2037 (time 1788446355.92 total 30.04 last 63.68) - open/close 2731 (time 1788446365.93 total 40.05 last 69.32) - open/close 3359 (time 1788446375.94 total 50.06 last 62.74) - open/close 3967 (time 1788446385.95 total 60.06 last 60.77) total: 4500 open/close in 68.74 seconds: 65.47 ops/second - /unlink 1323 (time 1788446409.66 total 10.00 last 132.26) - /unlink 2554 (time 1788446419.66 total 20.01 last 123.06) - /unlink 3758 (time 1788446429.68 total 30.02 last 120.17) total: 4500 /unlink in 35.89 seconds: 125.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 668 (time 1788446449.98 total 10.00 last 66.77) - open/close 1369 (time 1788446459.98 total 20.01 last 70.07) - open/close 2029 (time 1788446469.99 total 30.01 last 65.97) - open/close 2737 (time 1788446479.99 total 40.02 last 70.77) - open/close 3440 (time 1788446490.01 total 50.03 last 70.21) - open/close 4114 (time 1788446500.01 total 60.03 last 67.40) total: 4500 open/close in 65.40 seconds: 68.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1220 (time 1788446519.56 total 10.00 last 122.00) - /unlink 2395 (time 1788446529.56 total 20.01 last 117.42) - /unlink 3617 (time 1788446539.56 total 30.01 last 122.19) total: 4500 /unlink in 37.37 seconds: 120.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 709 (time 1788446561.16 total 10.01 last 70.80) - open/close 1352 (time 1788446571.18 total 20.03 last 64.21) - open/close 2012 (time 1788446581.18 total 30.03 last 65.99) - open/close 2734 (time 1788446591.19 total 40.04 last 72.12) - open/close 3455 (time 1788446601.19 total 50.04 last 72.08) - open/close 4125 (time 1788446611.20 total 60.05 last 66.96) total: 4500 open/close in 64.91 seconds: 69.33 ops/second - /unlink 1207 (time 1788446630.32 total 10.02 last 120.49) - /unlink 2439 (time 1788446640.32 total 20.02 last 123.20) - /unlink 3702 (time 1788446650.33 total 30.02 last 126.24) total: 4500 /unlink in 36.16 seconds: 124.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 675 (time 1788446670.11 total 10.01 last 67.43) - open/close 1339 (time 1788446680.13 total 20.03 last 66.31) - open/close 1984 (time 1788446690.13 total 30.03 last 64.46) - open/close 2619 (time 1788446700.13 total 40.04 last 63.48) - open/close 3253 (time 1788446710.14 total 50.04 last 63.36) - open/close 3855 (time 1788446720.15 total 60.05 last 60.13) total: 4500 open/close in 69.74 seconds: 64.52 ops/second - /unlink 1343 (time 1788446743.84 total 10.01 last 134.21) - /unlink 2682 (time 1788446753.85 total 20.01 last 133.86) - /unlink 3969 (time 1788446763.85 total 30.01 last 128.69) total: 4500 /unlink in 34.11 seconds: 131.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 714 (time 1788446782.34 total 10.01 last 71.36) - open/close 1472 (time 1788446792.34 total 20.01 last 75.79) - open/close 2194 (time 1788446802.35 total 30.02 last 72.12) - open/close 2872 (time 1788446812.36 total 40.03 last 67.73) - open/close 3528 (time 1788446822.38 total 50.04 last 65.51) - open/close 4215 (time 1788446832.38 total 60.04 last 68.69) total: 4500 open/close in 64.50 seconds: 69.77 ops/second - /unlink 1238 (time 1788446851.66 total 10.00 last 123.76) - /unlink 2451 (time 1788446861.66 total 20.00 last 121.30) - /unlink 3690 (time 1788446871.66 total 30.00 last 123.90) total: 4500 /unlink in 36.83 seconds: 122.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 666 (time 1788446893.93 total 10.00 last 66.59) - open/close 1324 (time 1788446903.93 total 20.00 last 65.79) - open/close 1938 (time 1788446913.94 total 30.01 last 61.36) - open/close 2628 (time 1788446923.94 total 40.01 last 68.98) - open/close 3319 (time 1788446933.94 total 50.02 last 69.08) - open/close 4022 (time 1788446943.95 total 60.02 last 70.28) total: 4500 open/close in 66.17 seconds: 68.00 ops/second - /unlink 1312 (time 1788446963.87 total 10.00 last 131.15) - /unlink 2486 (time 1788446973.88 total 20.01 last 117.34) - /unlink 3800 (time 1788446983.88 total 30.02 last 131.29) total: 4500 /unlink in 35.62 seconds: 126.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 671 (time 1788447004.24 total 10.02 last 66.97) - open/close 1341 (time 1788447014.24 total 20.02 last 67.00) - open/close 2046 (time 1788447024.24 total 30.02 last 70.49) - open/close 2727 (time 1788447034.25 total 40.03 last 68.05) - open/close 3453 (time 1788447044.27 total 50.05 last 72.47) - open/close 4118 (time 1788447054.27 total 60.05 last 66.46) total: 4500 open/close in 65.48 seconds: 68.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1214 (time 1788447073.84 total 10.00 last 121.35) - /unlink 2473 (time 1788447083.85 total 20.01 last 125.79) - /unlink 3758 (time 1788447093.86 total 30.02 last 128.43) total: 4500 /unlink in 36.39 seconds: 123.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 675 (time 1788447114.25 total 10.04 last 67.25) - open/close 1313 (time 1788447124.26 total 20.05 last 63.74) - open/close 2005 (time 1788447134.27 total 30.06 last 69.09) - open/close 2634 (time 1788447144.27 total 40.06 last 62.90) - open/close 3324 (time 1788447154.31 total 50.10 last 68.78) - open/close 3967 (time 1788447164.31 total 60.10 last 64.30) total: 4500 open/close in 68.59 seconds: 65.61 ops/second - /unlink 1240 (time 1788447187.15 total 10.00 last 123.95) - /unlink 2496 (time 1788447197.15 total 20.01 last 125.58) - /unlink 3576 (time 1788447207.16 total 30.02 last 107.84) total: 4500 /unlink in 39.50 seconds: 113.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 641 (time 1788447233.32 total 10.02 last 64.00) - open/close 1234 (time 1788447243.35 total 20.04 last 59.16) - open/close 1918 (time 1788447253.36 total 30.05 last 68.33) - open/close 2588 (time 1788447263.38 total 40.07 last 66.87) - open/close 3317 (time 1788447273.38 total 50.08 last 72.85) - open/close 3963 (time 1788447283.38 total 60.08 last 64.59) total: 4500 open/close in 67.89 seconds: 66.29 ops/second - /unlink 1215 (time 1788447305.75 total 10.00 last 121.46) - /unlink 2373 (time 1788447315.75 total 20.00 last 115.78) - /unlink 3621 (time 1788447325.76 total 30.01 last 124.76) total: 4500 /unlink in 37.75 seconds: 119.20 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95265 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules via unload_modules_local on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: modules unloaded.