== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:44:57 (1788590697) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: kmemleak disabled oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 627 (time 1788590913.18 total 10.00 last 62.70) - open/close 1303 (time 1788590923.18 total 20.00 last 67.58) - open/close 1871 (time 1788590933.19 total 30.01 last 56.75) - open/close 2473 (time 1788590943.20 total 40.03 last 60.10) - open/close 3143 (time 1788590953.21 total 50.03 last 66.98) - open/close 3752 (time 1788590963.22 total 60.05 last 60.79) - open/close 4377 (time 1788590973.23 total 70.05 last 62.46) total: 4500 open/close in 72.10 seconds: 62.41 ops/second - /unlink 1249 (time 1788590989.79 total 10.00 last 124.87) - /unlink 2495 (time 1788590999.80 total 20.01 last 124.51) - /unlink 3630 (time 1788591009.81 total 30.02 last 113.34) total: 4500 /unlink in 37.76 seconds: 119.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 614 (time 1788591033.18 total 10.01 last 61.35) - open/close 1243 (time 1788591043.21 total 20.04 last 62.73) - open/close 1904 (time 1788591053.21 total 30.04 last 66.08) - open/close 2571 (time 1788591063.22 total 40.05 last 66.62) - open/close 3209 (time 1788591073.24 total 50.07 last 63.68) - open/close 3808 (time 1788591083.25 total 60.08 last 59.84) - open/close 4455 (time 1788591093.26 total 70.08 last 64.68) total: 4500 open/close in 70.75 seconds: 63.60 ops/second - /unlink 1225 (time 1788591107.83 total 10.00 last 122.45) - /unlink 2384 (time 1788591117.83 total 20.01 last 115.83) - /unlink 3444 (time 1788591127.83 total 30.01 last 105.98) total: 4500 /unlink in 39.37 seconds: 114.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 620 (time 1788591152.40 total 10.00 last 61.98) - open/close 1140 (time 1788591162.41 total 20.01 last 51.99) - open/close 1721 (time 1788591172.43 total 30.03 last 57.96) - open/close 2244 (time 1788591182.44 total 40.04 last 52.24) - open/close 2829 (time 1788591192.44 total 50.04 last 58.48) - open/close 3410 (time 1788591202.45 total 60.05 last 58.04) - open/close 4034 (time 1788591212.46 total 70.06 last 62.36) total: 4500 open/close in 77.52 seconds: 58.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1234 (time 1788591234.31 total 10.01 last 123.23) - /unlink 2478 (time 1788591244.31 total 20.02 last 124.37) - /unlink 3602 (time 1788591254.31 total 30.02 last 112.31) total: 4500 /unlink in 37.91 seconds: 118.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 677 (time 1788591277.86 total 10.00 last 67.70) - open/close 1306 (time 1788591287.86 total 20.00 last 62.88) - open/close 1948 (time 1788591297.87 total 30.01 last 64.19) - open/close 2585 (time 1788591307.87 total 40.01 last 63.66) - open/close 3219 (time 1788591317.89 total 50.03 last 63.32) - open/close 3850 (time 1788591327.89 total 60.03 last 63.09) total: 4500 open/close in 69.05 seconds: 65.17 ops/second - /unlink 1237 (time 1788591351.75 total 10.01 last 123.63) - /unlink 2488 (time 1788591361.76 total 20.01 last 125.07) - /unlink 3738 (time 1788591371.76 total 30.01 last 124.93) total: 4500 /unlink in 36.20 seconds: 124.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 666 (time 1788591392.31 total 10.03 last 66.37) - open/close 1334 (time 1788591402.33 total 20.06 last 66.66) - open/close 2006 (time 1788591412.35 total 30.07 last 67.13) - open/close 2618 (time 1788591422.35 total 40.07 last 61.20) - open/close 3216 (time 1788591432.38 total 50.10 last 59.61) - open/close 3843 (time 1788591442.41 total 60.13 last 62.51) total: 4500 open/close in 69.10 seconds: 65.13 ops/second - /unlink 1232 (time 1788591466.13 total 10.01 last 123.06) - /unlink 2467 (time 1788591476.13 total 20.01 last 123.48) - /unlink 3637 (time 1788591486.13 total 30.02 last 116.98) total: 4500 /unlink in 37.49 seconds: 120.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 681 (time 1788591508.79 total 10.01 last 68.06) - open/close 1330 (time 1788591518.80 total 20.01 last 64.87) - open/close 1993 (time 1788591528.80 total 30.01 last 66.28) - open/close 2645 (time 1788591538.81 total 40.02 last 65.14) - open/close 3295 (time 1788591548.81 total 50.03 last 64.97) - open/close 3918 (time 1788591558.81 total 60.03 last 62.30) total: 4500 open/close in 69.19 seconds: 65.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1216 (time 1788591583.00 total 10.01 last 121.52) - /unlink 2458 (time 1788591593.00 total 20.01 last 124.20) - /unlink 3650 (time 1788591603.01 total 30.01 last 119.11) total: 4500 /unlink in 36.59 seconds: 122.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 699 (time 1788591624.51 total 10.01 last 69.81) - open/close 1335 (time 1788591634.52 total 20.03 last 63.48) - open/close 1881 (time 1788591644.54 total 30.05 last 54.52) - open/close 2488 (time 1788591654.54 total 40.05 last 60.67) - open/close 3157 (time 1788591664.56 total 50.06 last 66.82) - open/close 3761 (time 1788591674.57 total 60.08 last 60.31) - open/close 4411 (time 1788591684.57 total 70.08 last 64.99) total: 4500 open/close in 71.33 seconds: 63.09 ops/second - /unlink 1175 (time 1788591700.57 total 10.00 last 117.46) - /unlink 2386 (time 1788591710.57 total 20.00 last 121.09) - /unlink 3616 (time 1788591720.57 total 30.01 last 122.98) total: 4500 /unlink in 38.49 seconds: 116.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 486 (time 1788591745.28 total 10.02 last 48.52) - open/close 1055 (time 1788591755.29 total 20.02 last 56.86) - open/close 1674 (time 1788591765.30 total 30.04 last 61.80) - open/close 2356 (time 1788591775.32 total 40.05 last 68.09) - open/close 2947 (time 1788591785.32 total 50.06 last 59.08) - open/close 3453 (time 1788591795.35 total 60.09 last 50.43) - open/close 4054 (time 1788591805.37 total 70.10 last 60.02) total: 4500 open/close in 77.22 seconds: 58.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1155 (time 1788591827.28 total 10.00 last 115.44) - /unlink 2352 (time 1788591837.28 total 20.01 last 119.68) - /unlink 3427 (time 1788591847.28 total 30.01 last 107.48) total: 4500 /unlink in 39.45 seconds: 114.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 666 (time 1788591871.67 total 10.01 last 66.55) - open/close 1303 (time 1788591881.68 total 20.01 last 63.68) - open/close 1927 (time 1788591891.68 total 30.02 last 62.35) - open/close 2567 (time 1788591901.72 total 40.05 last 63.79) - open/close 3170 (time 1788591911.73 total 50.07 last 60.21) - open/close 3846 (time 1788591921.74 total 60.07 last 67.55) total: 4500 open/close in 70.05 seconds: 64.24 ops/second - /unlink 1190 (time 1788591946.69 total 10.00 last 119.00) - /unlink 2357 (time 1788591956.71 total 20.02 last 116.48) - /unlink 3516 (time 1788591966.71 total 30.02 last 115.85) total: 4500 /unlink in 38.91 seconds: 115.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 634 (time 1788591991.17 total 10.00 last 63.38) - open/close 1299 (time 1788592001.18 total 20.01 last 66.44) - open/close 1925 (time 1788592011.20 total 30.03 last 62.52) - open/close 2512 (time 1788592021.23 total 40.06 last 58.51) - open/close 3123 (time 1788592031.25 total 50.08 last 60.96) - open/close 3731 (time 1788592041.27 total 60.10 last 60.68) - open/close 4328 (time 1788592051.29 total 70.12 last 59.60) total: 4500 open/close in 72.84 seconds: 61.78 ops/second - /unlink 1164 (time 1788592068.38 total 10.00 last 116.39) - /unlink 2460 (time 1788592078.38 total 20.00 last 129.60) - /unlink 3646 (time 1788592088.40 total 30.02 last 118.40) total: 4500 /unlink in 37.27 seconds: 120.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 635 (time 1788592109.65 total 10.01 last 63.47) - open/close 1264 (time 1788592119.66 total 20.01 last 62.86) - open/close 1917 (time 1788592129.66 total 30.02 last 65.26) - open/close 2607 (time 1788592139.67 total 40.02 last 68.98) - open/close 3260 (time 1788592149.67 total 50.03 last 65.25) - open/close 3885 (time 1788592159.68 total 60.03 last 62.48) - open/close 4478 (time 1788592169.69 total 70.05 last 59.20) total: 4500 open/close in 70.37 seconds: 63.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1288 (time 1788592184.14 total 10.00 last 128.76) - /unlink 2463 (time 1788592194.15 total 20.00 last 117.48) - /unlink 3677 (time 1788592204.16 total 30.01 last 121.28) total: 4500 /unlink in 37.28 seconds: 120.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 566 (time 1788592226.41 total 10.00 last 56.58) - open/close 1107 (time 1788592236.42 total 20.01 last 54.04) - open/close 1688 (time 1788592246.43 total 30.03 last 58.04) - open/close 2331 (time 1788592256.44 total 40.03 last 64.27) - open/close 2944 (time 1788592266.45 total 50.05 last 61.20) - open/close 3606 (time 1788592276.46 total 60.06 last 66.13) - open/close 4252 (time 1788592286.46 total 70.06 last 64.59) total: 4500 open/close in 73.68 seconds: 61.07 ops/second - /unlink 1135 (time 1788592304.69 total 10.00 last 113.50) - /unlink 2243 (time 1788592314.70 total 20.01 last 110.74) - /unlink 3450 (time 1788592324.70 total 30.01 last 120.64) total: 4500 /unlink in 39.17 seconds: 114.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 669 (time 1788592349.16 total 10.01 last 66.85) - open/close 1308 (time 1788592359.17 total 20.02 last 63.81) - open/close 1957 (time 1788592369.18 total 30.03 last 64.86) - open/close 2586 (time 1788592379.18 total 40.03 last 62.90) - open/close 3215 (time 1788592389.20 total 50.05 last 62.79) - open/close 3887 (time 1788592399.20 total 60.05 last 67.19) total: 4500 open/close in 68.88 seconds: 65.33 ops/second - /unlink 1402 (time 1788592422.33 total 10.01 last 140.13) - /unlink 2574 (time 1788592432.33 total 20.01 last 117.17) - /unlink 3838 (time 1788592442.33 total 30.01 last 126.36) total: 4500 /unlink in 35.29 seconds: 127.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 661 (time 1788592462.75 total 10.02 last 65.99) - open/close 1305 (time 1788592472.76 total 20.02 last 64.37) - open/close 1930 (time 1788592482.77 total 30.03 last 62.45) - open/close 2587 (time 1788592492.78 total 40.04 last 65.63) - open/close 3238 (time 1788592502.79 total 50.05 last 65.04) - open/close 3874 (time 1788592512.79 total 60.05 last 63.58) - open/close 4460 (time 1788592522.81 total 70.07 last 58.48) total: 4500 open/close in 70.95 seconds: 63.43 ops/second - /unlink 1151 (time 1788592539.83 total 10.00 last 115.09) - /unlink 2364 (time 1788592549.83 total 20.00 last 121.30) - /unlink 3531 (time 1788592559.83 total 30.01 last 116.63) total: 4500 /unlink in 38.23 seconds: 117.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 671 (time 1788592582.95 total 10.01 last 67.02) - open/close 1321 (time 1788592592.96 total 20.03 last 64.91) - open/close 2011 (time 1788592602.98 total 30.04 last 68.91) - open/close 2642 (time 1788592612.99 total 40.05 last 63.02) - open/close 3289 (time 1788592623.00 total 50.06 last 64.63) - open/close 3909 (time 1788592633.01 total 60.08 last 61.92) total: 4500 open/close in 68.45 seconds: 65.74 ops/second - /unlink 1111 (time 1788592655.74 total 10.01 last 111.04) - /unlink 2307 (time 1788592665.75 total 20.01 last 119.52) - /unlink 3580 (time 1788592675.75 total 30.01 last 127.28) total: 4500 /unlink in 38.48 seconds: 116.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 553 (time 1788592699.49 total 10.01 last 55.26) - open/close 1125 (time 1788592709.51 total 20.02 last 57.11) - open/close 1692 (time 1788592719.52 total 30.03 last 56.63) - open/close 2228 (time 1788592729.52 total 40.04 last 53.58) - open/close 2831 (time 1788592739.53 total 50.05 last 60.26) - open/close 3499 (time 1788592749.53 total 60.05 last 66.78) - open/close 4052 (time 1788592759.54 total 70.06 last 55.26) total: 4500 open/close in 77.19 seconds: 58.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1135 (time 1788592782.18 total 10.00 last 113.49) - /unlink 2213 (time 1788592792.18 total 20.01 last 107.75) - /unlink 3413 (time 1788592802.19 total 30.01 last 119.94) total: 4500 /unlink in 39.16 seconds: 114.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 636 (time 1788592826.54 total 10.01 last 63.55) - open/close 1344 (time 1788592836.55 total 20.01 last 70.76) - open/close 1982 (time 1788592846.55 total 30.02 last 63.78) - open/close 2678 (time 1788592856.56 total 40.03 last 69.53) - open/close 3316 (time 1788592866.57 total 50.03 last 63.75) - open/close 4034 (time 1788592876.57 total 60.03 last 71.80) total: 4500 open/close in 67.79 seconds: 66.38 ops/second - /unlink 1205 (time 1788592899.90 total 10.00 last 120.47) - /unlink 2397 (time 1788592909.90 total 20.01 last 119.12) - /unlink 3669 (time 1788592919.91 total 30.01 last 127.18) total: 4500 /unlink in 36.84 seconds: 122.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 699 (time 1788592942.06 total 10.00 last 69.88) - open/close 1335 (time 1788592952.08 total 20.02 last 63.48) - open/close 1947 (time 1788592962.08 total 30.02 last 61.20) - open/close 2554 (time 1788592972.08 total 40.02 last 60.69) - open/close 3229 (time 1788592982.09 total 50.03 last 67.46) - open/close 3874 (time 1788592992.09 total 60.04 last 64.45) total: 4500 open/close in 69.51 seconds: 64.74 ops/second - /unlink 1270 (time 1788593017.36 total 10.00 last 126.98) - /unlink 2549 (time 1788593027.38 total 20.02 last 127.70) - /unlink 3789 (time 1788593037.38 total 30.02 last 123.95) total: 4500 /unlink in 36.76 seconds: 122.43 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95438 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: kmemleak disabled modules unloaded.