== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:41:11 (1788662471) Stopping clients: oleg442-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg442-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg442-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg442-server oleg442-server: oleg442-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: kmemleak disabled oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg442-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg442-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 639 (time 1788662684.90 total 10.01 last 63.85) - open/close 1285 (time 1788662694.90 total 20.01 last 64.56) - open/close 1926 (time 1788662704.91 total 30.02 last 64.09) - open/close 2468 (time 1788662714.92 total 40.03 last 54.15) - open/close 3042 (time 1788662724.92 total 50.03 last 57.36) - open/close 3613 (time 1788662734.93 total 60.04 last 57.07) - open/close 4304 (time 1788662744.94 total 70.05 last 69.04) total: 4500 open/close in 72.83 seconds: 61.79 ops/second - /unlink 1267 (time 1788662762.27 total 10.00 last 126.69) - /unlink 2413 (time 1788662772.27 total 20.00 last 114.60) - /unlink 3567 (time 1788662782.27 total 30.00 last 115.38) total: 4500 /unlink in 38.00 seconds: 118.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 670 (time 1788662805.55 total 10.01 last 66.96) - open/close 1285 (time 1788662815.55 total 20.01 last 61.47) - open/close 1856 (time 1788662825.55 total 30.01 last 57.10) - open/close 2495 (time 1788662835.56 total 40.02 last 63.84) - open/close 3132 (time 1788662845.57 total 50.02 last 63.68) - open/close 3681 (time 1788662855.57 total 60.03 last 54.86) - open/close 4336 (time 1788662865.58 total 70.03 last 65.48) total: 4500 open/close in 72.72 seconds: 61.88 ops/second - /unlink 1262 (time 1788662882.61 total 10.00 last 126.14) - /unlink 2393 (time 1788662892.62 total 20.01 last 113.01) - /unlink 3614 (time 1788662902.62 total 30.01 last 122.09) total: 4500 /unlink in 37.50 seconds: 120.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 683 (time 1788662925.20 total 10.01 last 68.25) - open/close 1325 (time 1788662935.22 total 20.03 last 64.06) - open/close 2016 (time 1788662945.23 total 30.03 last 69.07) - open/close 2648 (time 1788662955.23 total 40.03 last 63.20) - open/close 3316 (time 1788662965.25 total 50.05 last 66.67) - open/close 3955 (time 1788662975.26 total 60.06 last 63.84) total: 4500 open/close in 68.44 seconds: 65.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1322 (time 1788662999.04 total 10.00 last 132.14) - /unlink 2605 (time 1788663009.04 total 20.01 last 128.26) - /unlink 3753 (time 1788663019.05 total 30.01 last 114.73) total: 4500 /unlink in 36.26 seconds: 124.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 660 (time 1788663040.46 total 10.03 last 65.80) - open/close 1276 (time 1788663050.48 total 20.05 last 61.47) - open/close 1922 (time 1788663060.50 total 30.07 last 64.49) - open/close 2573 (time 1788663070.51 total 40.08 last 65.04) - open/close 3267 (time 1788663080.52 total 50.08 last 69.37) - open/close 3883 (time 1788663090.52 total 60.09 last 61.56) total: 4500 open/close in 69.71 seconds: 64.56 ops/second - /unlink 1177 (time 1788663115.29 total 10.00 last 117.67) - /unlink 2239 (time 1788663125.29 total 20.00 last 106.18) - /unlink 3423 (time 1788663135.29 total 30.01 last 118.36) total: 4500 /unlink in 39.00 seconds: 115.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 684 (time 1788663159.64 total 10.00 last 68.37) - open/close 1280 (time 1788663169.65 total 20.02 last 59.53) - open/close 1898 (time 1788663179.66 total 30.02 last 61.76) - open/close 2514 (time 1788663189.67 total 40.03 last 61.55) - open/close 3205 (time 1788663199.68 total 50.05 last 69.00) - open/close 3846 (time 1788663209.68 total 60.05 last 64.09) - open/close 4386 (time 1788663219.69 total 70.06 last 53.94) total: 4500 open/close in 71.94 seconds: 62.55 ops/second - /unlink 1222 (time 1788663237.17 total 10.01 last 122.08) - /unlink 2365 (time 1788663247.17 total 20.01 last 114.27) - /unlink 3510 (time 1788663257.18 total 30.02 last 114.41) total: 4500 /unlink in 38.08 seconds: 118.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 637 (time 1788663281.03 total 10.01 last 63.65) - open/close 1324 (time 1788663291.04 total 20.02 last 68.64) - open/close 2002 (time 1788663301.05 total 30.02 last 67.75) - open/close 2683 (time 1788663311.06 total 40.04 last 68.01) - open/close 3411 (time 1788663321.08 total 50.05 last 72.69) - open/close 4066 (time 1788663331.10 total 60.07 last 65.38) total: 4500 open/close in 66.65 seconds: 67.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1223 (time 1788663352.43 total 10.00 last 122.26) - /unlink 2527 (time 1788663362.43 total 20.01 last 130.37) - /unlink 3813 (time 1788663372.44 total 30.01 last 128.59) total: 4500 /unlink in 34.95 seconds: 128.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 696 (time 1788663392.24 total 10.02 last 69.43) - open/close 1261 (time 1788663402.25 total 20.03 last 56.48) - open/close 1940 (time 1788663412.27 total 30.05 last 67.77) - open/close 2602 (time 1788663422.28 total 40.06 last 66.15) - open/close 3306 (time 1788663432.28 total 50.06 last 70.34) - open/close 3943 (time 1788663442.29 total 60.07 last 63.68) total: 4500 open/close in 67.70 seconds: 66.47 ops/second - /unlink 1334 (time 1788663464.70 total 10.01 last 133.29) - /unlink 2505 (time 1788663474.71 total 20.01 last 117.04) - /unlink 3714 (time 1788663484.71 total 30.02 last 120.88) total: 4500 /unlink in 36.81 seconds: 122.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 677 (time 1788663506.17 total 10.02 last 67.58) - open/close 1357 (time 1788663516.17 total 20.02 last 68.00) - open/close 2066 (time 1788663526.17 total 30.03 last 70.84) - open/close 2780 (time 1788663536.17 total 40.03 last 71.40) - open/close 3430 (time 1788663546.18 total 50.03 last 64.96) - open/close 4176 (time 1788663556.18 total 60.03 last 74.60) total: 4500 open/close in 64.76 seconds: 69.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1201 (time 1788663575.56 total 10.01 last 120.03) - /unlink 2422 (time 1788663585.56 total 20.01 last 122.09) - /unlink 3530 (time 1788663595.57 total 30.01 last 110.72) total: 4500 /unlink in 38.55 seconds: 116.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 652 (time 1788663619.89 total 10.02 last 65.07) - open/close 1293 (time 1788663629.90 total 20.03 last 64.05) - open/close 1936 (time 1788663639.90 total 30.03 last 64.28) - open/close 2569 (time 1788663649.90 total 40.03 last 63.29) - open/close 3221 (time 1788663659.92 total 50.05 last 65.11) - open/close 3845 (time 1788663669.93 total 60.06 last 62.34) - open/close 4488 (time 1788663679.93 total 70.06 last 64.26) total: 4500 open/close in 70.18 seconds: 64.12 ops/second - /unlink 1230 (time 1788663694.69 total 10.00 last 122.99) - /unlink 2357 (time 1788663704.69 total 20.00 last 112.68) - /unlink 3464 (time 1788663714.72 total 30.02 last 110.46) - /unlink 4430 (time 1788663724.72 total 40.03 last 96.58) total: 4500 /unlink in 41.24 seconds: 109.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 592 (time 1788663743.97 total 10.01 last 59.15) - open/close 1255 (time 1788663754.01 total 20.05 last 66.03) - open/close 1958 (time 1788663764.02 total 30.05 last 70.26) - open/close 2575 (time 1788663774.03 total 40.06 last 61.64) - open/close 3190 (time 1788663784.03 total 50.07 last 61.46) - open/close 3891 (time 1788663794.04 total 60.08 last 70.05) total: 4500 open/close in 68.46 seconds: 65.73 ops/second - /unlink 1263 (time 1788663816.30 total 10.00 last 126.27) - /unlink 2508 (time 1788663826.30 total 20.01 last 124.45) - /unlink 3747 (time 1788663836.32 total 30.02 last 123.68) total: 4500 /unlink in 36.26 seconds: 124.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 650 (time 1788663858.93 total 10.01 last 64.93) - open/close 1345 (time 1788663868.93 total 20.02 last 69.44) - open/close 2010 (time 1788663878.94 total 30.02 last 66.48) - open/close 2625 (time 1788663888.95 total 40.03 last 61.45) - open/close 3282 (time 1788663898.95 total 50.04 last 65.67) - open/close 3815 (time 1788663908.95 total 60.04 last 53.29) - open/close 4320 (time 1788663918.97 total 70.06 last 50.41) total: 4500 open/close in 73.77 seconds: 61.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1307 (time 1788663938.37 total 10.00 last 130.70) - /unlink 2379 (time 1788663948.37 total 20.00 last 107.16) - /unlink 3491 (time 1788663958.38 total 30.01 last 111.16) total: 4500 /unlink in 39.52 seconds: 113.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 673 (time 1788663984.43 total 10.01 last 67.22) - open/close 1319 (time 1788663994.44 total 20.02 last 64.52) - open/close 1874 (time 1788664004.44 total 30.03 last 55.49) - open/close 2407 (time 1788664014.44 total 40.03 last 53.27) - open/close 2928 (time 1788664024.46 total 50.04 last 52.04) - open/close 3554 (time 1788664034.49 total 60.07 last 62.41) - open/close 4138 (time 1788664044.49 total 70.08 last 58.36) total: 4500 open/close in 76.59 seconds: 58.76 ops/second - /unlink 1276 (time 1788664065.93 total 10.01 last 127.51) - /unlink 2451 (time 1788664075.93 total 20.01 last 117.48) - /unlink 3662 (time 1788664085.94 total 30.02 last 120.99) total: 4500 /unlink in 37.00 seconds: 121.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 674 (time 1788664107.45 total 10.01 last 67.35) - open/close 1399 (time 1788664117.45 total 20.01 last 72.49) - open/close 2071 (time 1788664127.45 total 30.01 last 67.17) - open/close 2699 (time 1788664137.46 total 40.02 last 62.76) - open/close 3347 (time 1788664147.47 total 50.03 last 64.75) - open/close 4081 (time 1788664157.47 total 60.03 last 73.38) total: 4500 open/close in 66.87 seconds: 67.29 ops/second - /unlink 1115 (time 1788664179.48 total 10.00 last 111.46) - /unlink 2384 (time 1788664189.48 total 20.01 last 126.84) - /unlink 3697 (time 1788664199.48 total 30.01 last 131.27) total: 4500 /unlink in 36.56 seconds: 123.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 668 (time 1788664221.68 total 10.00 last 66.79) - open/close 1298 (time 1788664231.69 total 20.01 last 62.97) - open/close 1928 (time 1788664241.70 total 30.02 last 62.93) - open/close 2608 (time 1788664251.71 total 40.03 last 67.94) - open/close 3333 (time 1788664261.71 total 50.03 last 72.49) - open/close 3966 (time 1788664271.72 total 60.04 last 63.25) total: 4500 open/close in 69.92 seconds: 64.36 ops/second - /unlink 1099 (time 1788664297.60 total 10.00 last 109.88) - /unlink 2219 (time 1788664307.61 total 20.01 last 111.91) - /unlink 3458 (time 1788664317.62 total 30.02 last 123.79) total: 4500 /unlink in 38.59 seconds: 116.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 542 (time 1788664341.90 total 10.02 last 54.09) - open/close 1148 (time 1788664351.91 total 20.03 last 60.54) - open/close 1691 (time 1788664361.92 total 30.04 last 54.25) - open/close 2287 (time 1788664371.93 total 40.05 last 59.53) - open/close 2916 (time 1788664381.95 total 50.07 last 62.78) - open/close 3613 (time 1788664391.95 total 60.08 last 69.66) - open/close 4441 (time 1788664401.96 total 70.08 last 82.77) total: 4500 open/close in 70.90 seconds: 63.47 ops/second - /unlink 1091 (time 1788664417.39 total 10.00 last 109.07) - /unlink 2052 (time 1788664427.39 total 20.01 last 96.06) - /unlink 2845 (time 1788664437.40 total 30.02 last 79.22) - /unlink 3992 (time 1788664447.41 total 40.02 last 114.68) total: 4500 /unlink in 44.93 seconds: 100.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 572 (time 1788664466.96 total 10.02 last 57.10) - open/close 1169 (time 1788664476.98 total 20.04 last 59.57) - open/close 1890 (time 1788664486.99 total 30.05 last 72.02) - open/close 2481 (time 1788664497.01 total 40.06 last 59.04) - open/close 3068 (time 1788664507.01 total 50.07 last 58.67) - open/close 3688 (time 1788664517.02 total 60.07 last 61.95) - open/close 4407 (time 1788664527.03 total 70.08 last 71.84) total: 4500 open/close in 71.43 seconds: 63.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1293 (time 1788664542.77 total 10.01 last 129.21) - /unlink 2585 (time 1788664552.78 total 20.02 last 129.01) - /unlink 3802 (time 1788664562.79 total 30.02 last 121.66) total: 4500 /unlink in 35.62 seconds: 126.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 598 (time 1788664583.85 total 10.00 last 59.79) - open/close 1199 (time 1788664593.87 total 20.02 last 60.01) - open/close 1757 (time 1788664603.87 total 30.02 last 55.78) - open/close 2396 (time 1788664613.88 total 40.03 last 63.83) - open/close 2960 (time 1788664623.88 total 50.03 last 56.40) - open/close 3433 (time 1788664633.89 total 60.03 last 47.29) - open/close 3930 (time 1788664643.91 total 70.06 last 49.59) - open/close 4489 (time 1788664653.91 total 80.06 last 55.90) total: 4500 open/close in 80.23 seconds: 56.09 ops/second - /unlink 1273 (time 1788664669.65 total 10.01 last 127.16) - /unlink 2430 (time 1788664679.66 total 20.01 last 115.66) - /unlink 3619 (time 1788664689.66 total 30.02 last 118.89) total: 4500 /unlink in 36.70 seconds: 122.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 607 (time 1788664711.53 total 10.00 last 60.69) - open/close 1290 (time 1788664721.54 total 20.01 last 68.25) - open/close 1986 (time 1788664731.55 total 30.02 last 69.52) - open/close 2534 (time 1788664741.56 total 40.04 last 54.71) - open/close 3161 (time 1788664751.56 total 50.04 last 62.70) - open/close 3771 (time 1788664761.58 total 60.05 last 60.93) - open/close 4350 (time 1788664771.58 total 70.06 last 57.86) total: 4500 open/close in 72.24 seconds: 62.29 ops/second - /unlink 1351 (time 1788664788.98 total 10.00 last 135.08) - /unlink 2483 (time 1788664798.98 total 20.01 last 113.16) - /unlink 3713 (time 1788664808.98 total 30.01 last 123.00) total: 4500 /unlink in 36.68 seconds: 122.70 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95166 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: kmemleak disabled modules unloaded.