== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:17:43 (1788628663) Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg323-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg323-server oleg323-server: oleg323-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: kmemleak disabled oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg323-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg323-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 758 (time 1788628891.05 total 10.00 last 75.79) - open/close 1426 (time 1788628901.06 total 20.00 last 66.78) - open/close 2142 (time 1788628911.07 total 30.02 last 71.49) - open/close 2869 (time 1788628921.08 total 40.02 last 72.67) - open/close 3574 (time 1788628931.08 total 50.03 last 70.43) - open/close 4289 (time 1788628941.10 total 60.05 last 71.37) total: 4500 open/close in 62.89 seconds: 71.55 ops/second - /unlink 1329 (time 1788628958.05 total 10.00 last 132.85) - /unlink 2335 (time 1788628968.06 total 20.01 last 100.53) - /unlink 3592 (time 1788628978.06 total 30.01 last 125.68) total: 4500 /unlink in 37.23 seconds: 120.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 615 (time 1788628999.48 total 10.01 last 61.44) - open/close 1233 (time 1788629009.49 total 20.02 last 61.73) - open/close 1903 (time 1788629019.49 total 30.02 last 66.99) - open/close 2576 (time 1788629029.51 total 40.04 last 67.22) - open/close 3229 (time 1788629039.51 total 50.04 last 65.29) - open/close 3948 (time 1788629049.52 total 60.05 last 71.78) total: 4500 open/close in 67.70 seconds: 66.47 ops/second - /unlink 1356 (time 1788629071.26 total 10.00 last 135.57) - /unlink 2593 (time 1788629081.26 total 20.01 last 123.64) - /unlink 3879 (time 1788629091.27 total 30.01 last 128.54) total: 4500 /unlink in 34.25 seconds: 131.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 677 (time 1788629110.65 total 10.01 last 67.63) - open/close 1412 (time 1788629120.67 total 20.02 last 73.40) - open/close 2133 (time 1788629130.68 total 30.03 last 72.03) - open/close 2865 (time 1788629140.69 total 40.05 last 73.09) - open/close 3549 (time 1788629150.69 total 50.05 last 68.39) - open/close 4176 (time 1788629160.70 total 60.05 last 62.68) total: 4500 open/close in 65.01 seconds: 69.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1288 (time 1788629179.75 total 10.00 last 128.77) - /unlink 2624 (time 1788629189.76 total 20.00 last 133.57) - /unlink 3866 (time 1788629199.76 total 30.01 last 124.17) total: 4500 /unlink in 34.60 seconds: 130.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 668 (time 1788629219.24 total 10.00 last 66.80) - open/close 1361 (time 1788629229.25 total 20.01 last 69.22) - open/close 2137 (time 1788629239.26 total 30.02 last 77.56) - open/close 2777 (time 1788629249.26 total 40.02 last 63.95) - open/close 3405 (time 1788629259.27 total 50.03 last 62.73) - open/close 4076 (time 1788629269.28 total 60.04 last 67.03) total: 4500 open/close in 66.36 seconds: 67.81 ops/second - /unlink 1285 (time 1788629289.65 total 10.00 last 128.47) - /unlink 2640 (time 1788629299.66 total 20.01 last 135.39) - /unlink 3862 (time 1788629309.66 total 30.01 last 122.15) total: 4500 /unlink in 34.72 seconds: 129.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 726 (time 1788629328.91 total 10.01 last 72.51) - open/close 1424 (time 1788629338.93 total 20.03 last 69.69) - open/close 2118 (time 1788629348.93 total 30.03 last 69.39) - open/close 2848 (time 1788629358.93 total 40.03 last 72.97) - open/close 3520 (time 1788629368.94 total 50.04 last 67.16) - open/close 4234 (time 1788629378.94 total 60.04 last 71.37) total: 4500 open/close in 64.07 seconds: 70.24 ops/second - /unlink 1282 (time 1788629397.52 total 10.01 last 128.01) - /unlink 2619 (time 1788629407.54 total 20.03 last 133.52) - /unlink 3924 (time 1788629417.54 total 30.03 last 130.45) total: 4500 /unlink in 34.60 seconds: 130.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 679 (time 1788629438.61 total 10.02 last 67.74) - open/close 1321 (time 1788629448.61 total 20.02 last 64.20) - open/close 2020 (time 1788629458.61 total 30.03 last 69.89) - open/close 2695 (time 1788629468.61 total 40.03 last 67.48) - open/close 3372 (time 1788629478.62 total 50.04 last 67.63) - open/close 4065 (time 1788629488.63 total 60.04 last 69.25) total: 4500 open/close in 66.51 seconds: 67.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1316 (time 1788629509.49 total 10.00 last 131.54) - /unlink 2619 (time 1788629519.50 total 20.01 last 130.29) - /unlink 4021 (time 1788629529.50 total 30.01 last 140.15) total: 4500 /unlink in 33.32 seconds: 135.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 750 (time 1788629547.15 total 10.01 last 74.95) - open/close 1469 (time 1788629557.16 total 20.01 last 71.85) - open/close 2202 (time 1788629567.17 total 30.02 last 73.24) - open/close 2916 (time 1788629577.19 total 40.04 last 71.29) - open/close 3611 (time 1788629587.20 total 50.05 last 69.41) - open/close 4322 (time 1788629597.20 total 60.05 last 71.08) total: 4500 open/close in 63.03 seconds: 71.40 ops/second - /unlink 1189 (time 1788629615.25 total 10.01 last 118.82) - /unlink 2449 (time 1788629625.25 total 20.01 last 126.00) - /unlink 3714 (time 1788629635.25 total 30.01 last 126.45) total: 4500 /unlink in 36.29 seconds: 123.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 626 (time 1788629655.35 total 10.02 last 62.49) - open/close 1307 (time 1788629665.36 total 20.03 last 68.04) - open/close 1904 (time 1788629675.39 total 30.05 last 59.53) - open/close 2636 (time 1788629685.40 total 40.06 last 73.12) - open/close 3345 (time 1788629695.41 total 50.08 last 70.82) - open/close 4042 (time 1788629705.42 total 60.09 last 69.63) total: 4500 open/close in 66.46 seconds: 67.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1290 (time 1788629725.88 total 10.02 last 128.80) - /unlink 2384 (time 1788629735.89 total 20.03 last 109.30) - /unlink 3323 (time 1788629745.90 total 30.04 last 93.78) - /unlink 4383 (time 1788629755.90 total 40.04 last 105.99) total: 4500 /unlink in 41.09 seconds: 109.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 607 (time 1788629771.07 total 10.00 last 60.70) - open/close 1296 (time 1788629781.08 total 20.01 last 68.84) - open/close 2001 (time 1788629791.09 total 30.02 last 70.40) - open/close 2663 (time 1788629801.11 total 40.04 last 66.07) - open/close 3258 (time 1788629811.12 total 50.06 last 59.41) - open/close 4011 (time 1788629821.13 total 60.06 last 75.25) total: 4500 open/close in 67.50 seconds: 66.67 ops/second - /unlink 1342 (time 1788629843.06 total 10.00 last 134.16) - /unlink 2776 (time 1788629853.07 total 20.01 last 143.31) - /unlink 4199 (time 1788629863.07 total 30.01 last 142.26) total: 4500 /unlink in 32.10 seconds: 140.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 708 (time 1788629879.07 total 10.00 last 70.78) - open/close 1412 (time 1788629889.10 total 20.03 last 70.22) - open/close 2077 (time 1788629899.10 total 30.03 last 66.47) - open/close 2788 (time 1788629909.13 total 40.06 last 70.92) - open/close 3432 (time 1788629919.14 total 50.06 last 64.36) - open/close 4120 (time 1788629929.15 total 60.08 last 68.71) total: 4500 open/close in 67.06 seconds: 67.10 ops/second - /unlink 1055 (time 1788629952.97 total 10.02 last 105.33) - /unlink 2347 (time 1788629962.97 total 20.02 last 129.20) - /unlink 3677 (time 1788629972.98 total 30.02 last 132.97) total: 4500 /unlink in 36.03 seconds: 124.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 705 (time 1788629992.93 total 10.00 last 70.48) - open/close 1467 (time 1788630002.93 total 20.01 last 76.17) - open/close 2188 (time 1788630012.93 total 30.01 last 72.10) - open/close 2955 (time 1788630022.94 total 40.02 last 76.64) - open/close 3670 (time 1788630032.95 total 50.02 last 71.44) - open/close 4367 (time 1788630042.95 total 60.02 last 69.70) total: 4500 open/close in 62.09 seconds: 72.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1358 (time 1788630059.79 total 10.01 last 135.66) - /unlink 2581 (time 1788630069.80 total 20.02 last 122.17) - /unlink 4004 (time 1788630079.81 total 30.03 last 142.15) total: 4500 /unlink in 33.68 seconds: 133.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 742 (time 1788630097.93 total 10.01 last 74.14) - open/close 1489 (time 1788630107.95 total 20.03 last 74.57) - open/close 2272 (time 1788630117.95 total 30.03 last 78.28) - open/close 3091 (time 1788630127.96 total 40.04 last 81.84) - open/close 3812 (time 1788630137.96 total 50.04 last 72.09) total: 4500 open/close in 59.48 seconds: 75.66 ops/second - /unlink 1448 (time 1788630161.31 total 10.00 last 144.78) - /unlink 2770 (time 1788630171.31 total 20.00 last 132.17) - /unlink 4056 (time 1788630181.32 total 30.01 last 128.52) total: 4500 /unlink in 33.54 seconds: 134.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 695 (time 1788630199.05 total 10.01 last 69.45) - open/close 1462 (time 1788630209.05 total 20.01 last 76.67) - open/close 2135 (time 1788630219.06 total 30.02 last 67.24) - open/close 2829 (time 1788630229.07 total 40.03 last 69.31) - open/close 3585 (time 1788630239.08 total 50.03 last 75.59) - open/close 4290 (time 1788630249.08 total 60.04 last 70.48) total: 4500 open/close in 63.22 seconds: 71.18 ops/second - /unlink 1367 (time 1788630266.16 total 10.01 last 136.60) - /unlink 2686 (time 1788630276.17 total 20.01 last 131.83) - /unlink 3872 (time 1788630286.17 total 30.02 last 118.56) total: 4500 /unlink in 35.79 seconds: 125.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 605 (time 1788630306.81 total 10.01 last 60.47) - open/close 1275 (time 1788630316.83 total 20.03 last 66.85) - open/close 1930 (time 1788630326.84 total 30.04 last 65.40) - open/close 2576 (time 1788630336.86 total 40.06 last 64.50) - open/close 3241 (time 1788630346.87 total 50.07 last 66.42) - open/close 3954 (time 1788630356.89 total 60.09 last 71.15) total: 4500 open/close in 68.06 seconds: 66.12 ops/second - /unlink 1314 (time 1788630379.32 total 10.00 last 131.35) - /unlink 2581 (time 1788630389.32 total 20.01 last 126.65) - /unlink 3828 (time 1788630399.33 total 30.02 last 124.58) total: 4500 /unlink in 35.60 seconds: 126.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 632 (time 1788630419.24 total 10.02 last 63.07) - open/close 1262 (time 1788630429.24 total 20.03 last 62.96) - open/close 1909 (time 1788630439.24 total 30.03 last 64.68) - open/close 2544 (time 1788630449.25 total 40.04 last 63.44) - open/close 3031 (time 1788630459.26 total 50.04 last 48.69) - open/close 3644 (time 1788630469.27 total 60.06 last 61.20) - open/close 4233 (time 1788630479.27 total 70.06 last 58.88) total: 4500 open/close in 75.23 seconds: 59.82 ops/second - /unlink 1258 (time 1788630499.10 total 10.01 last 125.69) - /unlink 2551 (time 1788630509.10 total 20.01 last 129.30) - /unlink 3868 (time 1788630519.10 total 30.01 last 131.63) total: 4500 /unlink in 34.98 seconds: 128.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 726 (time 1788630538.53 total 10.01 last 72.54) - open/close 1424 (time 1788630548.53 total 20.01 last 69.77) - open/close 2079 (time 1788630558.54 total 30.02 last 65.48) - open/close 2793 (time 1788630568.55 total 40.03 last 71.31) - open/close 3524 (time 1788630578.58 total 50.06 last 72.90) - open/close 4131 (time 1788630588.58 total 60.06 last 60.67) total: 4500 open/close in 65.35 seconds: 68.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1027 (time 1788630608.88 total 10.01 last 102.56) - /unlink 2218 (time 1788630618.89 total 20.02 last 118.98) - /unlink 3296 (time 1788630628.90 total 30.03 last 107.74) - /unlink 4404 (time 1788630638.90 total 40.03 last 110.77) total: 4500 /unlink in 41.29 seconds: 108.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 609 (time 1788630656.08 total 10.00 last 60.88) - open/close 1239 (time 1788630666.09 total 20.01 last 62.97) - open/close 1892 (time 1788630676.10 total 30.02 last 65.22) - open/close 2604 (time 1788630686.10 total 40.02 last 71.19) - open/close 3272 (time 1788630696.12 total 50.04 last 66.71) - open/close 3890 (time 1788630706.12 total 60.04 last 61.76) total: 4500 open/close in 69.21 seconds: 65.02 ops/second - /unlink 1258 (time 1788630731.39 total 10.01 last 125.66) - /unlink 2492 (time 1788630741.39 total 20.02 last 123.32) - /unlink 3801 (time 1788630751.40 total 30.02 last 130.89) total: 4500 /unlink in 35.75 seconds: 125.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 674 (time 1788630771.26 total 10.00 last 67.39) - open/close 1403 (time 1788630781.26 total 20.01 last 72.85) - open/close 2125 (time 1788630791.28 total 30.02 last 72.10) - open/close 2822 (time 1788630801.28 total 40.03 last 69.65) - open/close 3467 (time 1788630811.29 total 50.03 last 64.48) - open/close 4107 (time 1788630821.31 total 60.05 last 63.88) total: 4500 open/close in 65.55 seconds: 68.65 ops/second - /unlink 1352 (time 1788630841.43 total 10.00 last 135.16) - /unlink 2637 (time 1788630851.43 total 20.01 last 128.45) - /unlink 3895 (time 1788630861.44 total 30.02 last 125.65) total: 4500 /unlink in 35.59 seconds: 126.45 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95276 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: kmemleak disabled modules unloaded.