== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:28:46 (1788816526) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg112-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: kmemleak disabled oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: MODOPTS_LIBCFS= oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg112-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg112-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg112-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 871 (time 1788816705.66 total 10.00 last 87.08) - open/close 1685 (time 1788816715.67 total 20.00 last 81.39) - open/close 2562 (time 1788816725.68 total 30.01 last 87.61) - open/close 3417 (time 1788816735.68 total 40.02 last 85.43) - open/close 4274 (time 1788816745.69 total 50.02 last 85.68) total: 4500 open/close in 52.55 seconds: 85.64 ops/second - /unlink 1604 (time 1788816761.87 total 10.01 last 160.30) - /unlink 3153 (time 1788816771.88 total 20.01 last 154.83) total: 4500 /unlink in 29.50 seconds: 152.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 755 (time 1788816795.98 total 10.01 last 75.44) - open/close 1634 (time 1788816805.99 total 20.02 last 87.82) - open/close 2504 (time 1788816816.01 total 30.04 last 86.83) - open/close 3401 (time 1788816826.01 total 40.04 last 89.64) - open/close 4257 (time 1788816836.02 total 50.05 last 85.51) total: 4500 open/close in 52.88 seconds: 85.09 ops/second - /unlink 1547 (time 1788816852.79 total 10.01 last 154.60) - /unlink 3178 (time 1788816862.79 total 20.01 last 163.09) total: 4500 /unlink in 27.88 seconds: 161.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 850 (time 1788816884.95 total 10.00 last 84.97) - open/close 1661 (time 1788816894.95 total 20.01 last 81.06) - open/close 2596 (time 1788816904.96 total 30.01 last 93.45) - open/close 3642 (time 1788816914.96 total 40.02 last 104.54) total: 4500 open/close in 48.45 seconds: 92.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1614 (time 1788816936.51 total 10.01 last 161.31) - /unlink 3220 (time 1788816946.51 total 20.01 last 160.59) total: 4500 /unlink in 28.10 seconds: 160.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 978 (time 1788816967.97 total 10.01 last 97.75) - open/close 1985 (time 1788816977.98 total 20.01 last 100.63) - open/close 2979 (time 1788816987.98 total 30.02 last 99.37) - open/close 3875 (time 1788816997.98 total 40.02 last 89.59) total: 4500 open/close in 47.03 seconds: 95.68 ops/second - /unlink 1625 (time 1788817018.50 total 10.00 last 162.46) - /unlink 3534 (time 1788817028.50 total 20.00 last 190.88) total: 4500 /unlink in 24.89 seconds: 180.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 995 (time 1788817046.91 total 10.02 last 99.28) - open/close 1968 (time 1788817056.91 total 20.03 last 97.26) - open/close 2982 (time 1788817066.92 total 30.03 last 101.33) - open/close 3980 (time 1788817076.92 total 40.04 last 99.78) total: 4500 open/close in 45.09 seconds: 99.81 ops/second - /unlink 2043 (time 1788817094.88 total 10.00 last 204.27) - /unlink 3886 (time 1788817104.89 total 20.01 last 184.18) total: 4500 /unlink in 23.35 seconds: 192.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 941 (time 1788817121.12 total 10.01 last 94.05) - open/close 1863 (time 1788817131.12 total 20.01 last 92.16) - open/close 2861 (time 1788817141.13 total 30.02 last 99.72) - open/close 3860 (time 1788817151.13 total 40.02 last 99.88) total: 4500 open/close in 44.85 seconds: 100.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1986 (time 1788817168.36 total 10.00 last 198.59) - /unlink 4293 (time 1788817178.36 total 20.00 last 230.67) total: 4500 /unlink in 20.95 seconds: 214.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 1029 (time 1788817191.63 total 10.01 last 102.80) - open/close 2019 (time 1788817201.63 total 20.01 last 98.98) - open/close 3155 (time 1788817211.64 total 30.01 last 113.56) - open/close 4231 (time 1788817221.65 total 40.03 last 107.49) total: 4500 open/close in 42.60 seconds: 105.64 ops/second - /unlink 2027 (time 1788817237.04 total 10.00 last 202.68) - /unlink 4011 (time 1788817247.04 total 20.00 last 198.38) total: 4500 /unlink in 22.46 seconds: 200.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 1082 (time 1788817262.22 total 10.00 last 108.19) - open/close 2161 (time 1788817272.23 total 20.01 last 107.82) - open/close 3191 (time 1788817282.24 total 30.02 last 102.90) - open/close 4267 (time 1788817292.25 total 40.03 last 107.45) total: 4500 open/close in 42.23 seconds: 106.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 2438 (time 1788817306.92 total 10.00 last 243.79) total: 4500 /unlink in 19.18 seconds: 234.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1606 (time 1788817328.54 total 10.01 last 160.50) - open/close 2134 (time 1788817338.55 total 20.01 last 52.77) - open/close 2784 (time 1788817348.55 total 30.02 last 64.97) - open/close 3267 (time 1788817358.56 total 40.02 last 48.26) - open/close 3650 (time 1788817368.56 total 50.03 last 38.29) - open/close 3985 (time 1788817378.57 total 60.04 last 33.46) - open/close 4447 (time 1788817388.57 total 70.04 last 46.19) total: 4500 open/close in 71.17 seconds: 63.23 ops/second - /unlink 1186 (time 1788817405.17 total 10.00 last 118.57) - /unlink 2336 (time 1788817415.18 total 20.01 last 114.95) - /unlink 3789 (time 1788817425.19 total 30.02 last 145.09) total: 4500 /unlink in 35.84 seconds: 125.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 832 (time 1788817445.90 total 10.01 last 83.08) - open/close 1632 (time 1788817455.90 total 20.02 last 79.99) - open/close 2469 (time 1788817465.90 total 30.02 last 83.68) - open/close 3305 (time 1788817475.91 total 40.03 last 83.51) - open/close 4064 (time 1788817485.92 total 50.04 last 75.83) total: 4500 open/close in 55.47 seconds: 81.13 ops/second - /unlink 1529 (time 1788817505.52 total 10.00 last 152.88) - /unlink 3027 (time 1788817515.53 total 20.01 last 149.74) total: 4500 /unlink in 29.51 seconds: 152.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 775 (time 1788817539.69 total 10.02 last 77.38) - open/close 1558 (time 1788817549.70 total 20.03 last 78.21) - open/close 2361 (time 1788817559.71 total 30.03 last 80.28) - open/close 3172 (time 1788817569.71 total 40.03 last 81.08) - open/close 3843 (time 1788817579.71 total 50.04 last 67.07) total: 4500 open/close in 59.15 seconds: 76.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1319 (time 1788817603.77 total 10.00 last 131.89) - /unlink 2669 (time 1788817613.78 total 20.00 last 134.95) - /unlink 4211 (time 1788817623.78 total 30.01 last 154.12) total: 4500 /unlink in 32.40 seconds: 138.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 759 (time 1788817642.02 total 10.01 last 75.86) - open/close 1475 (time 1788817652.02 total 20.01 last 71.57) - open/close 2103 (time 1788817662.02 total 30.01 last 62.80) - open/close 2912 (time 1788817672.03 total 40.02 last 80.82) - open/close 3572 (time 1788817682.04 total 50.03 last 65.94) - open/close 4322 (time 1788817692.05 total 60.03 last 74.96) total: 4500 open/close in 62.10 seconds: 72.47 ops/second - /unlink 1383 (time 1788817708.92 total 10.00 last 138.30) - /unlink 2730 (time 1788817718.92 total 20.00 last 134.65) - /unlink 4110 (time 1788817728.93 total 30.01 last 137.95) total: 4500 /unlink in 32.55 seconds: 138.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 725 (time 1788817747.33 total 10.01 last 72.44) - open/close 1440 (time 1788817757.35 total 20.02 last 71.38) - open/close 2129 (time 1788817767.37 total 30.04 last 68.80) - open/close 2889 (time 1788817777.37 total 40.04 last 75.99) - open/close 3645 (time 1788817787.37 total 50.04 last 75.57) - open/close 4430 (time 1788817797.38 total 60.05 last 78.42) total: 4500 open/close in 60.99 seconds: 73.78 ops/second - /unlink 1373 (time 1788817813.33 total 10.00 last 137.27) - /unlink 2763 (time 1788817823.33 total 20.00 last 139.00) - /unlink 4184 (time 1788817833.33 total 30.01 last 142.06) total: 4500 /unlink in 32.24 seconds: 139.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 791 (time 1788817850.43 total 10.00 last 79.08) - open/close 1557 (time 1788817860.44 total 20.01 last 76.56) - open/close 2213 (time 1788817870.44 total 30.01 last 65.57) - open/close 2852 (time 1788817880.45 total 40.02 last 63.83) - open/close 3251 (time 1788817890.45 total 50.03 last 39.90) - open/close 3893 (time 1788817900.46 total 60.03 last 64.15) total: 4500 open/close in 68.21 seconds: 65.98 ops/second - /unlink 1447 (time 1788817923.76 total 10.01 last 144.62) - /unlink 2506 (time 1788817933.77 total 20.02 last 105.78) - /unlink 3469 (time 1788817943.78 total 30.02 last 96.27) total: 4500 /unlink in 39.87 seconds: 112.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 725 (time 1788817969.84 total 10.01 last 72.45) - open/close 1356 (time 1788817979.85 total 20.01 last 63.06) - open/close 2136 (time 1788817989.85 total 30.02 last 77.98) - open/close 2806 (time 1788817999.85 total 40.02 last 66.96) - open/close 3537 (time 1788818009.86 total 50.03 last 73.05) - open/close 4239 (time 1788818019.88 total 60.04 last 70.10) total: 4500 open/close in 64.07 seconds: 70.24 ops/second - /unlink 1413 (time 1788818038.03 total 10.01 last 141.16) - /unlink 2648 (time 1788818048.03 total 20.02 last 123.43) - /unlink 3878 (time 1788818058.04 total 30.02 last 122.89) total: 4500 /unlink in 34.97 seconds: 128.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 710 (time 1788818077.90 total 10.03 last 70.80) - open/close 1482 (time 1788818087.92 total 20.05 last 77.03) - open/close 2114 (time 1788818097.93 total 30.06 last 63.14) - open/close 2880 (time 1788818107.93 total 40.06 last 76.56) - open/close 3618 (time 1788818117.94 total 50.07 last 73.77) - open/close 4387 (time 1788818127.95 total 60.08 last 76.79) total: 4500 open/close in 61.78 seconds: 72.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1382 (time 1788818144.29 total 10.00 last 138.16) - /unlink 2707 (time 1788818154.29 total 20.01 last 132.44) - /unlink 4025 (time 1788818164.31 total 30.02 last 131.62) total: 4500 /unlink in 33.94 seconds: 132.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 733 (time 1788818183.83 total 10.02 last 73.18) - open/close 1398 (time 1788818193.86 total 20.04 last 66.33) - open/close 1977 (time 1788818203.88 total 30.06 last 57.77) - open/close 2676 (time 1788818213.88 total 40.07 last 69.89) - open/close 3331 (time 1788818223.88 total 50.07 last 65.50) - open/close 3965 (time 1788818233.89 total 60.08 last 63.32) total: 4500 open/close in 68.41 seconds: 65.78 ops/second - /unlink 1325 (time 1788818257.40 total 10.01 last 132.42) - /unlink 2629 (time 1788818267.41 total 20.01 last 130.29) - /unlink 3827 (time 1788818277.41 total 30.02 last 119.78) total: 4500 /unlink in 36.05 seconds: 124.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 565 (time 1788818298.82 total 10.01 last 56.43) - open/close 1262 (time 1788818308.82 total 20.02 last 69.67) - open/close 1923 (time 1788818318.86 total 30.06 last 65.81) - open/close 2512 (time 1788818328.87 total 40.06 last 58.88) - open/close 3216 (time 1788818338.88 total 50.07 last 70.33) - open/close 3888 (time 1788818348.88 total 60.08 last 67.14) total: 4500 open/close in 68.10 seconds: 66.08 ops/second - /unlink 1050 (time 1788818371.90 total 10.00 last 104.99) - /unlink 1853 (time 1788818381.92 total 20.03 last 80.09) - /unlink 3187 (time 1788818391.93 total 30.03 last 133.35) total: 4500 /unlink in 39.48 seconds: 113.98 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95086 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: kmemleak disabled modules unloaded.