== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:27:08 (1788488828) Stopping clients: oleg609-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg609-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg609-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg609-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg609-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg609-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg609-server oleg609-server: oleg609-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg609-server' oleg609-server: oleg609-server.virtnet: executing load_modules_local oleg609-server: kmemleak disabled oleg609-server: Loading modules from /home/green/git/lustre-release/lustre oleg609-server: detected 4 online CPUs by sysfs oleg609-server: MODOPTS_LIBCFS= oleg609-server: Force libcfs to create 2 CPU partitions oleg609-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg609-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg609-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg609-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg609-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg609-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg609-server: oleg609-server.virtnet: executing set_default_debug -1 all pdsh@oleg609-client: oleg609-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg609-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg609-client.virtnet: -o user_xattr,flock 192.168.206.109@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 678 (time 1788489013.08 total 10.03 last 67.57) - open/close 1367 (time 1788489023.08 total 20.04 last 68.85) - open/close 2056 (time 1788489033.09 total 30.05 last 68.86) - open/close 2745 (time 1788489043.10 total 40.06 last 68.84) - open/close 3497 (time 1788489053.10 total 50.06 last 75.16) - open/close 4267 (time 1788489063.11 total 60.07 last 76.94) total: 4500 open/close in 63.34 seconds: 71.05 ops/second - /unlink 1421 (time 1788489080.51 total 10.01 last 142.00) - /unlink 2830 (time 1788489090.51 total 20.01 last 140.84) - /unlink 4178 (time 1788489100.52 total 30.02 last 134.62) total: 4500 /unlink in 32.45 seconds: 138.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 724 (time 1788489117.81 total 10.00 last 72.37) - open/close 1439 (time 1788489127.82 total 20.01 last 71.46) - open/close 2210 (time 1788489137.83 total 30.02 last 77.05) - open/close 2972 (time 1788489147.83 total 40.02 last 76.15) - open/close 3631 (time 1788489157.83 total 50.03 last 65.89) - open/close 4410 (time 1788489167.83 total 60.03 last 77.89) total: 4500 open/close in 61.18 seconds: 73.55 ops/second - /unlink 1591 (time 1788489183.21 total 10.00 last 159.06) - /unlink 2975 (time 1788489193.21 total 20.01 last 138.34) - /unlink 4388 (time 1788489203.22 total 30.02 last 141.17) total: 4500 /unlink in 30.91 seconds: 145.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 747 (time 1788489218.32 total 10.00 last 74.70) - open/close 1476 (time 1788489228.33 total 20.00 last 72.87) - open/close 2292 (time 1788489238.33 total 30.01 last 81.56) - open/close 3022 (time 1788489248.34 total 40.02 last 72.95) - open/close 3702 (time 1788489258.34 total 50.02 last 67.98) - open/close 4498 (time 1788489268.36 total 60.03 last 79.47) total: 4500 open/close in 60.06 seconds: 74.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1384 (time 1788489282.48 total 10.00 last 138.36) - /unlink 2659 (time 1788489292.48 total 20.01 last 127.47) - /unlink 4124 (time 1788489302.49 total 30.01 last 146.47) total: 4500 /unlink in 32.61 seconds: 137.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 747 (time 1788489318.76 total 10.01 last 74.65) - open/close 1412 (time 1788489328.77 total 20.02 last 66.42) - open/close 2073 (time 1788489338.79 total 30.04 last 65.98) - open/close 2777 (time 1788489348.79 total 40.04 last 70.39) - open/close 3488 (time 1788489358.80 total 50.04 last 71.05) - open/close 4181 (time 1788489368.80 total 60.05 last 69.28) total: 4500 open/close in 64.33 seconds: 69.95 ops/second - /unlink 1356 (time 1788489388.12 total 10.02 last 135.39) - /unlink 2635 (time 1788489398.12 total 20.02 last 127.89) - /unlink 3763 (time 1788489408.13 total 30.03 last 112.67) total: 4500 /unlink in 36.05 seconds: 124.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 719 (time 1788489428.51 total 10.02 last 71.76) - open/close 1425 (time 1788489438.51 total 20.02 last 70.57) - open/close 2153 (time 1788489448.53 total 30.04 last 72.69) - open/close 2787 (time 1788489458.54 total 40.05 last 63.33) - open/close 3461 (time 1788489468.58 total 50.09 last 67.15) - open/close 4191 (time 1788489478.58 total 60.09 last 72.95) total: 4500 open/close in 64.07 seconds: 70.24 ops/second - /unlink 1301 (time 1788489497.35 total 10.00 last 130.04) - /unlink 2554 (time 1788489507.36 total 20.02 last 125.16) - /unlink 3777 (time 1788489517.37 total 30.03 last 122.17) total: 4500 /unlink in 36.54 seconds: 123.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 724 (time 1788489539.33 total 10.01 last 72.29) - open/close 1502 (time 1788489549.35 total 20.03 last 77.64) - open/close 2246 (time 1788489559.36 total 30.04 last 74.36) - open/close 3017 (time 1788489569.37 total 40.05 last 76.99) - open/close 3795 (time 1788489579.37 total 50.06 last 77.78) - open/close 4469 (time 1788489589.38 total 60.06 last 67.40) total: 4500 open/close in 60.35 seconds: 74.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1299 (time 1788489604.75 total 10.00 last 129.86) - /unlink 2718 (time 1788489614.75 total 20.00 last 141.88) - /unlink 4053 (time 1788489624.75 total 30.01 last 133.49) total: 4500 /unlink in 33.38 seconds: 134.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 663 (time 1788489643.17 total 10.02 last 66.15) - open/close 1267 (time 1788489653.18 total 20.04 last 60.32) - open/close 2023 (time 1788489663.19 total 30.04 last 75.55) - open/close 2808 (time 1788489673.20 total 40.06 last 78.40) - open/close 3550 (time 1788489683.22 total 50.07 last 74.09) - open/close 4303 (time 1788489693.22 total 60.08 last 75.25) total: 4500 open/close in 62.86 seconds: 71.58 ops/second - /unlink 1424 (time 1788489710.31 total 10.01 last 142.30) - /unlink 2839 (time 1788489720.31 total 20.01 last 141.47) - /unlink 4215 (time 1788489730.31 total 30.01 last 137.57) total: 4500 /unlink in 32.22 seconds: 139.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 683 (time 1788489747.63 total 10.01 last 68.26) - open/close 1350 (time 1788489757.68 total 20.06 last 66.35) - open/close 1905 (time 1788489767.69 total 30.07 last 55.44) - open/close 2597 (time 1788489777.72 total 40.10 last 69.01) - open/close 3305 (time 1788489787.75 total 50.13 last 70.60) - open/close 4083 (time 1788489797.77 total 60.15 last 77.60) total: 4500 open/close in 65.75 seconds: 68.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1436 (time 1788489817.38 total 10.01 last 143.40) - /unlink 2775 (time 1788489827.38 total 20.01 last 133.90) - /unlink 4216 (time 1788489837.38 total 30.01 last 144.09) total: 4500 /unlink in 32.16 seconds: 139.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 712 (time 1788489854.33 total 10.01 last 71.16) - open/close 1418 (time 1788489864.33 total 20.01 last 70.57) - open/close 2058 (time 1788489874.34 total 30.02 last 63.94) - open/close 2732 (time 1788489884.36 total 40.04 last 67.29) - open/close 3405 (time 1788489894.36 total 50.04 last 67.27) - open/close 4143 (time 1788489904.38 total 60.06 last 73.67) total: 4500 open/close in 65.00 seconds: 69.23 ops/second - /unlink 1120 (time 1788489924.03 total 10.00 last 111.98) - /unlink 2174 (time 1788489934.03 total 20.01 last 105.34) - /unlink 3466 (time 1788489944.04 total 30.02 last 129.09) total: 4500 /unlink in 38.48 seconds: 116.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 714 (time 1788489966.53 total 10.00 last 71.39) - open/close 1387 (time 1788489976.54 total 20.01 last 67.26) - open/close 2001 (time 1788489986.54 total 30.01 last 61.38) - open/close 2687 (time 1788489996.56 total 40.03 last 68.44) - open/close 3381 (time 1788490006.57 total 50.04 last 69.33) - open/close 4107 (time 1788490016.59 total 60.06 last 72.51) total: 4500 open/close in 65.34 seconds: 68.87 ops/second - /unlink 1424 (time 1788490035.77 total 10.01 last 142.27) - /unlink 2790 (time 1788490045.78 total 20.01 last 136.53) - /unlink 4186 (time 1788490055.78 total 30.02 last 139.59) total: 4500 /unlink in 33.12 seconds: 135.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 687 (time 1788490074.73 total 10.01 last 68.63) - open/close 1443 (time 1788490084.74 total 20.02 last 75.53) - open/close 2187 (time 1788490094.75 total 30.03 last 74.33) - open/close 2862 (time 1788490104.76 total 40.04 last 67.45) - open/close 3584 (time 1788490114.77 total 50.05 last 72.09) - open/close 4334 (time 1788490124.78 total 60.05 last 74.99) total: 4500 open/close in 62.26 seconds: 72.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1418 (time 1788490141.10 total 10.00 last 141.75) - /unlink 2757 (time 1788490151.11 total 20.01 last 133.78) - /unlink 3898 (time 1788490161.11 total 30.01 last 114.09) total: 4500 /unlink in 34.96 seconds: 128.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 609 (time 1788490180.88 total 10.01 last 60.83) - open/close 1216 (time 1788490190.88 total 20.01 last 60.70) - open/close 1857 (time 1788490200.90 total 30.03 last 63.99) - open/close 2586 (time 1788490210.91 total 40.04 last 72.84) - open/close 3312 (time 1788490220.92 total 50.05 last 72.50) - open/close 4035 (time 1788490230.93 total 60.06 last 72.25) total: 4500 open/close in 66.60 seconds: 67.57 ops/second - /unlink 1233 (time 1788490251.63 total 10.00 last 123.24) - /unlink 2588 (time 1788490261.63 total 20.01 last 135.46) - /unlink 4030 (time 1788490271.64 total 30.01 last 144.12) total: 4500 /unlink in 33.48 seconds: 134.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 705 (time 1788490290.64 total 10.01 last 70.45) - open/close 1400 (time 1788490300.65 total 20.02 last 69.43) - open/close 2054 (time 1788490310.65 total 30.02 last 65.40) - open/close 2823 (time 1788490320.65 total 40.02 last 76.86) - open/close 3576 (time 1788490330.66 total 50.03 last 75.24) - open/close 4343 (time 1788490340.67 total 60.04 last 76.63) total: 4500 open/close in 62.45 seconds: 72.06 ops/second - /unlink 1387 (time 1788490357.50 total 10.00 last 138.66) - /unlink 2902 (time 1788490367.50 total 20.01 last 151.46) - /unlink 4406 (time 1788490377.51 total 30.01 last 150.27) total: 4500 /unlink in 30.75 seconds: 146.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 756 (time 1788490391.71 total 10.02 last 75.46) - open/close 1550 (time 1788490401.73 total 20.03 last 79.30) - open/close 2332 (time 1788490411.73 total 30.03 last 78.17) - open/close 3078 (time 1788490421.74 total 40.05 last 74.49) - open/close 3746 (time 1788490431.74 total 50.05 last 66.79) - open/close 4439 (time 1788490441.75 total 60.05 last 69.27) total: 4500 open/close in 60.86 seconds: 73.93 ops/second - /unlink 1309 (time 1788490456.75 total 10.01 last 130.83) - /unlink 2614 (time 1788490466.75 total 20.01 last 130.48) - /unlink 4016 (time 1788490476.75 total 30.01 last 140.19) total: 4500 /unlink in 34.19 seconds: 131.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 732 (time 1788490495.08 total 10.01 last 73.15) - open/close 1456 (time 1788490505.10 total 20.03 last 72.25) - open/close 2202 (time 1788490515.10 total 30.03 last 74.60) - open/close 2976 (time 1788490525.11 total 40.03 last 77.36) - open/close 3737 (time 1788490535.12 total 50.04 last 76.02) total: 4500 open/close in 59.58 seconds: 75.53 ops/second - /unlink 1544 (time 1788490558.60 total 10.00 last 154.35) - /unlink 3020 (time 1788490568.60 total 20.00 last 147.57) - /unlink 4388 (time 1788490578.60 total 30.01 last 136.79) total: 4500 /unlink in 30.92 seconds: 145.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 786 (time 1788490593.98 total 10.00 last 78.56) - open/close 1592 (time 1788490603.99 total 20.01 last 80.53) - open/close 2307 (time 1788490614.00 total 30.03 last 71.41) - open/close 3084 (time 1788490624.02 total 40.04 last 77.57) - open/close 3913 (time 1788490634.02 total 50.05 last 82.85) total: 4500 open/close in 56.99 seconds: 78.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1514 (time 1788490654.42 total 10.00 last 151.36) - /unlink 3073 (time 1788490664.42 total 20.00 last 155.89) total: 4500 /unlink in 28.53 seconds: 157.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 793 (time 1788490686.66 total 10.01 last 79.24) - open/close 1539 (time 1788490696.67 total 20.02 last 74.55) - open/close 2326 (time 1788490706.69 total 30.03 last 78.56) - open/close 3084 (time 1788490716.69 total 40.04 last 75.78) - open/close 3909 (time 1788490726.69 total 50.04 last 82.48) total: 4500 open/close in 57.59 seconds: 78.14 ops/second - /unlink 1568 (time 1788490747.92 total 10.01 last 156.72) - /unlink 2927 (time 1788490757.93 total 20.01 last 135.78) - /unlink 4451 (time 1788490767.94 total 30.02 last 152.29) total: 4500 /unlink in 30.41 seconds: 147.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 770 (time 1788490782.25 total 10.00 last 77.00) - open/close 1493 (time 1788490792.26 total 20.01 last 72.25) - open/close 2248 (time 1788490802.27 total 30.02 last 75.42) - open/close 3045 (time 1788490812.27 total 40.02 last 79.67) - open/close 3759 (time 1788490822.28 total 50.03 last 71.32) total: 4500 open/close in 59.58 seconds: 75.53 ops/second - /unlink 1409 (time 1788490845.73 total 10.01 last 140.79) - /unlink 2816 (time 1788490855.73 total 20.02 last 140.59) - /unlink 4166 (time 1788490865.74 total 30.02 last 134.95) total: 4500 /unlink in 32.42 seconds: 138.82 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95213 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg609-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg609-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg609-server stop mds service on oleg609-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg609-server stop mds service on oleg609-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg609-server unloading modules via unload_modules_local on: 'oleg609-server' oleg609-server: oleg609-server.virtnet: executing unload_modules_local oleg609-server: kmemleak disabled modules unloaded.