== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:08:04 (1788577684) Stopping clients: oleg329-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg329-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg329-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg329-server oleg329-server: oleg329-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: kmemleak disabled oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg329-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg329-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg329-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 594 (time 1788577903.68 total 10.01 last 59.33) - open/close 1324 (time 1788577913.68 total 20.02 last 72.97) - open/close 2027 (time 1788577923.70 total 30.04 last 70.15) - open/close 2796 (time 1788577933.72 total 40.05 last 76.79) - open/close 3517 (time 1788577943.72 total 50.06 last 72.04) - open/close 4122 (time 1788577953.74 total 60.08 last 60.38) total: 4500 open/close in 67.13 seconds: 67.04 ops/second - /unlink 1266 (time 1788577976.69 total 10.01 last 126.51) - /unlink 2453 (time 1788577986.70 total 20.01 last 118.63) - /unlink 3841 (time 1788577996.70 total 30.02 last 138.78) total: 4500 /unlink in 34.53 seconds: 130.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 687 (time 1788578015.54 total 10.00 last 68.67) - open/close 1378 (time 1788578025.55 total 20.01 last 69.05) - open/close 2058 (time 1788578035.55 total 30.01 last 67.99) - open/close 2716 (time 1788578045.55 total 40.01 last 65.79) - open/close 3365 (time 1788578055.57 total 50.03 last 64.78) - open/close 4090 (time 1788578065.57 total 60.03 last 72.49) total: 4500 open/close in 65.72 seconds: 68.47 ops/second - /unlink 1263 (time 1788578086.81 total 10.01 last 126.24) - /unlink 2337 (time 1788578096.81 total 20.01 last 107.37) - /unlink 3541 (time 1788578106.82 total 30.01 last 120.34) total: 4500 /unlink in 37.73 seconds: 119.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 671 (time 1788578129.31 total 10.01 last 67.01) - open/close 1304 (time 1788578139.34 total 20.04 last 63.12) - open/close 1960 (time 1788578149.37 total 30.08 last 65.38) - open/close 2524 (time 1788578159.37 total 40.08 last 56.39) - open/close 3185 (time 1788578169.39 total 50.10 last 65.96) - open/close 3943 (time 1788578179.41 total 60.11 last 75.71) total: 4500 open/close in 67.88 seconds: 66.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1292 (time 1788578201.81 total 10.01 last 129.12) - /unlink 2537 (time 1788578211.82 total 20.01 last 124.40) - /unlink 3853 (time 1788578221.82 total 30.01 last 131.59) total: 4500 /unlink in 34.52 seconds: 130.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 652 (time 1788578241.15 total 10.02 last 65.08) - open/close 1357 (time 1788578251.15 total 20.03 last 70.45) - open/close 2008 (time 1788578261.15 total 30.03 last 65.09) - open/close 2689 (time 1788578271.16 total 40.03 last 68.09) - open/close 3352 (time 1788578281.16 total 50.03 last 66.28) - open/close 4021 (time 1788578291.16 total 60.04 last 66.87) total: 4500 open/close in 67.19 seconds: 66.98 ops/second - /unlink 1415 (time 1788578312.39 total 10.00 last 141.44) - /unlink 2720 (time 1788578322.39 total 20.01 last 130.46) - /unlink 4008 (time 1788578332.39 total 30.01 last 128.75) total: 4500 /unlink in 33.28 seconds: 135.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 745 (time 1788578351.21 total 10.01 last 74.39) - open/close 1439 (time 1788578361.22 total 20.02 last 69.35) - open/close 2186 (time 1788578371.22 total 30.02 last 74.69) - open/close 2927 (time 1788578381.23 total 40.03 last 74.03) - open/close 3656 (time 1788578391.24 total 50.04 last 72.83) - open/close 4324 (time 1788578401.25 total 60.05 last 66.76) total: 4500 open/close in 62.47 seconds: 72.03 ops/second - /unlink 1434 (time 1788578417.99 total 10.00 last 143.33) - /unlink 2801 (time 1788578427.99 total 20.01 last 136.66) - /unlink 4057 (time 1788578438.00 total 30.02 last 125.45) total: 4500 /unlink in 33.46 seconds: 134.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 688 (time 1788578456.39 total 10.00 last 68.79) - open/close 1429 (time 1788578466.41 total 20.02 last 73.95) - open/close 2094 (time 1788578476.42 total 30.03 last 66.44) - open/close 2823 (time 1788578486.43 total 40.04 last 72.85) - open/close 3464 (time 1788578496.44 total 50.05 last 64.05) - open/close 4295 (time 1788578506.45 total 60.06 last 83.02) total: 4500 open/close in 62.79 seconds: 71.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1547 (time 1788578523.34 total 10.00 last 154.66) - /unlink 3001 (time 1788578533.35 total 20.00 last 145.36) - /unlink 4376 (time 1788578543.35 total 30.01 last 137.50) total: 4500 /unlink in 30.90 seconds: 145.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 794 (time 1788578557.83 total 10.00 last 79.39) - open/close 1552 (time 1788578567.83 total 20.01 last 75.77) - open/close 2303 (time 1788578577.85 total 30.02 last 74.99) - open/close 3079 (time 1788578587.87 total 40.04 last 77.45) - open/close 3871 (time 1788578597.88 total 50.05 last 79.13) total: 4500 open/close in 57.53 seconds: 78.22 ops/second - /unlink 1517 (time 1788578620.05 total 10.00 last 151.68) - /unlink 3057 (time 1788578630.05 total 20.00 last 153.97) total: 4500 /unlink in 29.12 seconds: 154.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 887 (time 1788578653.48 total 10.01 last 88.65) - open/close 1716 (time 1788578663.49 total 20.01 last 82.88) - open/close 2544 (time 1788578673.49 total 30.01 last 82.76) - open/close 3345 (time 1788578683.50 total 40.02 last 80.03) - open/close 4169 (time 1788578693.50 total 50.03 last 82.35) total: 4500 open/close in 53.92 seconds: 83.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1682 (time 1788578710.95 total 10.00 last 168.13) - /unlink 3284 (time 1788578720.96 total 20.01 last 160.13) total: 4500 /unlink in 28.24 seconds: 159.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 845 (time 1788578743.06 total 10.01 last 84.45) - open/close 1448 (time 1788578753.07 total 20.01 last 60.27) - open/close 1986 (time 1788578763.07 total 30.01 last 53.80) - open/close 2747 (time 1788578773.08 total 40.02 last 76.05) - open/close 3597 (time 1788578783.08 total 50.02 last 85.00) - open/close 4407 (time 1788578793.08 total 60.02 last 80.96) total: 4500 open/close in 60.99 seconds: 73.78 ops/second - /unlink 1592 (time 1788578807.48 total 10.01 last 159.11) - /unlink 3181 (time 1788578817.49 total 20.01 last 158.84) total: 4500 /unlink in 27.93 seconds: 161.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 893 (time 1788578839.69 total 10.02 last 89.09) - open/close 1669 (time 1788578849.71 total 20.04 last 77.43) - open/close 2543 (time 1788578859.72 total 30.05 last 87.36) - open/close 3463 (time 1788578869.72 total 40.05 last 91.96) - open/close 4315 (time 1788578879.72 total 50.05 last 85.20) total: 4500 open/close in 52.92 seconds: 85.04 ops/second - /unlink 1387 (time 1788578896.31 total 10.01 last 138.56) - /unlink 2770 (time 1788578906.32 total 20.02 last 138.17) - /unlink 4241 (time 1788578916.32 total 30.02 last 147.09) total: 4500 /unlink in 32.09 seconds: 140.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 720 (time 1788578932.71 total 10.01 last 71.92) - open/close 1537 (time 1788578942.71 total 20.01 last 81.68) - open/close 2364 (time 1788578952.71 total 30.01 last 82.69) - open/close 3121 (time 1788578962.72 total 40.03 last 75.61) - open/close 3844 (time 1788578972.73 total 50.04 last 72.23) total: 4500 open/close in 58.00 seconds: 77.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1616 (time 1788578994.76 total 10.00 last 161.58) - /unlink 2991 (time 1788579004.77 total 20.00 last 137.46) total: 4500 /unlink in 29.83 seconds: 150.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 832 (time 1788579028.50 total 10.00 last 83.18) - open/close 1630 (time 1788579038.51 total 20.01 last 79.71) - open/close 2392 (time 1788579048.52 total 30.02 last 76.14) - open/close 3266 (time 1788579058.52 total 40.02 last 87.38) - open/close 4112 (time 1788579068.53 total 50.03 last 84.55) total: 4500 open/close in 53.97 seconds: 83.38 ops/second - /unlink 1615 (time 1788579085.65 total 10.00 last 161.42) - /unlink 3123 (time 1788579095.66 total 20.01 last 150.68) total: 4500 /unlink in 28.10 seconds: 160.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 772 (time 1788579117.88 total 10.00 last 77.17) - open/close 1670 (time 1788579127.90 total 20.02 last 89.62) - open/close 2549 (time 1788579137.91 total 30.03 last 87.82) - open/close 3385 (time 1788579147.92 total 40.04 last 83.52) - open/close 4193 (time 1788579157.92 total 50.04 last 80.79) total: 4500 open/close in 54.08 seconds: 83.21 ops/second - /unlink 1632 (time 1788579175.69 total 10.01 last 163.11) - /unlink 3274 (time 1788579185.70 total 20.01 last 164.14) total: 4500 /unlink in 27.85 seconds: 161.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 803 (time 1788579207.59 total 10.01 last 80.21) - open/close 1536 (time 1788579217.61 total 20.03 last 73.15) - open/close 2278 (time 1788579227.61 total 30.03 last 74.18) - open/close 3027 (time 1788579237.63 total 40.05 last 74.80) - open/close 3669 (time 1788579247.63 total 50.05 last 64.18) - open/close 4331 (time 1788579257.64 total 60.06 last 66.15) total: 4500 open/close in 62.05 seconds: 72.52 ops/second - /unlink 1450 (time 1788579273.66 total 10.00 last 144.94) - /unlink 2887 (time 1788579283.66 total 20.00 last 143.70) - /unlink 4334 (time 1788579293.66 total 30.01 last 144.62) total: 4500 /unlink in 31.31 seconds: 143.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 805 (time 1788579308.72 total 10.01 last 80.46) - open/close 1626 (time 1788579318.72 total 20.01 last 82.07) - open/close 2459 (time 1788579328.72 total 30.01 last 83.29) - open/close 3306 (time 1788579338.73 total 40.02 last 84.63) - open/close 4156 (time 1788579348.74 total 50.03 last 84.90) total: 4500 open/close in 54.14 seconds: 83.12 ops/second - /unlink 1543 (time 1788579367.06 total 10.00 last 154.29) - /unlink 3056 (time 1788579377.06 total 20.01 last 151.20) - /unlink 4476 (time 1788579387.06 total 30.01 last 141.99) total: 4500 /unlink in 30.20 seconds: 149.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 772 (time 1788579401.10 total 10.01 last 77.15) - open/close 1548 (time 1788579411.10 total 20.01 last 77.57) - open/close 2317 (time 1788579421.11 total 30.02 last 76.85) - open/close 2994 (time 1788579431.11 total 40.02 last 67.68) - open/close 3721 (time 1788579441.11 total 50.02 last 72.69) - open/close 4431 (time 1788579451.12 total 60.03 last 70.96) total: 4500 open/close in 60.82 seconds: 73.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1626 (time 1788579466.04 total 10.01 last 162.50) - /unlink 3052 (time 1788579476.05 total 20.02 last 142.43) total: 4500 /unlink in 29.99 seconds: 150.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 743 (time 1788579500.61 total 10.01 last 74.21) - open/close 1505 (time 1788579510.62 total 20.02 last 76.16) - open/close 2300 (time 1788579520.64 total 30.05 last 79.27) - open/close 3004 (time 1788579530.65 total 40.05 last 70.37) - open/close 3805 (time 1788579540.66 total 50.06 last 80.00) total: 4500 open/close in 58.80 seconds: 76.52 ops/second - /unlink 1535 (time 1788579563.29 total 10.00 last 153.48) - /unlink 3069 (time 1788579573.29 total 20.00 last 153.37) total: 4500 /unlink in 29.81 seconds: 150.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 608 (time 1788579597.80 total 10.00 last 60.80) - open/close 1412 (time 1788579607.81 total 20.01 last 80.30) - open/close 2237 (time 1788579617.82 total 30.02 last 82.41) - open/close 3029 (time 1788579627.83 total 40.03 last 79.15) - open/close 3758 (time 1788579637.83 total 50.03 last 72.89) - open/close 4429 (time 1788579647.83 total 60.03 last 67.10) total: 4500 open/close in 61.06 seconds: 73.70 ops/second - /unlink 999 (time 1788579664.58 total 10.00 last 99.88) - /unlink 2383 (time 1788579674.59 total 20.01 last 138.35) - /unlink 3896 (time 1788579684.59 total 30.01 last 151.25) total: 4500 /unlink in 34.23 seconds: 131.45 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95323 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local oleg329-server: kmemleak disabled modules unloaded.