== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:40:02 (1788860402) Stopping clients: oleg147-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg147-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg147-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg147-server oleg147-server: oleg147-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing load_modules_local oleg147-server: kmemleak disabled oleg147-server: Loading modules from /home/green/git/lustre-release/lustre oleg147-server: detected 4 online CPUs by sysfs oleg147-server: MODOPTS_LIBCFS= oleg147-server: Force libcfs to create 2 CPU partitions oleg147-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg147-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg147-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg147-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 577 (time 1788860617.11 total 10.01 last 57.63) - open/close 1159 (time 1788860627.12 total 20.02 last 58.17) - open/close 1745 (time 1788860637.13 total 30.03 last 58.54) - open/close 2254 (time 1788860647.18 total 40.08 last 50.65) - open/close 2716 (time 1788860657.18 total 50.08 last 46.18) - open/close 3276 (time 1788860667.20 total 60.11 last 55.88) - open/close 3832 (time 1788860677.20 total 70.11 last 55.59) - open/close 4471 (time 1788860687.22 total 80.12 last 63.79) total: 4500 open/close in 80.49 seconds: 55.91 ops/second - /unlink 1212 (time 1788860702.53 total 10.02 last 120.99) - /unlink 2443 (time 1788860712.54 total 20.03 last 122.99) - /unlink 3844 (time 1788860722.54 total 30.03 last 140.09) total: 4500 /unlink in 34.36 seconds: 130.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 709 (time 1788860741.88 total 10.00 last 70.89) - open/close 1353 (time 1788860751.89 total 20.00 last 64.38) - open/close 1989 (time 1788860761.89 total 30.01 last 63.58) - open/close 2618 (time 1788860771.89 total 40.01 last 62.90) - open/close 3264 (time 1788860781.89 total 50.01 last 64.60) - open/close 3891 (time 1788860791.89 total 60.01 last 62.69) total: 4500 open/close in 68.36 seconds: 65.82 ops/second - /unlink 1243 (time 1788860814.82 total 10.00 last 124.27) - /unlink 2644 (time 1788860824.82 total 20.00 last 140.08) - /unlink 3804 (time 1788860834.83 total 30.01 last 115.93) total: 4500 /unlink in 36.21 seconds: 124.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 652 (time 1788860856.95 total 10.02 last 65.05) - open/close 1365 (time 1788860866.96 total 20.03 last 71.25) - open/close 2074 (time 1788860876.98 total 30.05 last 70.75) - open/close 2599 (time 1788860886.99 total 40.06 last 52.47) - open/close 3112 (time 1788860897.00 total 50.07 last 51.25) - open/close 3744 (time 1788860907.01 total 60.08 last 63.13) - open/close 4432 (time 1788860917.02 total 70.09 last 68.70) total: 4500 open/close in 71.04 seconds: 63.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1567 (time 1788860932.40 total 10.00 last 156.64) - /unlink 3011 (time 1788860942.40 total 20.01 last 144.34) - /unlink 4322 (time 1788860952.41 total 30.02 last 130.96) total: 4500 /unlink in 31.46 seconds: 143.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 639 (time 1788860969.88 total 10.00 last 63.90) - open/close 1286 (time 1788860979.88 total 20.01 last 64.66) - open/close 2016 (time 1788860989.89 total 30.01 last 72.99) - open/close 2834 (time 1788860999.91 total 40.03 last 81.61) - open/close 3562 (time 1788861009.92 total 50.04 last 72.73) - open/close 4114 (time 1788861019.92 total 60.05 last 55.16) total: 4500 open/close in 66.81 seconds: 67.35 ops/second - /unlink 1372 (time 1788861042.16 total 10.01 last 137.09) - /unlink 2676 (time 1788861052.16 total 20.01 last 130.34) - /unlink 4057 (time 1788861062.18 total 30.03 last 137.87) total: 4500 /unlink in 33.79 seconds: 133.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 778 (time 1788861080.95 total 10.01 last 77.71) - open/close 1344 (time 1788861090.95 total 20.02 last 56.58) - open/close 1812 (time 1788861100.95 total 30.02 last 46.79) - open/close 2482 (time 1788861110.98 total 40.04 last 66.84) - open/close 3183 (time 1788861120.98 total 50.05 last 70.06) - open/close 3752 (time 1788861131.00 total 60.06 last 56.81) - open/close 4417 (time 1788861141.01 total 70.07 last 66.45) total: 4500 open/close in 71.12 seconds: 63.27 ops/second - /unlink 1317 (time 1788861156.70 total 10.00 last 131.69) - /unlink 2742 (time 1788861166.70 total 20.01 last 142.43) - /unlink 4055 (time 1788861176.71 total 30.01 last 131.18) total: 4500 /unlink in 33.45 seconds: 134.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 790 (time 1788861195.68 total 10.01 last 78.92) - open/close 1426 (time 1788861205.69 total 20.02 last 63.53) - open/close 2102 (time 1788861215.69 total 30.02 last 67.59) - open/close 2641 (time 1788861225.70 total 40.02 last 53.90) - open/close 3321 (time 1788861235.70 total 50.03 last 67.98) - open/close 4132 (time 1788861245.70 total 60.03 last 81.10) total: 4500 open/close in 64.80 seconds: 69.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1515 (time 1788861265.08 total 10.00 last 151.46) - /unlink 2951 (time 1788861275.08 total 20.01 last 143.51) - /unlink 4352 (time 1788861285.08 total 30.01 last 140.09) total: 4500 /unlink in 31.07 seconds: 144.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 738 (time 1788861302.02 total 10.02 last 73.67) - open/close 1437 (time 1788861312.02 total 20.02 last 69.87) - open/close 2113 (time 1788861322.03 total 30.03 last 67.55) - open/close 2813 (time 1788861332.05 total 40.05 last 69.86) - open/close 3513 (time 1788861342.05 total 50.06 last 69.95) - open/close 4207 (time 1788861352.07 total 60.08 last 69.26) total: 4500 open/close in 64.70 seconds: 69.55 ops/second - /unlink 1367 (time 1788861371.25 total 10.01 last 136.61) - /unlink 2491 (time 1788861381.25 total 20.01 last 112.35) - /unlink 3453 (time 1788861391.25 total 30.01 last 96.19) - /unlink 4465 (time 1788861401.26 total 40.02 last 101.16) total: 4500 /unlink in 40.35 seconds: 111.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 562 (time 1788861418.79 total 10.02 last 56.10) - open/close 1132 (time 1788861428.81 total 20.05 last 56.84) - open/close 1530 (time 1788861438.83 total 30.06 last 39.76) - open/close 2132 (time 1788861448.84 total 40.07 last 60.13) - open/close 2859 (time 1788861458.84 total 50.07 last 72.67) - open/close 3531 (time 1788861468.85 total 60.08 last 67.17) - open/close 4282 (time 1788861478.86 total 70.09 last 75.00) total: 4500 open/close in 73.87 seconds: 60.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1213 (time 1788861497.88 total 10.01 last 121.19) - /unlink 2417 (time 1788861507.89 total 20.01 last 120.37) - /unlink 3727 (time 1788861517.89 total 30.01 last 130.98) total: 4500 /unlink in 36.47 seconds: 123.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 603 (time 1788861539.04 total 10.02 last 60.20) - open/close 1256 (time 1788861549.04 total 20.02 last 65.29) - open/close 1890 (time 1788861559.05 total 30.03 last 63.31) - open/close 2506 (time 1788861569.06 total 40.05 last 61.52) - open/close 3269 (time 1788861579.07 total 50.05 last 76.26) - open/close 4027 (time 1788861589.08 total 60.06 last 75.70) total: 4500 open/close in 66.13 seconds: 68.04 ops/second - /unlink 1326 (time 1788861609.87 total 10.01 last 132.47) - /unlink 2673 (time 1788861619.87 total 20.01 last 134.70) - /unlink 4144 (time 1788861629.87 total 30.01 last 147.10) total: 4500 /unlink in 33.38 seconds: 134.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 706 (time 1788861648.95 total 10.01 last 70.50) - open/close 1338 (time 1788861658.97 total 20.03 last 63.08) - open/close 2002 (time 1788861668.98 total 30.04 last 66.35) - open/close 2762 (time 1788861678.99 total 40.05 last 75.91) - open/close 3474 (time 1788861689.00 total 50.06 last 71.15) - open/close 4251 (time 1788861699.01 total 60.07 last 77.62) total: 4500 open/close in 63.23 seconds: 71.17 ops/second - /unlink 1301 (time 1788861716.44 total 10.00 last 130.08) - /unlink 2567 (time 1788861726.46 total 20.02 last 126.39) - /unlink 3896 (time 1788861736.46 total 30.02 last 132.89) total: 4500 /unlink in 35.39 seconds: 127.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 794 (time 1788861756.69 total 10.00 last 79.40) - open/close 1550 (time 1788861766.70 total 20.01 last 75.52) - open/close 2235 (time 1788861776.71 total 30.02 last 68.42) - open/close 2784 (time 1788861786.71 total 40.02 last 54.89) - open/close 3452 (time 1788861796.71 total 50.03 last 66.79) - open/close 4166 (time 1788861806.72 total 60.03 last 71.36) total: 4500 open/close in 64.67 seconds: 69.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1302 (time 1788861827.04 total 10.00 last 130.19) - /unlink 2665 (time 1788861837.05 total 20.01 last 136.17) - /unlink 3930 (time 1788861847.05 total 30.01 last 126.47) total: 4500 /unlink in 34.68 seconds: 129.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 658 (time 1788861866.95 total 10.00 last 65.78) - open/close 1352 (time 1788861876.95 total 20.01 last 69.37) - open/close 1921 (time 1788861886.96 total 30.01 last 56.86) - open/close 2526 (time 1788861896.98 total 40.04 last 60.36) - open/close 3306 (time 1788861907.00 total 50.05 last 77.87) - open/close 4075 (time 1788861917.01 total 60.06 last 76.82) total: 4500 open/close in 66.19 seconds: 67.99 ops/second - /unlink 1260 (time 1788861938.44 total 10.00 last 126.00) - /unlink 2464 (time 1788861948.44 total 20.00 last 120.37) - /unlink 3653 (time 1788861958.45 total 30.01 last 118.85) total: 4500 /unlink in 37.49 seconds: 120.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 625 (time 1788861981.77 total 10.01 last 62.45) - open/close 1192 (time 1788861991.79 total 20.02 last 56.60) - open/close 1906 (time 1788862001.79 total 30.03 last 71.36) - open/close 2521 (time 1788862011.82 total 40.05 last 61.35) - open/close 3176 (time 1788862021.82 total 50.06 last 65.49) - open/close 3938 (time 1788862031.83 total 60.06 last 76.13) total: 4500 open/close in 68.47 seconds: 65.72 ops/second - /unlink 1470 (time 1788862054.77 total 10.00 last 146.94) - /unlink 2963 (time 1788862064.77 total 20.01 last 149.22) - /unlink 4058 (time 1788862074.78 total 30.02 last 109.43) total: 4500 /unlink in 33.52 seconds: 134.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 738 (time 1788862094.09 total 10.01 last 73.71) - open/close 1539 (time 1788862104.09 total 20.01 last 80.09) - open/close 2294 (time 1788862114.10 total 30.02 last 75.45) - open/close 3106 (time 1788862124.11 total 40.03 last 81.12) - open/close 3744 (time 1788862134.15 total 50.07 last 63.52) - open/close 4427 (time 1788862144.15 total 60.07 last 68.29) total: 4500 open/close in 60.92 seconds: 73.87 ops/second - /unlink 1122 (time 1788862160.74 total 10.00 last 112.18) - /unlink 2520 (time 1788862170.75 total 20.01 last 139.71) - /unlink 4128 (time 1788862180.75 total 30.01 last 160.77) total: 4500 /unlink in 32.17 seconds: 139.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 723 (time 1788862199.51 total 10.00 last 72.29) - open/close 1392 (time 1788862209.52 total 20.01 last 66.82) - open/close 2104 (time 1788862219.52 total 30.02 last 71.16) - open/close 2887 (time 1788862229.52 total 40.02 last 78.29) - open/close 3562 (time 1788862239.53 total 50.02 last 67.48) - open/close 4374 (time 1788862249.53 total 60.02 last 81.19) total: 4500 open/close in 61.63 seconds: 73.02 ops/second - /unlink 1326 (time 1788862265.82 total 10.00 last 132.54) - /unlink 2489 (time 1788862275.82 total 20.01 last 116.28) - /unlink 3767 (time 1788862285.82 total 30.01 last 127.72) total: 4500 /unlink in 34.95 seconds: 128.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 484 (time 1788862305.72 total 10.04 last 48.22) - open/close 1081 (time 1788862315.75 total 20.07 last 59.50) - open/close 1680 (time 1788862325.75 total 30.07 last 59.89) - open/close 2367 (time 1788862335.76 total 40.08 last 68.64) - open/close 3130 (time 1788862345.77 total 50.09 last 76.25) - open/close 3696 (time 1788862355.79 total 60.11 last 56.48) - open/close 4337 (time 1788862365.81 total 70.13 last 63.98) total: 4500 open/close in 72.47 seconds: 62.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1385 (time 1788862383.24 total 10.00 last 138.49) - /unlink 2713 (time 1788862393.24 total 20.00 last 132.79) - /unlink 4131 (time 1788862403.25 total 30.01 last 141.74) total: 4500 /unlink in 32.78 seconds: 137.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 581 (time 1788862422.42 total 10.01 last 58.05) - open/close 1238 (time 1788862432.42 total 20.02 last 65.64) - open/close 1984 (time 1788862442.43 total 30.03 last 74.53) - open/close 2610 (time 1788862452.47 total 40.06 last 62.38) - open/close 3329 (time 1788862462.50 total 50.09 last 71.69) - open/close 3917 (time 1788862472.51 total 60.10 last 58.76) total: 4500 open/close in 68.48 seconds: 65.71 ops/second - /unlink 1199 (time 1788862496.72 total 10.00 last 119.86) - /unlink 2373 (time 1788862506.72 total 20.00 last 117.39) - /unlink 3594 (time 1788862516.72 total 30.01 last 122.07) total: 4500 /unlink in 37.56 seconds: 119.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 732 (time 1788862539.54 total 10.00 last 73.17) - open/close 1412 (time 1788862549.55 total 20.01 last 67.93) - open/close 1982 (time 1788862559.57 total 30.04 last 56.88) - open/close 2647 (time 1788862569.59 total 40.05 last 66.40) - open/close 3312 (time 1788862579.60 total 50.06 last 66.41) - open/close 3929 (time 1788862589.61 total 60.07 last 61.63) total: 4500 open/close in 69.02 seconds: 65.20 ops/second - /unlink 1054 (time 1788862613.64 total 10.01 last 105.33) - /unlink 2074 (time 1788862623.64 total 20.01 last 101.94) - /unlink 3175 (time 1788862633.65 total 30.02 last 110.04) - /unlink 4207 (time 1788862643.65 total 40.02 last 103.16) total: 4500 /unlink in 43.20 seconds: 104.18 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95299 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg147-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg147-server unloading modules via unload_modules_local on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing unload_modules_local oleg147-server: kmemleak disabled modules unloaded.