== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:25:17 (1788333917) Stopping clients: oleg448-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg448-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg448-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg448-server oleg448-server: oleg448-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: kmemleak disabled oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg448-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg448-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 621 (time 1788334143.23 total 10.00 last 62.09) - open/close 1236 (time 1788334153.24 total 20.02 last 61.40) - open/close 1904 (time 1788334163.25 total 30.02 last 66.77) - open/close 2545 (time 1788334173.26 total 40.03 last 64.05) - open/close 3119 (time 1788334183.26 total 50.04 last 57.37) - open/close 3795 (time 1788334193.27 total 60.04 last 67.55) total: 4500 open/close in 69.97 seconds: 64.31 ops/second - /unlink 1199 (time 1788334217.74 total 10.00 last 119.84) - /unlink 2478 (time 1788334227.75 total 20.02 last 127.73) - /unlink 3742 (time 1788334237.76 total 30.02 last 126.34) total: 4500 /unlink in 37.09 seconds: 121.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 662 (time 1788334259.80 total 10.01 last 66.15) - open/close 1277 (time 1788334269.80 total 20.01 last 61.48) - open/close 1849 (time 1788334279.82 total 30.03 last 57.11) - open/close 2427 (time 1788334289.85 total 40.06 last 57.63) - open/close 3087 (time 1788334299.85 total 50.06 last 65.96) - open/close 3754 (time 1788334309.86 total 60.07 last 66.67) - open/close 4379 (time 1788334319.87 total 70.07 last 62.46) total: 4500 open/close in 72.24 seconds: 62.29 ops/second - /unlink 1073 (time 1788334337.46 total 10.00 last 107.30) - /unlink 2122 (time 1788334347.47 total 20.01 last 104.81) - /unlink 3137 (time 1788334357.48 total 30.01 last 101.48) - /unlink 4439 (time 1788334367.48 total 40.01 last 130.20) total: 4500 /unlink in 40.52 seconds: 111.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 720 (time 1788334382.35 total 10.00 last 71.97) - open/close 1373 (time 1788334392.36 total 20.01 last 65.23) - open/close 1959 (time 1788334402.37 total 30.03 last 58.53) - open/close 2569 (time 1788334412.37 total 40.03 last 60.99) - open/close 3161 (time 1788334422.38 total 50.03 last 59.16) - open/close 3806 (time 1788334432.39 total 60.04 last 64.46) - open/close 4294 (time 1788334442.40 total 70.05 last 48.73) total: 4500 open/close in 74.14 seconds: 60.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1261 (time 1788334463.25 total 10.01 last 125.98) - /unlink 2332 (time 1788334473.25 total 20.01 last 107.05) - /unlink 3408 (time 1788334483.26 total 30.02 last 107.51) total: 4500 /unlink in 39.59 seconds: 113.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 537 (time 1788334509.56 total 10.01 last 53.64) - open/close 1090 (time 1788334519.56 total 20.02 last 55.27) - open/close 1617 (time 1788334529.59 total 30.04 last 52.58) - open/close 2192 (time 1788334539.60 total 40.05 last 57.43) - open/close 2616 (time 1788334549.63 total 50.08 last 42.28) - open/close 3113 (time 1788334559.63 total 60.08 last 49.69) - open/close 3637 (time 1788334569.64 total 70.09 last 52.36) - open/close 4201 (time 1788334579.64 total 80.10 last 56.35) total: 4500 open/close in 85.17 seconds: 52.84 ops/second - /unlink 1184 (time 1788334600.41 total 10.00 last 118.34) - /unlink 2222 (time 1788334610.42 total 20.01 last 103.71) - /unlink 3167 (time 1788334620.43 total 30.02 last 94.47) - /unlink 4252 (time 1788334630.43 total 40.02 last 108.48) total: 4500 /unlink in 42.49 seconds: 105.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 613 (time 1788334648.83 total 10.00 last 61.28) - open/close 1236 (time 1788334658.84 total 20.01 last 62.25) - open/close 1808 (time 1788334668.85 total 30.02 last 57.13) - open/close 2413 (time 1788334678.87 total 40.04 last 60.38) - open/close 3056 (time 1788334688.89 total 50.06 last 64.21) - open/close 3707 (time 1788334698.89 total 60.06 last 65.06) - open/close 4316 (time 1788334708.89 total 70.06 last 60.89) total: 4500 open/close in 73.33 seconds: 61.36 ops/second - /unlink 1140 (time 1788334727.57 total 10.01 last 113.87) - /unlink 2255 (time 1788334737.57 total 20.01 last 111.50) - /unlink 3429 (time 1788334747.57 total 30.02 last 117.35) total: 4500 /unlink in 38.79 seconds: 116.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 660 (time 1788334771.78 total 10.03 last 65.79) - open/close 1304 (time 1788334781.78 total 20.03 last 64.39) - open/close 2005 (time 1788334791.80 total 30.05 last 69.97) - open/close 2663 (time 1788334801.81 total 40.06 last 65.72) - open/close 3378 (time 1788334811.81 total 50.06 last 71.50) - open/close 4054 (time 1788334821.81 total 60.07 last 67.55) total: 4500 open/close in 66.26 seconds: 67.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1296 (time 1788334842.89 total 10.00 last 129.59) - /unlink 2666 (time 1788334852.89 total 20.00 last 136.96) - /unlink 3928 (time 1788334862.90 total 30.01 last 126.14) total: 4500 /unlink in 34.40 seconds: 130.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 654 (time 1788334881.33 total 10.03 last 65.21) - open/close 1369 (time 1788334891.34 total 20.04 last 71.45) - open/close 2073 (time 1788334901.34 total 30.04 last 70.40) - open/close 2732 (time 1788334911.34 total 40.04 last 65.89) - open/close 3414 (time 1788334921.34 total 50.04 last 68.17) - open/close 4117 (time 1788334931.34 total 60.05 last 70.28) total: 4500 open/close in 65.88 seconds: 68.30 ops/second - /unlink 1344 (time 1788334951.80 total 10.00 last 134.35) - /unlink 2630 (time 1788334961.81 total 20.01 last 128.46) - /unlink 4039 (time 1788334971.82 total 30.03 last 140.76) total: 4500 /unlink in 33.44 seconds: 134.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 666 (time 1788334989.60 total 10.00 last 66.58) - open/close 1429 (time 1788334999.61 total 20.01 last 76.24) - open/close 2131 (time 1788335009.62 total 30.02 last 70.16) - open/close 2870 (time 1788335019.62 total 40.02 last 73.86) - open/close 3607 (time 1788335029.64 total 50.04 last 73.59) - open/close 4345 (time 1788335039.65 total 60.05 last 73.72) total: 4500 open/close in 62.32 seconds: 72.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1371 (time 1788335055.64 total 10.02 last 136.89) - /unlink 2728 (time 1788335065.65 total 20.02 last 135.66) - /unlink 4103 (time 1788335075.65 total 30.02 last 137.45) total: 4500 /unlink in 33.12 seconds: 135.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 774 (time 1788335092.92 total 10.02 last 77.26) - open/close 1495 (time 1788335102.93 total 20.03 last 72.05) - open/close 2264 (time 1788335112.94 total 30.04 last 76.82) - open/close 3012 (time 1788335122.95 total 40.05 last 74.71) - open/close 3731 (time 1788335132.97 total 50.06 last 71.81) - open/close 4402 (time 1788335142.98 total 60.07 last 67.04) total: 4500 open/close in 61.72 seconds: 72.92 ops/second - /unlink 1346 (time 1788335159.11 total 10.00 last 134.59) - /unlink 2641 (time 1788335169.11 total 20.00 last 129.50) - /unlink 3984 (time 1788335179.11 total 30.00 last 134.27) total: 4500 /unlink in 33.50 seconds: 134.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 725 (time 1788335197.14 total 10.01 last 72.45) - open/close 1424 (time 1788335207.14 total 20.01 last 69.85) - open/close 2166 (time 1788335217.15 total 30.02 last 74.13) - open/close 2935 (time 1788335227.18 total 40.05 last 76.70) - open/close 3652 (time 1788335237.19 total 50.06 last 71.63) - open/close 4392 (time 1788335247.19 total 60.07 last 73.95) total: 4500 open/close in 61.42 seconds: 73.26 ops/second - /unlink 1438 (time 1788335262.39 total 10.01 last 143.73) - /unlink 2817 (time 1788335272.39 total 20.01 last 137.88) - /unlink 4255 (time 1788335282.40 total 30.01 last 143.76) total: 4500 /unlink in 31.80 seconds: 141.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 763 (time 1788335297.99 total 10.00 last 76.27) - open/close 1466 (time 1788335308.00 total 20.01 last 70.27) - open/close 2236 (time 1788335318.00 total 30.01 last 76.99) - open/close 2976 (time 1788335328.00 total 40.01 last 74.00) - open/close 3694 (time 1788335338.01 total 50.02 last 71.73) - open/close 4489 (time 1788335348.02 total 60.03 last 79.41) total: 4500 open/close in 60.19 seconds: 74.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1431 (time 1788335362.35 total 10.01 last 143.02) - /unlink 2941 (time 1788335372.35 total 20.01 last 150.88) - /unlink 4417 (time 1788335382.37 total 30.02 last 147.43) total: 4500 /unlink in 30.63 seconds: 146.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 750 (time 1788335397.73 total 10.01 last 74.94) - open/close 1513 (time 1788335407.74 total 20.02 last 76.22) - open/close 2277 (time 1788335417.75 total 30.02 last 76.36) - open/close 2993 (time 1788335427.75 total 40.03 last 71.58) - open/close 3698 (time 1788335437.76 total 50.03 last 70.47) - open/close 4448 (time 1788335447.79 total 60.07 last 74.72) total: 4500 open/close in 60.79 seconds: 74.02 ops/second - /unlink 1413 (time 1788335462.59 total 10.00 last 141.27) - /unlink 2834 (time 1788335472.60 total 20.01 last 142.05) - /unlink 4244 (time 1788335482.60 total 30.01 last 140.93) total: 4500 /unlink in 31.80 seconds: 141.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 733 (time 1788335498.34 total 10.01 last 73.25) - open/close 1500 (time 1788335508.35 total 20.01 last 76.66) - open/close 2251 (time 1788335518.36 total 30.02 last 75.04) - open/close 3072 (time 1788335528.37 total 40.03 last 81.99) - open/close 3835 (time 1788335538.37 total 50.04 last 76.30) total: 4500 open/close in 58.42 seconds: 77.03 ops/second - /unlink 1551 (time 1788335560.49 total 10.01 last 155.02) - /unlink 3052 (time 1788335570.49 total 20.01 last 150.04) total: 4500 /unlink in 29.56 seconds: 152.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 744 (time 1788335593.82 total 10.01 last 74.33) - open/close 1469 (time 1788335603.83 total 20.02 last 72.45) - open/close 2221 (time 1788335613.84 total 30.03 last 75.13) - open/close 3012 (time 1788335623.85 total 40.04 last 79.01) - open/close 3791 (time 1788335633.85 total 50.04 last 77.87) total: 4500 open/close in 59.25 seconds: 75.95 ops/second - /unlink 1532 (time 1788335656.47 total 10.01 last 153.06) - /unlink 3062 (time 1788335666.47 total 20.01 last 152.97) total: 4500 /unlink in 29.92 seconds: 150.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 779 (time 1788335689.57 total 10.01 last 77.84) - open/close 1560 (time 1788335699.58 total 20.02 last 78.03) - open/close 2315 (time 1788335709.58 total 30.02 last 75.48) - open/close 3117 (time 1788335719.58 total 40.03 last 80.15) - open/close 3989 (time 1788335729.59 total 50.03 last 87.18) total: 4500 open/close in 56.68 seconds: 79.39 ops/second - /unlink 1605 (time 1788335749.67 total 10.01 last 160.35) - /unlink 3195 (time 1788335759.68 total 20.02 last 158.85) total: 4500 /unlink in 28.30 seconds: 159.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 787 (time 1788335782.06 total 10.01 last 78.63) - open/close 1635 (time 1788335792.06 total 20.01 last 84.80) - open/close 2411 (time 1788335802.06 total 30.01 last 77.60) - open/close 3232 (time 1788335812.06 total 40.01 last 82.09) - open/close 4084 (time 1788335822.07 total 50.01 last 85.16) total: 4500 open/close in 54.95 seconds: 81.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1633 (time 1788335839.93 total 10.01 last 163.18) - /unlink 3206 (time 1788335849.94 total 20.01 last 157.22) total: 4500 /unlink in 28.13 seconds: 159.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 843 (time 1788335871.27 total 10.01 last 84.18) - open/close 1655 (time 1788335881.27 total 20.02 last 81.19) - open/close 2474 (time 1788335891.27 total 30.02 last 81.84) - open/close 3303 (time 1788335901.28 total 40.03 last 82.84) - open/close 4160 (time 1788335911.29 total 50.04 last 85.64) total: 4500 open/close in 53.55 seconds: 84.03 ops/second - /unlink 1716 (time 1788335927.62 total 10.01 last 171.51) - /unlink 3255 (time 1788335937.62 total 20.01 last 153.85) total: 4500 /unlink in 27.87 seconds: 161.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 836 (time 1788335959.33 total 10.02 last 83.47) - open/close 1699 (time 1788335969.34 total 20.02 last 86.23) - open/close 2529 (time 1788335979.35 total 30.03 last 82.91) - open/close 3331 (time 1788335989.36 total 40.04 last 80.14) - open/close 4112 (time 1788335999.36 total 50.04 last 78.09) total: 4500 open/close in 54.85 seconds: 82.04 ops/second - /unlink 1719 (time 1788336016.95 total 10.00 last 171.88) - /unlink 3356 (time 1788336026.95 total 20.00 last 163.69) total: 4500 /unlink in 26.85 seconds: 167.62 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95416 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: kmemleak disabled modules unloaded.