== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:14:51 (1788873291) Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg347-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg347-server oleg347-server: oleg347-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: kmemleak disabled oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: MODOPTS_LIBCFS= oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg347-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg347-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 686 (time 1788873499.31 total 10.00 last 68.58) - open/close 1370 (time 1788873509.32 total 20.01 last 68.35) - open/close 2022 (time 1788873519.33 total 30.02 last 65.16) - open/close 2678 (time 1788873529.33 total 40.02 last 65.56) - open/close 3355 (time 1788873539.34 total 50.03 last 67.63) - open/close 3944 (time 1788873549.37 total 60.06 last 58.72) total: 4500 open/close in 68.35 seconds: 65.84 ops/second - /unlink 1176 (time 1788873572.71 total 10.01 last 117.48) - /unlink 2307 (time 1788873582.71 total 20.01 last 113.09) - /unlink 3496 (time 1788873592.72 total 30.01 last 118.87) total: 4500 /unlink in 38.56 seconds: 116.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 624 (time 1788873617.10 total 10.01 last 62.35) - open/close 1247 (time 1788873627.12 total 20.03 last 62.19) - open/close 1903 (time 1788873637.12 total 30.03 last 65.59) - open/close 2581 (time 1788873647.13 total 40.04 last 67.73) - open/close 3247 (time 1788873657.15 total 50.05 last 66.50) - open/close 3852 (time 1788873667.16 total 60.07 last 60.41) - open/close 4479 (time 1788873677.18 total 70.09 last 62.59) total: 4500 open/close in 70.47 seconds: 63.85 ops/second - /unlink 1232 (time 1788873693.38 total 10.00 last 123.14) - /unlink 2372 (time 1788873703.40 total 20.02 last 113.79) - /unlink 3574 (time 1788873713.40 total 30.02 last 120.19) total: 4500 /unlink in 37.76 seconds: 119.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 677 (time 1788873736.17 total 10.01 last 67.65) - open/close 1297 (time 1788873746.17 total 20.01 last 61.98) - open/close 1986 (time 1788873756.18 total 30.02 last 68.86) - open/close 2693 (time 1788873766.18 total 40.02 last 70.67) - open/close 3350 (time 1788873776.20 total 50.03 last 65.61) - open/close 3973 (time 1788873786.20 total 60.04 last 62.28) total: 4500 open/close in 67.71 seconds: 66.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1224 (time 1788873808.54 total 10.01 last 122.27) - /unlink 2425 (time 1788873818.55 total 20.01 last 120.05) - /unlink 3537 (time 1788873828.57 total 30.04 last 110.96) total: 4500 /unlink in 37.82 seconds: 118.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 620 (time 1788873852.05 total 10.00 last 61.98) - open/close 1280 (time 1788873862.05 total 20.01 last 65.98) - open/close 1923 (time 1788873872.06 total 30.01 last 64.26) - open/close 2593 (time 1788873882.06 total 40.02 last 66.98) - open/close 3253 (time 1788873892.07 total 50.02 last 65.94) - open/close 3979 (time 1788873902.08 total 60.04 last 72.49) total: 4500 open/close in 69.16 seconds: 65.07 ops/second - /unlink 1169 (time 1788873926.82 total 10.01 last 116.83) - /unlink 2302 (time 1788873936.83 total 20.01 last 113.20) - /unlink 3554 (time 1788873946.84 total 30.02 last 125.12) total: 4500 /unlink in 38.25 seconds: 117.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 626 (time 1788873971.32 total 10.00 last 62.57) - open/close 1298 (time 1788873981.33 total 20.01 last 67.17) - open/close 1941 (time 1788873991.33 total 30.01 last 64.28) - open/close 2528 (time 1788874001.35 total 40.03 last 58.58) - open/close 3170 (time 1788874011.37 total 50.05 last 64.12) - open/close 3820 (time 1788874021.37 total 60.05 last 64.94) - open/close 4484 (time 1788874031.39 total 70.07 last 66.30) total: 4500 open/close in 70.26 seconds: 64.05 ops/second - /unlink 1238 (time 1788874045.88 total 10.00 last 123.78) - /unlink 2495 (time 1788874055.89 total 20.00 last 125.67) - /unlink 3877 (time 1788874065.89 total 30.01 last 138.14) total: 4500 /unlink in 35.35 seconds: 127.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 681 (time 1788874086.62 total 10.03 last 67.88) - open/close 1315 (time 1788874096.63 total 20.05 last 63.29) - open/close 2011 (time 1788874106.66 total 30.07 last 69.44) - open/close 2696 (time 1788874116.66 total 40.08 last 68.47) - open/close 3328 (time 1788874126.68 total 50.09 last 63.10) - open/close 3926 (time 1788874136.68 total 60.10 last 59.78) total: 4500 open/close in 68.43 seconds: 65.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1257 (time 1788874160.04 total 10.01 last 125.61) - /unlink 2381 (time 1788874170.04 total 20.01 last 112.38) - /unlink 3592 (time 1788874180.04 total 30.01 last 121.09) total: 4500 /unlink in 37.31 seconds: 120.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 751 (time 1788874202.42 total 10.00 last 75.10) - open/close 1390 (time 1788874212.42 total 20.01 last 63.86) - open/close 2063 (time 1788874222.42 total 30.01 last 67.30) - open/close 2777 (time 1788874232.44 total 40.02 last 71.31) - open/close 3530 (time 1788874242.44 total 50.02 last 75.30) - open/close 4248 (time 1788874252.45 total 60.03 last 71.74) total: 4500 open/close in 63.90 seconds: 70.42 ops/second - /unlink 1264 (time 1788874271.03 total 10.00 last 126.38) - /unlink 2402 (time 1788874281.03 total 20.01 last 113.74) - /unlink 3553 (time 1788874291.04 total 30.01 last 115.05) total: 4500 /unlink in 38.36 seconds: 117.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 605 (time 1788874314.53 total 10.01 last 60.44) - open/close 1282 (time 1788874324.53 total 20.01 last 67.69) - open/close 1922 (time 1788874334.54 total 30.02 last 63.97) - open/close 2587 (time 1788874344.55 total 40.03 last 66.42) - open/close 3287 (time 1788874354.56 total 50.04 last 69.95) - open/close 3946 (time 1788874364.56 total 60.04 last 65.87) total: 4500 open/close in 67.75 seconds: 66.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1321 (time 1788874386.48 total 10.00 last 132.05) - /unlink 2503 (time 1788874396.48 total 20.00 last 118.20) - /unlink 3779 (time 1788874406.48 total 30.01 last 127.55) total: 4500 /unlink in 35.57 seconds: 126.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 644 (time 1788874427.25 total 10.02 last 64.30) - open/close 1306 (time 1788874437.25 total 20.02 last 66.17) - open/close 1970 (time 1788874447.26 total 30.03 last 66.35) - open/close 2617 (time 1788874457.27 total 40.04 last 64.64) - open/close 3285 (time 1788874467.27 total 50.04 last 66.78) - open/close 3952 (time 1788874477.28 total 60.05 last 66.64) total: 4500 open/close in 67.63 seconds: 66.53 ops/second - /unlink 1282 (time 1788874499.65 total 10.01 last 128.10) - /unlink 2419 (time 1788874509.65 total 20.01 last 113.67) - /unlink 3704 (time 1788874519.65 total 30.01 last 128.47) total: 4500 /unlink in 36.84 seconds: 122.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 663 (time 1788874541.85 total 10.00 last 66.27) - open/close 1350 (time 1788874551.87 total 20.02 last 68.57) - open/close 2032 (time 1788874561.89 total 30.04 last 68.06) - open/close 2642 (time 1788874571.90 total 40.06 last 60.90) - open/close 3320 (time 1788874581.91 total 50.06 last 67.77) - open/close 4008 (time 1788874591.91 total 60.07 last 68.76) total: 4500 open/close in 67.29 seconds: 66.88 ops/second - /unlink 1195 (time 1788874613.70 total 10.00 last 119.47) - /unlink 2387 (time 1788874623.70 total 20.01 last 119.13) - /unlink 3589 (time 1788874633.71 total 30.01 last 120.14) total: 4500 /unlink in 38.22 seconds: 117.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 628 (time 1788874657.28 total 10.00 last 62.77) - open/close 1290 (time 1788874667.29 total 20.01 last 66.18) - open/close 1996 (time 1788874677.29 total 30.01 last 70.58) - open/close 2681 (time 1788874687.30 total 40.02 last 68.42) - open/close 3296 (time 1788874697.32 total 50.04 last 61.39) - open/close 3975 (time 1788874707.33 total 60.05 last 67.82) total: 4500 open/close in 68.22 seconds: 65.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1203 (time 1788874730.79 total 10.01 last 120.17) - /unlink 2380 (time 1788874740.79 total 20.01 last 117.68) - /unlink 3541 (time 1788874750.79 total 30.01 last 116.08) total: 4500 /unlink in 38.09 seconds: 118.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 629 (time 1788874774.97 total 10.01 last 62.83) - open/close 1232 (time 1788874784.97 total 20.01 last 60.29) - open/close 1865 (time 1788874794.97 total 30.02 last 63.27) - open/close 2443 (time 1788874805.03 total 40.07 last 57.49) - open/close 3087 (time 1788874815.04 total 50.08 last 64.34) - open/close 3694 (time 1788874825.04 total 60.08 last 60.70) - open/close 4372 (time 1788874835.05 total 70.09 last 67.73) total: 4500 open/close in 72.00 seconds: 62.50 ops/second - /unlink 1261 (time 1788874852.16 total 10.01 last 125.93) - /unlink 2583 (time 1788874862.16 total 20.02 last 132.10) - /unlink 3708 (time 1788874872.16 total 30.02 last 112.49) total: 4500 /unlink in 37.09 seconds: 121.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 645 (time 1788874894.14 total 10.02 last 64.35) - open/close 1263 (time 1788874904.14 total 20.03 last 61.76) - open/close 1912 (time 1788874914.15 total 30.04 last 64.85) - open/close 2575 (time 1788874924.16 total 40.04 last 66.25) - open/close 3143 (time 1788874934.19 total 50.08 last 56.60) - open/close 3756 (time 1788874944.21 total 60.09 last 61.23) - open/close 4295 (time 1788874954.22 total 70.11 last 53.82) total: 4500 open/close in 73.76 seconds: 61.01 ops/second - /unlink 1203 (time 1788874973.87 total 10.00 last 120.28) - /unlink 2472 (time 1788874983.88 total 20.01 last 126.85) - /unlink 3704 (time 1788874993.88 total 30.01 last 123.20) total: 4500 /unlink in 36.51 seconds: 123.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 644 (time 1788875015.87 total 10.02 last 64.26) - open/close 1272 (time 1788875025.88 total 20.03 last 62.73) - open/close 1917 (time 1788875035.89 total 30.04 last 64.46) - open/close 2585 (time 1788875045.89 total 40.04 last 66.76) - open/close 3273 (time 1788875055.91 total 50.06 last 68.67) - open/close 3892 (time 1788875065.93 total 60.08 last 61.80) total: 4500 open/close in 69.85 seconds: 64.42 ops/second - /unlink 1426 (time 1788875090.69 total 10.00 last 142.55) - /unlink 2647 (time 1788875100.69 total 20.01 last 122.07) - /unlink 3939 (time 1788875110.70 total 30.01 last 129.13) total: 4500 /unlink in 34.48 seconds: 130.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 645 (time 1788875130.47 total 10.00 last 64.49) - open/close 1336 (time 1788875140.48 total 20.01 last 69.04) - open/close 1991 (time 1788875150.48 total 30.01 last 65.49) - open/close 2656 (time 1788875160.49 total 40.02 last 66.45) - open/close 3319 (time 1788875170.50 total 50.03 last 66.22) - open/close 4013 (time 1788875180.51 total 60.04 last 69.36) total: 4500 open/close in 66.50 seconds: 67.67 ops/second - /unlink 1226 (time 1788875202.34 total 10.01 last 122.52) - /unlink 2513 (time 1788875212.35 total 20.01 last 128.60) - /unlink 3752 (time 1788875222.35 total 30.02 last 123.85) total: 4500 /unlink in 36.08 seconds: 124.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 629 (time 1788875243.77 total 10.01 last 62.84) - open/close 1174 (time 1788875253.78 total 20.02 last 54.45) - open/close 1715 (time 1788875263.81 total 30.05 last 53.92) - open/close 2265 (time 1788875273.83 total 40.06 last 54.94) - open/close 2833 (time 1788875283.87 total 50.11 last 56.55) - open/close 3514 (time 1788875293.88 total 60.12 last 68.06) - open/close 4227 (time 1788875303.91 total 70.15 last 71.08) total: 4500 open/close in 74.22 seconds: 60.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1200 (time 1788875323.39 total 10.01 last 119.91) - /unlink 2397 (time 1788875333.39 total 20.01 last 119.62) - /unlink 3742 (time 1788875343.41 total 30.03 last 134.30) total: 4500 /unlink in 36.60 seconds: 122.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 672 (time 1788875365.44 total 10.00 last 67.17) - open/close 1335 (time 1788875375.47 total 20.03 last 66.13) - open/close 1946 (time 1788875385.48 total 30.04 last 61.05) - open/close 2518 (time 1788875395.48 total 40.04 last 57.17) - open/close 3149 (time 1788875405.51 total 50.07 last 62.96) - open/close 3830 (time 1788875415.51 total 60.07 last 68.07) total: 4500 open/close in 69.53 seconds: 64.72 ops/second - /unlink 1109 (time 1788875440.22 total 10.00 last 110.89) - /unlink 2350 (time 1788875450.22 total 20.00 last 124.09) - /unlink 3676 (time 1788875460.23 total 30.01 last 132.53) total: 4500 /unlink in 36.47 seconds: 123.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 625 (time 1788875481.49 total 10.01 last 62.45) - open/close 1257 (time 1788875491.52 total 20.04 last 63.01) - open/close 1939 (time 1788875501.52 total 30.05 last 68.16) - open/close 2584 (time 1788875511.53 total 40.05 last 64.48) - open/close 3239 (time 1788875521.53 total 50.05 last 65.49) - open/close 3910 (time 1788875531.53 total 60.05 last 67.06) total: 4500 open/close in 68.62 seconds: 65.58 ops/second - /unlink 1308 (time 1788875555.15 total 10.01 last 130.71) - /unlink 2445 (time 1788875565.16 total 20.01 last 113.63) - /unlink 3679 (time 1788875575.18 total 30.03 last 123.14) total: 4500 /unlink in 35.93 seconds: 125.24 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95311 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules via unload_modules_local on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: kmemleak disabled modules unloaded.