== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:43:08 (1788446588) Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg402-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg402-server oleg402-server: oleg402-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg402-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 677 (time 1788446793.48 total 10.01 last 67.66) - open/close 1333 (time 1788446803.48 total 20.01 last 65.58) - open/close 2060 (time 1788446813.49 total 30.01 last 72.66) - open/close 2715 (time 1788446823.50 total 40.02 last 65.45) - open/close 3391 (time 1788446833.51 total 50.04 last 67.51) - open/close 4087 (time 1788446843.51 total 60.04 last 69.60) total: 4500 open/close in 66.66 seconds: 67.51 ops/second - /unlink 1258 (time 1788446864.87 total 10.01 last 125.74) - /unlink 2504 (time 1788446874.88 total 20.01 last 124.55) - /unlink 3782 (time 1788446884.88 total 30.01 last 127.77) total: 4500 /unlink in 35.58 seconds: 126.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 738 (time 1788446904.98 total 10.00 last 73.77) - open/close 1465 (time 1788446914.98 total 20.00 last 72.70) - open/close 2164 (time 1788446924.98 total 30.01 last 69.89) - open/close 2799 (time 1788446935.00 total 40.02 last 63.40) - open/close 3363 (time 1788446945.02 total 50.05 last 56.25) - open/close 4051 (time 1788446955.04 total 60.07 last 68.68) total: 4500 open/close in 68.05 seconds: 66.13 ops/second - /unlink 1211 (time 1788446977.99 total 10.01 last 121.03) - /unlink 2440 (time 1788446987.99 total 20.01 last 122.88) - /unlink 3697 (time 1788446997.99 total 30.01 last 125.69) total: 4500 /unlink in 36.26 seconds: 124.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 721 (time 1788447018.28 total 10.00 last 72.09) - open/close 1364 (time 1788447028.28 total 20.00 last 64.29) - open/close 2012 (time 1788447038.28 total 30.00 last 64.79) - open/close 2647 (time 1788447048.29 total 40.01 last 63.45) - open/close 3363 (time 1788447058.29 total 50.01 last 71.59) - open/close 3988 (time 1788447068.30 total 60.02 last 62.48) total: 4500 open/close in 67.10 seconds: 67.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1287 (time 1788447090.03 total 10.00 last 128.67) - /unlink 2561 (time 1788447100.04 total 20.00 last 127.37) - /unlink 3693 (time 1788447110.04 total 30.01 last 113.15) total: 4500 /unlink in 37.39 seconds: 120.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 712 (time 1788447132.18 total 10.01 last 71.11) - open/close 1398 (time 1788447142.18 total 20.02 last 68.57) - open/close 2129 (time 1788447152.19 total 30.03 last 73.01) - open/close 2799 (time 1788447162.19 total 40.03 last 66.99) - open/close 3473 (time 1788447172.19 total 50.03 last 67.40) - open/close 4146 (time 1788447182.21 total 60.05 last 67.19) total: 4500 open/close in 66.01 seconds: 68.17 ops/second - /unlink 1239 (time 1788447203.01 total 10.01 last 123.81) - /unlink 2394 (time 1788447213.01 total 20.01 last 115.49) - /unlink 3561 (time 1788447223.01 total 30.01 last 116.63) total: 4500 /unlink in 36.89 seconds: 121.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 669 (time 1788447244.58 total 10.01 last 66.82) - open/close 1349 (time 1788447254.60 total 20.03 last 67.86) - open/close 2046 (time 1788447264.60 total 30.04 last 69.66) - open/close 2694 (time 1788447274.61 total 40.04 last 64.77) - open/close 3384 (time 1788447284.63 total 50.06 last 68.87) - open/close 4055 (time 1788447294.63 total 60.07 last 67.04) total: 4500 open/close in 66.65 seconds: 67.51 ops/second - /unlink 1258 (time 1788447315.93 total 10.01 last 125.72) - /unlink 2550 (time 1788447325.94 total 20.01 last 129.18) - /unlink 3933 (time 1788447335.94 total 30.01 last 138.26) total: 4500 /unlink in 34.26 seconds: 131.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 698 (time 1788447354.40 total 10.00 last 69.79) - open/close 1398 (time 1788447364.41 total 20.01 last 69.94) - open/close 2066 (time 1788447374.41 total 30.02 last 66.76) - open/close 2801 (time 1788447384.42 total 40.02 last 73.46) - open/close 3505 (time 1788447394.44 total 50.04 last 70.26) - open/close 4184 (time 1788447404.46 total 60.06 last 67.75) total: 4500 open/close in 64.79 seconds: 69.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1299 (time 1788447423.33 total 10.02 last 129.70) - /unlink 2344 (time 1788447433.33 total 20.02 last 104.50) - /unlink 3566 (time 1788447443.34 total 30.02 last 122.10) total: 4500 /unlink in 37.59 seconds: 119.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 603 (time 1788447466.32 total 10.01 last 60.26) - open/close 1235 (time 1788447476.33 total 20.01 last 63.15) - open/close 1903 (time 1788447486.34 total 30.02 last 66.73) - open/close 2568 (time 1788447496.35 total 40.03 last 66.44) - open/close 3240 (time 1788447506.35 total 50.04 last 67.18) - open/close 3868 (time 1788447516.37 total 60.06 last 62.68) total: 4500 open/close in 69.32 seconds: 64.92 ops/second - /unlink 1391 (time 1788447540.43 total 10.00 last 139.06) - /unlink 2662 (time 1788447550.44 total 20.01 last 127.03) - /unlink 3749 (time 1788447560.44 total 30.01 last 108.66) total: 4500 /unlink in 36.68 seconds: 122.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 705 (time 1788447582.82 total 10.00 last 70.49) - open/close 1351 (time 1788447592.83 total 20.01 last 64.54) - open/close 2074 (time 1788447602.84 total 30.02 last 72.25) - open/close 2773 (time 1788447612.85 total 40.03 last 69.82) - open/close 3479 (time 1788447622.86 total 50.03 last 70.55) - open/close 4128 (time 1788447632.86 total 60.04 last 64.88) total: 4500 open/close in 65.84 seconds: 68.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1266 (time 1788447653.40 total 10.01 last 126.54) - /unlink 2333 (time 1788447663.40 total 20.01 last 106.67) - /unlink 3631 (time 1788447673.41 total 30.01 last 129.73) total: 4500 /unlink in 37.60 seconds: 119.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 666 (time 1788447697.66 total 10.02 last 66.45) - open/close 1414 (time 1788447707.67 total 20.04 last 74.70) - open/close 2197 (time 1788447717.68 total 30.04 last 78.28) - open/close 2903 (time 1788447727.70 total 40.06 last 70.44) - open/close 3690 (time 1788447737.74 total 50.10 last 78.40) - open/close 4366 (time 1788447747.74 total 60.10 last 67.58) total: 4500 open/close in 62.34 seconds: 72.18 ops/second - /unlink 1245 (time 1788447764.77 total 10.00 last 124.48) - /unlink 2498 (time 1788447774.78 total 20.01 last 125.25) - /unlink 3775 (time 1788447784.78 total 30.01 last 127.66) total: 4500 /unlink in 34.86 seconds: 129.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 650 (time 1788447804.45 total 10.01 last 64.96) - open/close 1350 (time 1788447814.45 total 20.01 last 69.94) - open/close 2070 (time 1788447824.46 total 30.02 last 71.97) - open/close 2791 (time 1788447834.47 total 40.03 last 72.02) - open/close 3526 (time 1788447844.47 total 50.04 last 73.45) - open/close 4273 (time 1788447854.47 total 60.04 last 74.70) total: 4500 open/close in 63.01 seconds: 71.41 ops/second - /unlink 1362 (time 1788447871.25 total 10.01 last 136.13) - /unlink 2772 (time 1788447881.25 total 20.01 last 140.99) - /unlink 4103 (time 1788447891.25 total 30.01 last 133.10) total: 4500 /unlink in 32.72 seconds: 137.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 638 (time 1788447908.06 total 10.01 last 63.75) - open/close 1355 (time 1788447918.07 total 20.02 last 71.63) - open/close 2038 (time 1788447928.07 total 30.02 last 68.27) - open/close 2757 (time 1788447938.11 total 40.06 last 71.63) - open/close 3360 (time 1788447948.13 total 50.08 last 60.19) - open/close 3955 (time 1788447958.15 total 60.10 last 59.36) total: 4500 open/close in 67.87 seconds: 66.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1368 (time 1788447980.51 total 10.01 last 136.73) - /unlink 2516 (time 1788447990.52 total 20.01 last 114.69) - /unlink 3903 (time 1788448000.53 total 30.02 last 138.68) total: 4500 /unlink in 34.37 seconds: 130.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 716 (time 1788448019.30 total 10.06 last 71.20) - open/close 1357 (time 1788448029.30 total 20.06 last 64.08) - open/close 2152 (time 1788448039.32 total 30.07 last 79.40) - open/close 2859 (time 1788448049.32 total 40.08 last 70.66) - open/close 3636 (time 1788448059.32 total 50.08 last 77.70) - open/close 4397 (time 1788448069.32 total 60.08 last 76.10) total: 4500 open/close in 61.57 seconds: 73.09 ops/second - /unlink 1372 (time 1788448085.67 total 10.01 last 137.10) - /unlink 2802 (time 1788448095.67 total 20.01 last 142.99) - /unlink 4236 (time 1788448105.68 total 30.01 last 143.35) total: 4500 /unlink in 32.65 seconds: 137.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 724 (time 1788448123.85 total 10.01 last 72.35) - open/close 1394 (time 1788448133.86 total 20.02 last 66.92) - open/close 2084 (time 1788448143.88 total 30.04 last 68.88) - open/close 2739 (time 1788448153.89 total 40.05 last 65.43) - open/close 3343 (time 1788448163.89 total 50.05 last 60.40) - open/close 3773 (time 1788448173.90 total 60.05 last 42.97) - open/close 4280 (time 1788448183.90 total 70.05 last 50.69) total: 4500 open/close in 74.98 seconds: 60.01 ops/second - /unlink 1129 (time 1788448203.56 total 10.01 last 112.81) - /unlink 2271 (time 1788448213.56 total 20.01 last 114.14) - /unlink 3596 (time 1788448223.57 total 30.02 last 132.42) total: 4500 /unlink in 37.85 seconds: 118.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 637 (time 1788448247.20 total 10.01 last 63.62) - open/close 1303 (time 1788448257.20 total 20.01 last 66.60) - open/close 2016 (time 1788448267.21 total 30.02 last 71.26) - open/close 2823 (time 1788448277.21 total 40.02 last 80.69) - open/close 3514 (time 1788448287.22 total 50.03 last 69.02) - open/close 4126 (time 1788448297.23 total 60.04 last 61.16) total: 4500 open/close in 66.08 seconds: 68.10 ops/second - /unlink 1246 (time 1788448317.23 total 10.00 last 124.58) - /unlink 2408 (time 1788448327.23 total 20.00 last 116.18) - /unlink 3668 (time 1788448337.23 total 30.01 last 125.96) total: 4500 /unlink in 36.85 seconds: 122.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 659 (time 1788448359.95 total 10.01 last 65.83) - open/close 1389 (time 1788448369.97 total 20.04 last 72.82) - open/close 2122 (time 1788448379.99 total 30.05 last 73.18) - open/close 2792 (time 1788448390.00 total 40.06 last 66.96) - open/close 3431 (time 1788448400.01 total 50.07 last 63.84) - open/close 4223 (time 1788448410.01 total 60.07 last 79.14) total: 4500 open/close in 63.87 seconds: 70.46 ops/second - /unlink 1292 (time 1788448428.20 total 10.00 last 129.19) - /unlink 2729 (time 1788448438.21 total 20.01 last 143.56) - /unlink 4068 (time 1788448448.23 total 30.02 last 133.74) total: 4500 /unlink in 33.17 seconds: 135.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 834 (time 1788448465.61 total 10.01 last 83.33) - open/close 1603 (time 1788448475.63 total 20.02 last 76.77) - open/close 2358 (time 1788448485.65 total 30.05 last 75.33) - open/close 3041 (time 1788448495.66 total 40.05 last 68.25) - open/close 3777 (time 1788448505.66 total 50.06 last 73.58) total: 4500 open/close in 59.41 seconds: 75.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1191 (time 1788448530.56 total 10.01 last 118.93) - /unlink 2312 (time 1788448540.57 total 20.02 last 112.00) - /unlink 3380 (time 1788448550.58 total 30.04 last 106.64) total: 4500 /unlink in 38.86 seconds: 115.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 727 (time 1788448574.40 total 10.00 last 72.69) - open/close 1458 (time 1788448584.40 total 20.00 last 73.09) - open/close 2276 (time 1788448594.41 total 30.00 last 81.79) - open/close 3031 (time 1788448604.41 total 40.01 last 75.49) - open/close 3748 (time 1788448614.41 total 50.01 last 71.68) - open/close 4498 (time 1788448624.42 total 60.02 last 74.94) total: 4500 open/close in 60.03 seconds: 74.96 ops/second - /unlink 1342 (time 1788448638.54 total 10.00 last 134.16) - /unlink 2609 (time 1788448648.54 total 20.00 last 126.68) - /unlink 3713 (time 1788448658.54 total 30.01 last 110.38) total: 4500 /unlink in 35.63 seconds: 126.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 739 (time 1788448678.71 total 10.01 last 73.86) - open/close 1518 (time 1788448688.71 total 20.01 last 77.90) - open/close 2182 (time 1788448698.71 total 30.01 last 66.35) - open/close 2947 (time 1788448708.72 total 40.02 last 76.43) - open/close 3702 (time 1788448718.73 total 50.03 last 75.44) - open/close 4466 (time 1788448728.73 total 60.03 last 76.37) total: 4500 open/close in 60.54 seconds: 74.33 ops/second - /unlink 1332 (time 1788448743.89 total 10.00 last 133.14) - /unlink 2636 (time 1788448753.89 total 20.01 last 130.34) - /unlink 4064 (time 1788448763.89 total 30.01 last 142.79) total: 4500 /unlink in 33.86 seconds: 132.90 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95412 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local oleg402-server: modules unloaded.