== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:10:13 (1789441813) Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg348-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server oleg348-server: oleg348-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: kmemleak disabled oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg348-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg348-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 754 (time 1789442022.21 total 10.00 last 75.38) - open/close 1473 (time 1789442032.23 total 20.02 last 71.78) - open/close 2139 (time 1789442042.25 total 30.04 last 66.47) - open/close 2822 (time 1789442052.25 total 40.04 last 68.29) - open/close 3560 (time 1789442062.26 total 50.05 last 73.75) - open/close 4288 (time 1789442072.26 total 60.05 last 72.76) total: 4500 open/close in 63.00 seconds: 71.43 ops/second - /unlink 1360 (time 1789442089.51 total 10.00 last 135.95) - /unlink 2687 (time 1789442099.51 total 20.01 last 132.62) - /unlink 4029 (time 1789442109.51 total 30.01 last 134.19) total: 4500 /unlink in 33.88 seconds: 132.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 747 (time 1789442128.30 total 10.01 last 74.64) - open/close 1461 (time 1789442138.31 total 20.02 last 71.34) - open/close 2220 (time 1789442148.31 total 30.02 last 75.84) - open/close 2938 (time 1789442158.33 total 40.04 last 71.70) - open/close 3660 (time 1789442168.33 total 50.04 last 72.18) - open/close 4436 (time 1789442178.33 total 60.04 last 77.58) total: 4500 open/close in 60.91 seconds: 73.87 ops/second - /unlink 1337 (time 1789442194.35 total 10.00 last 133.68) - /unlink 2815 (time 1789442204.35 total 20.00 last 147.77) - /unlink 4240 (time 1789442214.36 total 30.01 last 142.42) total: 4500 /unlink in 32.11 seconds: 140.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 728 (time 1789442231.24 total 10.01 last 72.72) - open/close 1477 (time 1789442241.26 total 20.02 last 74.81) - open/close 2157 (time 1789442251.26 total 30.03 last 67.97) - open/close 2887 (time 1789442261.27 total 40.03 last 72.96) - open/close 3597 (time 1789442271.27 total 50.04 last 70.98) - open/close 4278 (time 1789442281.27 total 60.04 last 68.06) total: 4500 open/close in 63.04 seconds: 71.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1243 (time 1789442299.08 total 10.00 last 124.29) - /unlink 2476 (time 1789442309.08 total 20.01 last 123.25) - /unlink 3729 (time 1789442319.09 total 30.01 last 125.24) total: 4500 /unlink in 36.33 seconds: 123.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 612 (time 1789442340.12 total 10.03 last 61.03) - open/close 1304 (time 1789442350.13 total 20.04 last 69.13) - open/close 2017 (time 1789442360.13 total 30.04 last 71.26) - open/close 2670 (time 1789442370.14 total 40.06 last 65.22) - open/close 3320 (time 1789442380.15 total 50.06 last 64.97) - open/close 4027 (time 1789442390.15 total 60.06 last 70.66) total: 4500 open/close in 68.26 seconds: 65.93 ops/second - /unlink 1306 (time 1789442412.99 total 10.00 last 130.56) - /unlink 2493 (time 1789442422.99 total 20.00 last 118.69) - /unlink 3829 (time 1789442432.99 total 30.00 last 133.58) total: 4500 /unlink in 34.76 seconds: 129.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 718 (time 1789442452.11 total 10.01 last 71.74) - open/close 1399 (time 1789442462.12 total 20.02 last 68.05) - open/close 2129 (time 1789442472.13 total 30.03 last 72.91) - open/close 2780 (time 1789442482.14 total 40.04 last 65.02) - open/close 3467 (time 1789442492.15 total 50.05 last 68.65) - open/close 4205 (time 1789442502.15 total 60.05 last 73.80) total: 4500 open/close in 64.04 seconds: 70.27 ops/second - /unlink 1410 (time 1789442520.67 total 10.00 last 140.93) - /unlink 2708 (time 1789442530.68 total 20.01 last 129.70) - /unlink 3873 (time 1789442540.68 total 30.01 last 116.48) total: 4500 /unlink in 35.05 seconds: 128.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 609 (time 1789442561.68 total 10.00 last 60.89) - open/close 1254 (time 1789442571.68 total 20.01 last 64.47) - open/close 1994 (time 1789442581.70 total 30.02 last 73.88) - open/close 2591 (time 1789442591.70 total 40.03 last 59.67) - open/close 3224 (time 1789442601.71 total 50.04 last 63.24) - open/close 3925 (time 1789442611.73 total 60.05 last 70.01) total: 4500 open/close in 68.77 seconds: 65.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1268 (time 1789442635.13 total 10.00 last 126.79) - /unlink 2517 (time 1789442645.13 total 20.01 last 124.81) - /unlink 3761 (time 1789442655.14 total 30.01 last 124.33) total: 4500 /unlink in 36.51 seconds: 123.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 642 (time 1789442676.73 total 10.00 last 64.18) - open/close 1338 (time 1789442686.73 total 20.01 last 69.57) - open/close 1928 (time 1789442696.74 total 30.02 last 58.93) - open/close 2556 (time 1789442706.75 total 40.02 last 62.76) - open/close 3139 (time 1789442716.75 total 50.03 last 58.27) - open/close 3768 (time 1789442726.77 total 60.05 last 62.80) - open/close 4405 (time 1789442736.81 total 70.09 last 63.44) total: 4500 open/close in 71.44 seconds: 62.99 ops/second - /unlink 1298 (time 1789442752.65 total 10.01 last 129.69) - /unlink 2505 (time 1789442762.65 total 20.01 last 120.67) - /unlink 3642 (time 1789442772.66 total 30.01 last 113.65) total: 4500 /unlink in 37.11 seconds: 121.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 709 (time 1789442794.44 total 10.01 last 70.81) - open/close 1322 (time 1789442804.45 total 20.02 last 61.26) - open/close 1944 (time 1789442814.46 total 30.02 last 62.17) - open/close 2624 (time 1789442824.46 total 40.03 last 67.98) - open/close 3207 (time 1789442834.46 total 50.03 last 58.29) - open/close 3858 (time 1789442844.46 total 60.03 last 65.08) - open/close 4459 (time 1789442854.49 total 70.06 last 59.95) total: 4500 open/close in 70.85 seconds: 63.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1264 (time 1789442871.10 total 10.01 last 126.25) - /unlink 2383 (time 1789442881.11 total 20.01 last 111.87) - /unlink 3599 (time 1789442891.11 total 30.02 last 121.58) total: 4500 /unlink in 37.34 seconds: 120.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 727 (time 1789442913.39 total 10.00 last 72.68) - open/close 1428 (time 1789442923.39 total 20.01 last 70.04) - open/close 2083 (time 1789442933.41 total 30.03 last 65.37) - open/close 2763 (time 1789442943.42 total 40.03 last 67.98) - open/close 3467 (time 1789442953.42 total 50.03 last 70.39) - open/close 4209 (time 1789442963.43 total 60.04 last 74.13) total: 4500 open/close in 64.39 seconds: 69.88 ops/second - /unlink 1177 (time 1789442982.65 total 10.00 last 117.66) - /unlink 2281 (time 1789442992.67 total 20.02 last 110.17) - /unlink 3442 (time 1789443002.68 total 30.03 last 116.04) total: 4500 /unlink in 39.08 seconds: 115.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 651 (time 1789443027.09 total 10.01 last 65.03) - open/close 1308 (time 1789443037.11 total 20.03 last 65.56) - open/close 1976 (time 1789443047.12 total 30.05 last 66.71) - open/close 2613 (time 1789443057.12 total 40.05 last 63.70) - open/close 3290 (time 1789443067.13 total 50.05 last 67.67) - open/close 3899 (time 1789443077.14 total 60.06 last 60.84) total: 4500 open/close in 69.11 seconds: 65.12 ops/second - /unlink 1278 (time 1789443100.98 total 10.01 last 127.72) - /unlink 2503 (time 1789443110.98 total 20.01 last 122.48) - /unlink 3605 (time 1789443120.99 total 30.02 last 110.09) total: 4500 /unlink in 37.73 seconds: 119.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 696 (time 1789443143.64 total 10.01 last 69.56) - open/close 1367 (time 1789443153.65 total 20.01 last 67.05) - open/close 2064 (time 1789443163.65 total 30.02 last 69.66) - open/close 2767 (time 1789443173.66 total 40.02 last 70.25) - open/close 3377 (time 1789443183.67 total 50.03 last 60.94) - open/close 4056 (time 1789443193.68 total 60.04 last 67.84) total: 4500 open/close in 67.57 seconds: 66.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1214 (time 1789443217.44 total 10.01 last 121.32) - /unlink 2545 (time 1789443227.45 total 20.01 last 133.03) - /unlink 3843 (time 1789443237.45 total 30.01 last 129.79) total: 4500 /unlink in 35.30 seconds: 127.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 683 (time 1789443256.90 total 10.00 last 68.28) - open/close 1361 (time 1789443266.91 total 20.01 last 67.74) - open/close 2063 (time 1789443276.93 total 30.03 last 70.08) - open/close 2795 (time 1789443286.93 total 40.03 last 73.18) - open/close 3493 (time 1789443296.94 total 50.04 last 69.75) - open/close 4222 (time 1789443306.94 total 60.04 last 72.86) total: 4500 open/close in 63.81 seconds: 70.52 ops/second - /unlink 1400 (time 1789443325.35 total 10.01 last 139.92) - /unlink 2750 (time 1789443335.36 total 20.01 last 134.94) - /unlink 4023 (time 1789443345.36 total 30.01 last 127.27) total: 4500 /unlink in 33.23 seconds: 135.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 604 (time 1789443362.55 total 10.02 last 60.29) - open/close 1189 (time 1789443372.55 total 20.02 last 58.50) - open/close 1858 (time 1789443382.55 total 30.02 last 66.89) - open/close 2486 (time 1789443392.55 total 40.02 last 62.79) - open/close 3085 (time 1789443402.59 total 50.06 last 59.70) - open/close 3736 (time 1789443412.59 total 60.06 last 65.10) - open/close 4341 (time 1789443422.60 total 70.07 last 60.43) total: 4500 open/close in 73.60 seconds: 61.14 ops/second - /unlink 1082 (time 1789443443.03 total 10.01 last 108.12) - /unlink 2192 (time 1789443453.04 total 20.01 last 110.94) - /unlink 3423 (time 1789443463.04 total 30.02 last 123.02) total: 4500 /unlink in 38.60 seconds: 116.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 554 (time 1789443487.93 total 10.01 last 55.36) - open/close 1174 (time 1789443497.94 total 20.02 last 61.94) - open/close 1843 (time 1789443507.96 total 30.03 last 66.81) - open/close 2531 (time 1789443517.97 total 40.04 last 68.71) - open/close 3203 (time 1789443527.98 total 50.05 last 67.15) - open/close 3855 (time 1789443537.98 total 60.06 last 65.17) total: 4500 open/close in 69.31 seconds: 64.92 ops/second - /unlink 1300 (time 1789443561.81 total 10.00 last 130.00) - /unlink 2587 (time 1789443571.81 total 20.00 last 128.65) - /unlink 3857 (time 1789443581.82 total 30.01 last 126.96) total: 4500 /unlink in 34.93 seconds: 128.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 723 (time 1789443601.92 total 10.00 last 72.29) - open/close 1429 (time 1789443611.94 total 20.02 last 70.49) - open/close 2121 (time 1789443621.94 total 30.02 last 69.16) - open/close 2822 (time 1789443631.95 total 40.03 last 70.03) - open/close 3588 (time 1789443641.96 total 50.04 last 76.52) - open/close 4282 (time 1789443651.98 total 60.06 last 69.30) total: 4500 open/close in 63.02 seconds: 71.40 ops/second - /unlink 1239 (time 1789443669.61 total 10.00 last 123.87) - /unlink 2423 (time 1789443679.61 total 20.01 last 118.36) - /unlink 3696 (time 1789443689.62 total 30.01 last 127.27) total: 4500 /unlink in 36.80 seconds: 122.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 675 (time 1789443711.71 total 10.00 last 67.48) - open/close 1376 (time 1789443721.72 total 20.01 last 70.07) - open/close 2054 (time 1789443731.72 total 30.02 last 67.74) - open/close 2801 (time 1789443741.73 total 40.03 last 74.63) - open/close 3507 (time 1789443751.74 total 50.03 last 70.58) - open/close 4242 (time 1789443761.76 total 60.05 last 73.34) total: 4500 open/close in 63.39 seconds: 70.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1191 (time 1789443779.59 total 10.00 last 119.06) - /unlink 2426 (time 1789443789.59 total 20.00 last 123.50) - /unlink 3686 (time 1789443799.59 total 30.01 last 125.94) total: 4500 /unlink in 36.39 seconds: 123.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 649 (time 1789443821.24 total 10.01 last 64.84) - open/close 1364 (time 1789443831.24 total 20.01 last 71.49) - open/close 2061 (time 1789443841.24 total 30.01 last 69.69) - open/close 2775 (time 1789443851.25 total 40.02 last 71.34) - open/close 3480 (time 1789443861.26 total 50.03 last 70.45) - open/close 4180 (time 1789443871.26 total 60.03 last 69.98) total: 4500 open/close in 64.66 seconds: 69.59 ops/second - /unlink 1314 (time 1789443890.09 total 10.00 last 131.35) - /unlink 2647 (time 1789443900.10 total 20.02 last 133.12) - /unlink 3939 (time 1789443910.10 total 30.02 last 129.17) total: 4500 /unlink in 34.05 seconds: 132.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 721 (time 1789443928.61 total 10.01 last 72.05) - open/close 1462 (time 1789443938.62 total 20.01 last 74.07) - open/close 2208 (time 1789443948.62 total 30.01 last 74.58) - open/close 2960 (time 1789443958.62 total 40.02 last 75.18) - open/close 3665 (time 1789443968.64 total 50.03 last 70.41) - open/close 4406 (time 1789443978.65 total 60.05 last 73.97) total: 4500 open/close in 61.33 seconds: 73.38 ops/second - /unlink 1324 (time 1789443994.92 total 10.01 last 132.33) - /unlink 2665 (time 1789444004.93 total 20.01 last 133.99) - /unlink 3963 (time 1789444014.93 total 30.02 last 129.74) total: 4500 /unlink in 34.13 seconds: 131.84 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11350: 95238 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: kmemleak disabled modules unloaded.