== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:50:07 (1789267807) Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg314-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: kmemleak disabled oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: MODOPTS_LIBCFS= oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg314-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg314-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg314-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 632 (time 1789268024.04 total 10.03 last 63.01) - open/close 1310 (time 1789268034.05 total 20.04 last 67.72) - open/close 1919 (time 1789268044.06 total 30.04 last 60.89) - open/close 2530 (time 1789268054.07 total 40.06 last 61.02) - open/close 3201 (time 1789268064.08 total 50.06 last 67.06) - open/close 3892 (time 1789268074.08 total 60.07 last 69.06) - open/close 4477 (time 1789268084.09 total 70.07 last 58.47) total: 4500 open/close in 70.50 seconds: 63.83 ops/second - /unlink 1198 (time 1789268100.21 total 10.00 last 119.76) - /unlink 2252 (time 1789268110.21 total 20.01 last 105.35) - /unlink 3199 (time 1789268120.21 total 30.01 last 94.67) - /unlink 4251 (time 1789268130.22 total 40.01 last 105.17) total: 4500 /unlink in 42.29 seconds: 106.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 668 (time 1789268147.71 total 10.01 last 66.77) - open/close 1331 (time 1789268157.72 total 20.02 last 66.23) - open/close 1976 (time 1789268167.74 total 30.03 last 64.41) - open/close 2701 (time 1789268177.74 total 40.03 last 72.49) - open/close 3432 (time 1789268187.75 total 50.04 last 73.05) - open/close 4139 (time 1789268197.75 total 60.04 last 70.66) total: 4500 open/close in 65.23 seconds: 68.99 ops/second - /unlink 1398 (time 1789268217.24 total 10.00 last 139.76) - /unlink 2562 (time 1789268227.24 total 20.01 last 116.36) - /unlink 3678 (time 1789268237.25 total 30.01 last 111.59) total: 4500 /unlink in 36.63 seconds: 122.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 704 (time 1789268258.10 total 10.02 last 70.28) - open/close 1331 (time 1789268268.11 total 20.03 last 62.65) - open/close 2014 (time 1789268278.12 total 30.04 last 68.21) - open/close 2730 (time 1789268288.12 total 40.04 last 71.58) - open/close 3439 (time 1789268298.13 total 50.05 last 70.83) - open/close 4135 (time 1789268308.14 total 60.06 last 69.53) total: 4500 open/close in 65.23 seconds: 68.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1306 (time 1789268327.97 total 10.00 last 130.55) - /unlink 2630 (time 1789268337.97 total 20.01 last 132.33) - /unlink 3946 (time 1789268347.98 total 30.02 last 131.47) total: 4500 /unlink in 34.33 seconds: 131.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 721 (time 1789268367.23 total 10.00 last 72.09) - open/close 1416 (time 1789268377.24 total 20.01 last 69.44) - open/close 2181 (time 1789268387.25 total 30.02 last 76.46) - open/close 2847 (time 1789268397.26 total 40.03 last 66.53) - open/close 3555 (time 1789268407.28 total 50.04 last 70.69) - open/close 4263 (time 1789268417.28 total 60.05 last 70.78) total: 4500 open/close in 63.26 seconds: 71.13 ops/second - /unlink 1254 (time 1789268434.02 total 10.00 last 125.39) - /unlink 2526 (time 1789268444.02 total 20.00 last 127.19) - /unlink 3702 (time 1789268454.03 total 30.01 last 117.53) total: 4500 /unlink in 36.33 seconds: 123.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 677 (time 1789268475.47 total 10.01 last 67.64) - open/close 1434 (time 1789268485.47 total 20.01 last 75.67) - open/close 2084 (time 1789268495.48 total 30.02 last 64.97) - open/close 2620 (time 1789268505.49 total 40.03 last 53.53) - open/close 3319 (time 1789268515.49 total 50.04 last 69.88) - open/close 3971 (time 1789268525.50 total 60.04 last 65.14) total: 4500 open/close in 67.64 seconds: 66.53 ops/second - /unlink 1201 (time 1789268547.53 total 10.00 last 120.06) - /unlink 2557 (time 1789268557.53 total 20.01 last 135.56) - /unlink 3819 (time 1789268567.53 total 30.01 last 126.16) total: 4500 /unlink in 35.66 seconds: 126.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 621 (time 1789268587.40 total 10.00 last 62.08) - open/close 1335 (time 1789268597.42 total 20.01 last 71.32) - open/close 2038 (time 1789268607.42 total 30.02 last 70.30) - open/close 2737 (time 1789268617.43 total 40.02 last 69.84) - open/close 3435 (time 1789268627.45 total 50.05 last 69.65) - open/close 4002 (time 1789268637.46 total 60.06 last 56.65) total: 4500 open/close in 67.16 seconds: 67.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1161 (time 1789268658.93 total 10.00 last 116.06) - /unlink 2450 (time 1789268668.94 total 20.01 last 128.78) - /unlink 3874 (time 1789268678.94 total 30.02 last 142.36) total: 4500 /unlink in 34.81 seconds: 129.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 748 (time 1789268697.86 total 10.00 last 74.79) - open/close 1446 (time 1789268707.88 total 20.02 last 69.69) - open/close 2161 (time 1789268717.89 total 30.03 last 71.44) - open/close 2879 (time 1789268727.89 total 40.03 last 71.76) - open/close 3599 (time 1789268737.90 total 50.04 last 71.95) - open/close 4326 (time 1789268747.90 total 60.04 last 72.69) total: 4500 open/close in 62.48 seconds: 72.02 ops/second - /unlink 1294 (time 1789268764.69 total 10.01 last 129.30) - /unlink 2557 (time 1789268774.69 total 20.01 last 126.28) - /unlink 3846 (time 1789268784.70 total 30.01 last 128.89) total: 4500 /unlink in 34.77 seconds: 129.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 654 (time 1789268804.53 total 10.01 last 65.35) - open/close 1366 (time 1789268814.54 total 20.02 last 71.11) - open/close 1941 (time 1789268824.54 total 30.03 last 57.47) - open/close 2499 (time 1789268834.55 total 40.03 last 55.80) - open/close 3103 (time 1789268844.55 total 50.03 last 60.37) - open/close 3808 (time 1789268854.56 total 60.05 last 70.40) - open/close 4494 (time 1789268864.57 total 70.05 last 68.55) total: 4500 open/close in 70.11 seconds: 64.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1149 (time 1789268880.04 total 10.01 last 114.76) - /unlink 2100 (time 1789268890.06 total 20.03 last 94.98) - /unlink 3034 (time 1789268900.06 total 30.03 last 93.33) - /unlink 4261 (time 1789268910.07 total 40.04 last 122.66) total: 4500 /unlink in 42.44 seconds: 106.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 614 (time 1789268928.13 total 10.03 last 61.24) - open/close 1206 (time 1789268938.14 total 20.04 last 59.14) - open/close 1815 (time 1789268948.15 total 30.05 last 60.80) - open/close 2456 (time 1789268958.17 total 40.07 last 64.00) - open/close 3106 (time 1789268968.18 total 50.08 last 64.92) - open/close 3852 (time 1789268978.19 total 60.09 last 74.54) total: 4500 open/close in 69.22 seconds: 65.01 ops/second - /unlink 1344 (time 1789269001.10 total 10.01 last 134.33) - /unlink 2693 (time 1789269011.10 total 20.01 last 134.88) - /unlink 3983 (time 1789269021.11 total 30.01 last 128.92) total: 4500 /unlink in 33.77 seconds: 133.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 741 (time 1789269039.61 total 10.01 last 74.03) - open/close 1475 (time 1789269049.62 total 20.01 last 73.37) - open/close 2220 (time 1789269059.62 total 30.02 last 74.48) - open/close 2882 (time 1789269069.63 total 40.02 last 66.15) - open/close 3558 (time 1789269079.64 total 50.04 last 67.49) - open/close 4207 (time 1789269089.65 total 60.04 last 64.89) total: 4500 open/close in 64.29 seconds: 69.99 ops/second - /unlink 1399 (time 1789269108.17 total 10.00 last 139.87) - /unlink 2800 (time 1789269118.18 total 20.00 last 140.07) - /unlink 4113 (time 1789269128.18 total 30.01 last 131.28) total: 4500 /unlink in 33.02 seconds: 136.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 667 (time 1789269145.78 total 10.01 last 66.65) - open/close 1375 (time 1789269155.79 total 20.02 last 70.70) - open/close 2071 (time 1789269165.80 total 30.03 last 69.55) - open/close 2806 (time 1789269175.82 total 40.05 last 73.37) - open/close 3566 (time 1789269185.82 total 50.05 last 75.96) - open/close 4332 (time 1789269195.83 total 60.06 last 76.57) total: 4500 open/close in 62.16 seconds: 72.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1443 (time 1789269211.42 total 10.00 last 144.30) - /unlink 2798 (time 1789269221.42 total 20.00 last 135.48) - /unlink 4280 (time 1789269231.43 total 30.00 last 148.16) total: 4500 /unlink in 31.72 seconds: 141.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 686 (time 1789269248.81 total 10.01 last 68.55) - open/close 1329 (time 1789269258.81 total 20.01 last 64.26) - open/close 2030 (time 1789269268.82 total 30.02 last 70.08) - open/close 2756 (time 1789269278.82 total 40.02 last 72.56) - open/close 3509 (time 1789269288.84 total 50.04 last 75.19) - open/close 4310 (time 1789269298.85 total 60.05 last 79.97) total: 4500 open/close in 62.34 seconds: 72.18 ops/second - /unlink 1450 (time 1789269314.76 total 10.00 last 144.99) - /unlink 2714 (time 1789269324.76 total 20.00 last 126.39) - /unlink 4011 (time 1789269334.77 total 30.01 last 129.53) total: 4500 /unlink in 34.04 seconds: 132.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 703 (time 1789269353.23 total 10.01 last 70.26) - open/close 1269 (time 1789269363.25 total 20.03 last 56.49) - open/close 1737 (time 1789269373.26 total 30.04 last 46.75) - open/close 2363 (time 1789269383.28 total 40.06 last 62.47) - open/close 3015 (time 1789269393.31 total 50.08 last 65.03) - open/close 3739 (time 1789269403.32 total 60.09 last 72.35) - open/close 4357 (time 1789269413.32 total 70.09 last 61.77) total: 4500 open/close in 72.14 seconds: 62.38 ops/second - /unlink 1338 (time 1789269429.54 total 10.00 last 133.79) - /unlink 2665 (time 1789269439.55 total 20.00 last 132.65) - /unlink 3962 (time 1789269449.55 total 30.01 last 129.66) total: 4500 /unlink in 34.19 seconds: 131.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 715 (time 1789269469.29 total 10.01 last 71.46) - open/close 1347 (time 1789269479.30 total 20.01 last 63.16) - open/close 2012 (time 1789269489.31 total 30.02 last 66.43) - open/close 2729 (time 1789269499.31 total 40.02 last 71.70) - open/close 3452 (time 1789269509.31 total 50.02 last 72.29) - open/close 4134 (time 1789269519.32 total 60.03 last 68.15) total: 4500 open/close in 65.14 seconds: 69.08 ops/second - /unlink 1325 (time 1789269539.03 total 10.00 last 132.46) - /unlink 2683 (time 1789269549.04 total 20.01 last 135.76) - /unlink 3917 (time 1789269559.05 total 30.02 last 123.24) total: 4500 /unlink in 34.42 seconds: 130.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 690 (time 1789269578.15 total 10.01 last 68.95) - open/close 1340 (time 1789269588.15 total 20.01 last 64.96) - open/close 2040 (time 1789269598.19 total 30.05 last 69.75) - open/close 2776 (time 1789269608.20 total 40.06 last 73.51) - open/close 3590 (time 1789269618.21 total 50.07 last 81.34) - open/close 4278 (time 1789269628.21 total 60.07 last 68.77) total: 4500 open/close in 63.19 seconds: 71.22 ops/second - /unlink 1070 (time 1789269646.85 total 10.00 last 106.97) - /unlink 2045 (time 1789269656.86 total 20.01 last 97.40) - /unlink 3289 (time 1789269666.86 total 30.02 last 124.32) - /unlink 4354 (time 1789269676.88 total 40.04 last 106.29) total: 4500 /unlink in 41.31 seconds: 108.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 723 (time 1789269693.81 total 10.02 last 72.19) - open/close 1375 (time 1789269703.81 total 20.02 last 65.18) - open/close 2061 (time 1789269713.83 total 30.03 last 68.50) - open/close 2769 (time 1789269723.83 total 40.03 last 70.80) - open/close 3450 (time 1789269733.83 total 50.04 last 68.08) - open/close 4090 (time 1789269743.84 total 60.05 last 63.93) total: 4500 open/close in 66.37 seconds: 67.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1335 (time 1789269765.27 total 10.01 last 133.39) - /unlink 2661 (time 1789269775.28 total 20.02 last 132.43) - /unlink 3983 (time 1789269785.29 total 30.03 last 132.13) total: 4500 /unlink in 33.89 seconds: 132.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 779 (time 1789269804.29 total 10.02 last 77.72) - open/close 1380 (time 1789269814.31 total 20.05 last 59.97) - open/close 2057 (time 1789269824.32 total 30.06 last 67.63) - open/close 2577 (time 1789269834.35 total 40.09 last 51.82) - open/close 3158 (time 1789269844.37 total 50.11 last 58.01) - open/close 3820 (time 1789269854.38 total 60.12 last 66.10) - open/close 4489 (time 1789269864.40 total 70.14 last 66.78) total: 4500 open/close in 70.38 seconds: 63.94 ops/second - /unlink 1322 (time 1789269879.52 total 10.00 last 132.15) - /unlink 2605 (time 1789269889.52 total 20.01 last 128.24) - /unlink 3822 (time 1789269899.53 total 30.01 last 121.69) total: 4500 /unlink in 34.93 seconds: 128.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 723 (time 1789269919.13 total 10.01 last 72.25) - open/close 1395 (time 1789269929.13 total 20.01 last 67.16) - open/close 2091 (time 1789269939.14 total 30.02 last 69.56) - open/close 2666 (time 1789269949.15 total 40.03 last 57.44) - open/close 3266 (time 1789269959.16 total 50.04 last 59.92) - open/close 3941 (time 1789269969.17 total 60.05 last 67.43) total: 4500 open/close in 68.85 seconds: 65.36 ops/second - /unlink 1340 (time 1789269994.18 total 10.00 last 133.97) - /unlink 2676 (time 1789270004.18 total 20.00 last 133.58) - /unlink 3972 (time 1789270014.19 total 30.01 last 129.46) total: 4500 /unlink in 34.09 seconds: 132.01 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95276 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server unloading modules via unload_modules_local on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local oleg314-server: kmemleak disabled modules unloaded.