== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:01:19 (1783951279) Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg328-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg328-server oleg328-server: oleg328-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg328-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg328-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg328-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 675 (time 1783951495.32 total 10.00 last 67.47) - open/close 1371 (time 1783951505.33 total 20.01 last 69.56) - open/close 2049 (time 1783951515.34 total 30.02 last 67.75) - open/close 2786 (time 1783951525.34 total 40.03 last 73.65) - open/close 3437 (time 1783951535.35 total 50.03 last 65.09) - open/close 4169 (time 1783951545.36 total 60.04 last 73.13) total: 4500 open/close in 65.12 seconds: 69.11 ops/second - /unlink 1273 (time 1783951564.98 total 10.01 last 127.22) - /unlink 2531 (time 1783951574.98 total 20.01 last 125.77) - /unlink 3739 (time 1783951584.98 total 30.01 last 120.77) total: 4500 /unlink in 36.54 seconds: 123.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 680 (time 1783951606.22 total 10.01 last 67.92) - open/close 1353 (time 1783951616.22 total 20.01 last 67.29) - open/close 1985 (time 1783951626.24 total 30.03 last 63.07) - open/close 2659 (time 1783951636.25 total 40.04 last 67.34) - open/close 3330 (time 1783951646.26 total 50.05 last 67.07) - open/close 4029 (time 1783951656.27 total 60.06 last 69.82) total: 4500 open/close in 67.03 seconds: 67.13 ops/second - /unlink 1180 (time 1783951677.48 total 10.00 last 117.96) - /unlink 2467 (time 1783951687.48 total 20.00 last 128.68) - /unlink 3828 (time 1783951697.49 total 30.01 last 135.98) total: 4500 /unlink in 35.41 seconds: 127.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 637 (time 1783951718.47 total 10.01 last 63.65) - open/close 1330 (time 1783951728.51 total 20.05 last 69.04) - open/close 2014 (time 1783951738.52 total 30.06 last 68.33) - open/close 2718 (time 1783951748.53 total 40.06 last 70.36) - open/close 3378 (time 1783951758.54 total 50.07 last 65.93) - open/close 4027 (time 1783951768.54 total 60.08 last 64.88) total: 4500 open/close in 65.99 seconds: 68.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1250 (time 1783951789.14 total 10.00 last 125.00) - /unlink 2379 (time 1783951799.15 total 20.01 last 112.82) - /unlink 3485 (time 1783951809.16 total 30.02 last 110.47) total: 4500 /unlink in 39.60 seconds: 113.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 575 (time 1783951835.32 total 10.01 last 57.44) - open/close 1173 (time 1783951845.32 total 20.01 last 59.80) - open/close 1848 (time 1783951855.32 total 30.01 last 67.49) - open/close 2577 (time 1783951865.33 total 40.02 last 72.84) - open/close 3206 (time 1783951875.33 total 50.02 last 62.90) - open/close 3907 (time 1783951885.34 total 60.03 last 70.04) total: 4500 open/close in 68.54 seconds: 65.66 ops/second - /unlink 1239 (time 1783951908.28 total 10.01 last 123.82) - /unlink 2457 (time 1783951918.29 total 20.02 last 121.67) - /unlink 3694 (time 1783951928.30 total 30.02 last 123.69) total: 4500 /unlink in 36.43 seconds: 123.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 662 (time 1783951949.70 total 10.02 last 66.07) - open/close 1312 (time 1783951959.71 total 20.03 last 64.96) - open/close 1968 (time 1783951969.71 total 30.03 last 65.56) - open/close 2699 (time 1783951979.72 total 40.03 last 73.09) - open/close 3371 (time 1783951989.72 total 50.04 last 67.18) - open/close 4097 (time 1783951999.73 total 60.04 last 72.54) total: 4500 open/close in 65.46 seconds: 68.74 ops/second - /unlink 1273 (time 1783952019.28 total 10.01 last 127.21) - /unlink 2470 (time 1783952029.29 total 20.02 last 119.59) - /unlink 3755 (time 1783952039.29 total 30.02 last 128.49) total: 4500 /unlink in 36.47 seconds: 123.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 561 (time 1783952061.38 total 10.00 last 56.08) - open/close 1177 (time 1783952071.39 total 20.01 last 61.58) - open/close 1885 (time 1783952081.40 total 30.01 last 70.74) - open/close 2582 (time 1783952091.40 total 40.02 last 69.68) - open/close 3243 (time 1783952101.40 total 50.02 last 66.09) - open/close 3902 (time 1783952111.42 total 60.04 last 65.80) total: 4500 open/close in 68.54 seconds: 65.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1349 (time 1783952134.53 total 10.00 last 134.87) - /unlink 2582 (time 1783952144.53 total 20.01 last 123.24) - /unlink 3787 (time 1783952154.53 total 30.01 last 120.44) total: 4500 /unlink in 36.32 seconds: 123.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 453 (time 1783952176.50 total 10.01 last 45.25) - open/close 1004 (time 1783952186.50 total 20.01 last 55.08) - open/close 1612 (time 1783952196.51 total 30.02 last 60.77) - open/close 2227 (time 1783952206.52 total 40.03 last 61.41) - open/close 2835 (time 1783952216.52 total 50.03 last 60.80) - open/close 3385 (time 1783952226.56 total 60.07 last 54.82) - open/close 3968 (time 1783952236.57 total 70.08 last 58.24) total: 4500 open/close in 78.35 seconds: 57.43 ops/second - /unlink 1183 (time 1783952260.41 total 10.01 last 118.18) - /unlink 2428 (time 1783952270.42 total 20.02 last 124.39) - /unlink 3490 (time 1783952280.42 total 30.02 last 106.19) total: 4500 /unlink in 38.32 seconds: 117.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 624 (time 1783952303.89 total 10.02 last 62.25) - open/close 1183 (time 1783952313.89 total 20.03 last 55.88) - open/close 1717 (time 1783952323.90 total 30.03 last 53.36) - open/close 2427 (time 1783952333.90 total 40.04 last 71.00) - open/close 3120 (time 1783952343.91 total 50.05 last 69.19) - open/close 3716 (time 1783952353.94 total 60.07 last 59.47) - open/close 4347 (time 1783952363.94 total 70.07 last 63.09) total: 4500 open/close in 72.62 seconds: 61.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1293 (time 1783952381.70 total 10.00 last 129.29) - /unlink 2526 (time 1783952391.71 total 20.01 last 123.24) - /unlink 3741 (time 1783952401.72 total 30.02 last 121.34) total: 4500 /unlink in 36.35 seconds: 123.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 667 (time 1783952423.12 total 10.00 last 66.67) - open/close 1343 (time 1783952433.14 total 20.02 last 67.48) - open/close 1977 (time 1783952443.15 total 30.03 last 63.37) - open/close 2542 (time 1783952453.16 total 40.04 last 56.43) - open/close 3174 (time 1783952463.16 total 50.04 last 63.19) - open/close 3865 (time 1783952473.16 total 60.04 last 69.09) total: 4500 open/close in 68.91 seconds: 65.30 ops/second - /unlink 1274 (time 1783952496.80 total 10.00 last 127.40) - /unlink 2543 (time 1783952506.80 total 20.00 last 126.90) - /unlink 3706 (time 1783952516.81 total 30.00 last 116.25) total: 4500 /unlink in 37.33 seconds: 120.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 625 (time 1783952539.76 total 10.01 last 62.43) - open/close 1319 (time 1783952549.77 total 20.02 last 69.35) - open/close 2059 (time 1783952559.77 total 30.02 last 74.00) - open/close 2733 (time 1783952569.78 total 40.03 last 67.34) - open/close 3425 (time 1783952579.79 total 50.04 last 69.13) - open/close 4053 (time 1783952589.79 total 60.04 last 62.77) total: 4500 open/close in 67.26 seconds: 66.90 ops/second - /unlink 1169 (time 1783952613.12 total 10.00 last 116.85) - /unlink 2266 (time 1783952623.12 total 20.00 last 109.70) - /unlink 3439 (time 1783952633.12 total 30.01 last 117.28) total: 4500 /unlink in 39.91 seconds: 112.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 672 (time 1783952658.37 total 10.00 last 67.19) - open/close 1364 (time 1783952668.38 total 20.01 last 69.16) - open/close 2076 (time 1783952678.39 total 30.02 last 71.13) - open/close 2803 (time 1783952688.42 total 40.04 last 72.50) - open/close 3522 (time 1783952698.42 total 50.05 last 71.86) - open/close 4178 (time 1783952708.43 total 60.05 last 65.57) total: 4500 open/close in 64.97 seconds: 69.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1281 (time 1783952728.04 total 10.00 last 128.04) - /unlink 2515 (time 1783952738.04 total 20.01 last 123.38) - /unlink 3728 (time 1783952748.04 total 30.01 last 121.28) total: 4500 /unlink in 36.51 seconds: 123.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 680 (time 1783952770.01 total 10.01 last 67.92) - open/close 1322 (time 1783952780.02 total 20.01 last 64.18) - open/close 2011 (time 1783952790.05 total 30.04 last 68.70) - open/close 2664 (time 1783952800.05 total 40.04 last 65.30) - open/close 3359 (time 1783952810.06 total 50.05 last 69.44) - open/close 4050 (time 1783952820.06 total 60.06 last 69.06) total: 4500 open/close in 66.74 seconds: 67.43 ops/second - /unlink 1303 (time 1783952840.45 total 10.00 last 130.27) - /unlink 2601 (time 1783952850.46 total 20.00 last 129.77) - /unlink 3733 (time 1783952860.46 total 30.01 last 113.12) total: 4500 /unlink in 35.88 seconds: 125.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 695 (time 1783952881.61 total 10.02 last 69.35) - open/close 1460 (time 1783952891.62 total 20.02 last 76.48) - open/close 2252 (time 1783952901.62 total 30.03 last 79.14) - open/close 2988 (time 1783952911.62 total 40.03 last 73.58) - open/close 3736 (time 1783952921.64 total 50.04 last 74.72) - open/close 4459 (time 1783952931.65 total 60.05 last 72.22) total: 4500 open/close in 60.76 seconds: 74.07 ops/second - /unlink 1121 (time 1783952947.30 total 10.01 last 112.03) - /unlink 2235 (time 1783952957.32 total 20.02 last 111.22) - /unlink 3285 (time 1783952967.32 total 30.02 last 105.00) - /unlink 4443 (time 1783952977.32 total 40.03 last 115.75) total: 4500 /unlink in 40.50 seconds: 111.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 663 (time 1783952992.81 total 10.04 last 66.07) - open/close 1331 (time 1783953002.81 total 20.04 last 66.77) - open/close 2101 (time 1783953012.82 total 30.05 last 76.94) - open/close 2869 (time 1783953022.82 total 40.05 last 76.79) - open/close 3563 (time 1783953032.83 total 50.06 last 69.33) - open/close 4271 (time 1783953042.84 total 60.07 last 70.74) total: 4500 open/close in 63.21 seconds: 71.20 ops/second - /unlink 1235 (time 1783953059.87 total 10.00 last 123.49) - /unlink 2368 (time 1783953069.88 total 20.01 last 113.22) - /unlink 3611 (time 1783953079.88 total 30.01 last 124.23) total: 4500 /unlink in 36.69 seconds: 122.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 608 (time 1783953100.60 total 10.01 last 60.76) - open/close 1202 (time 1783953110.62 total 20.03 last 59.28) - open/close 1751 (time 1783953120.62 total 30.03 last 54.87) - open/close 2515 (time 1783953130.62 total 40.03 last 76.40) - open/close 3270 (time 1783953140.63 total 50.04 last 75.48) - open/close 3960 (time 1783953150.63 total 60.04 last 69.00) total: 4500 open/close in 68.41 seconds: 65.78 ops/second - /unlink 1312 (time 1783953173.55 total 10.00 last 131.19) - /unlink 2513 (time 1783953183.55 total 20.01 last 120.03) - /unlink 3700 (time 1783953193.55 total 30.01 last 118.70) total: 4500 /unlink in 36.22 seconds: 124.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 724 (time 1783953214.66 total 10.01 last 72.34) - open/close 1388 (time 1783953224.67 total 20.02 last 66.34) - open/close 2073 (time 1783953234.67 total 30.02 last 68.49) - open/close 2667 (time 1783953244.68 total 40.03 last 59.34) - open/close 3301 (time 1783953254.69 total 50.03 last 63.37) - open/close 3954 (time 1783953264.69 total 60.04 last 65.26) total: 4500 open/close in 68.27 seconds: 65.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1203 (time 1783953287.43 total 10.01 last 120.21) - /unlink 2445 (time 1783953297.44 total 20.02 last 124.09) - /unlink 3572 (time 1783953307.44 total 30.02 last 112.66) total: 4500 /unlink in 38.63 seconds: 116.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 516 (time 1783953331.56 total 10.01 last 51.54) - open/close 1061 (time 1783953341.57 total 20.02 last 54.45) - open/close 1579 (time 1783953351.57 total 30.02 last 51.78) - open/close 2251 (time 1783953361.58 total 40.03 last 67.14) - open/close 2817 (time 1783953371.60 total 50.05 last 56.51) - open/close 3401 (time 1783953381.61 total 60.06 last 58.31) - open/close 4089 (time 1783953391.62 total 70.07 last 68.73) total: 4500 open/close in 76.94 seconds: 58.49 ops/second - /unlink 1240 (time 1783953413.61 total 10.00 last 123.99) - /unlink 2423 (time 1783953423.61 total 20.01 last 118.22) - /unlink 3654 (time 1783953433.62 total 30.02 last 122.99) total: 4500 /unlink in 36.80 seconds: 122.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 687 (time 1783953454.99 total 10.00 last 68.68) - open/close 1370 (time 1783953465.00 total 20.01 last 68.23) - open/close 2054 (time 1783953475.01 total 30.02 last 68.38) - open/close 2724 (time 1783953485.01 total 40.02 last 66.96) - open/close 3428 (time 1783953495.02 total 50.03 last 70.38) - open/close 4087 (time 1783953505.02 total 60.03 last 65.87) total: 4500 open/close in 66.06 seconds: 68.12 ops/second - /unlink 1182 (time 1783953525.77 total 10.00 last 118.16) - /unlink 2363 (time 1783953535.77 total 20.01 last 118.07) - /unlink 3631 (time 1783953545.78 total 30.01 last 126.75) total: 4500 /unlink in 36.80 seconds: 122.27 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95312 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg328-server unloading modules via unload_modules_local on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local oleg328-server: modules unloaded.