== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:09:25 (1787465365) Stopping clients: oleg625-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg625-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg625-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg625-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg625-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg625-server oleg625-server: oleg625-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing load_modules_local oleg625-server: Loading modules from /home/green/git/lustre-release/lustre oleg625-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg625-server: detected 4 online CPUs by sysfs oleg625-server: MODOPTS_LIBCFS= oleg625-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg625-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg625-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg625-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 634 (time 1787465597.65 total 10.00 last 63.38) - open/close 1256 (time 1787465607.67 total 20.02 last 62.09) - open/close 1980 (time 1787465617.67 total 30.02 last 72.38) - open/close 2676 (time 1787465627.69 total 40.04 last 69.49) - open/close 3351 (time 1787465637.70 total 50.05 last 67.44) - open/close 4006 (time 1787465647.72 total 60.07 last 65.37) total: 4500 open/close in 68.53 seconds: 65.67 ops/second - /unlink 1496 (time 1787465671.60 total 10.01 last 149.46) - /unlink 2851 (time 1787465681.61 total 20.02 last 135.38) - /unlink 3980 (time 1787465691.61 total 30.02 last 112.86) total: 4500 /unlink in 33.61 seconds: 133.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 669 (time 1787465711.63 total 10.01 last 66.84) - open/close 1232 (time 1787465721.63 total 20.01 last 56.29) - open/close 1882 (time 1787465731.63 total 30.01 last 64.97) - open/close 2463 (time 1787465741.65 total 40.03 last 58.00) - open/close 3010 (time 1787465751.65 total 50.03 last 54.69) - open/close 3629 (time 1787465761.66 total 60.04 last 61.84) - open/close 4317 (time 1787465771.67 total 70.05 last 68.72) total: 4500 open/close in 72.48 seconds: 62.09 ops/second - /unlink 898 (time 1787465791.26 total 10.01 last 89.75) - /unlink 1705 (time 1787465801.27 total 20.01 last 80.67) - /unlink 2575 (time 1787465811.27 total 30.02 last 86.93) - /unlink 3390 (time 1787465821.28 total 40.02 last 81.46) - /unlink 4329 (time 1787465831.28 total 50.03 last 93.86) total: 4500 /unlink in 51.48 seconds: 87.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 572 (time 1787465848.92 total 10.01 last 57.15) - open/close 1197 (time 1787465858.93 total 20.01 last 62.47) - open/close 1703 (time 1787465868.93 total 30.02 last 50.57) - open/close 2323 (time 1787465878.94 total 40.03 last 61.94) - open/close 2923 (time 1787465888.95 total 50.04 last 59.96) - open/close 3619 (time 1787465898.97 total 60.05 last 69.48) - open/close 4322 (time 1787465908.97 total 70.06 last 70.25) total: 4500 open/close in 73.10 seconds: 61.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1263 (time 1787465927.41 total 10.01 last 126.22) - /unlink 2160 (time 1787465937.42 total 20.02 last 89.57) - /unlink 2990 (time 1787465947.43 total 30.03 last 82.93) - /unlink 4035 (time 1787465957.43 total 40.03 last 104.47) total: 4500 /unlink in 44.16 seconds: 101.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 576 (time 1787465977.62 total 10.01 last 57.52) - open/close 1204 (time 1787465987.63 total 20.02 last 62.79) - open/close 1798 (time 1787465997.63 total 30.02 last 59.38) - open/close 2380 (time 1787466007.66 total 40.05 last 58.04) - open/close 3084 (time 1787466017.66 total 50.05 last 70.36) - open/close 3767 (time 1787466027.66 total 60.05 last 68.29) - open/close 4396 (time 1787466037.67 total 70.06 last 62.84) total: 4500 open/close in 71.48 seconds: 62.95 ops/second - /unlink 1008 (time 1787466054.98 total 10.00 last 100.76) - /unlink 2003 (time 1787466064.99 total 20.01 last 99.47) - /unlink 3154 (time 1787466075.00 total 30.02 last 115.00) - /unlink 4324 (time 1787466085.01 total 40.03 last 116.86) total: 4500 /unlink in 41.66 seconds: 108.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 572 (time 1787466102.53 total 10.01 last 57.15) - open/close 1190 (time 1787466112.55 total 20.03 last 61.65) - open/close 1804 (time 1787466122.57 total 30.05 last 61.28) - open/close 2347 (time 1787466132.60 total 40.08 last 54.16) - open/close 2948 (time 1787466142.61 total 50.09 last 60.04) - open/close 3408 (time 1787466152.64 total 60.13 last 45.82) - open/close 3836 (time 1787466162.66 total 70.14 last 42.73) - open/close 4329 (time 1787466172.67 total 80.15 last 49.26) total: 4500 open/close in 84.30 seconds: 53.38 ops/second - /unlink 1190 (time 1787466195.20 total 10.01 last 118.94) - /unlink 2316 (time 1787466205.22 total 20.02 last 112.40) - /unlink 3358 (time 1787466215.22 total 30.03 last 104.18) - /unlink 4409 (time 1787466225.22 total 40.03 last 105.08) total: 4500 /unlink in 41.32 seconds: 108.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 585 (time 1787466245.70 total 10.02 last 58.41) - open/close 1113 (time 1787466255.70 total 20.02 last 52.80) - open/close 1729 (time 1787466265.71 total 30.02 last 61.56) - open/close 2307 (time 1787466275.72 total 40.04 last 57.72) - open/close 2935 (time 1787466285.74 total 50.05 last 62.70) - open/close 3568 (time 1787466295.76 total 60.08 last 63.15) - open/close 4214 (time 1787466305.77 total 70.08 last 64.57) total: 4500 open/close in 74.62 seconds: 60.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1125 (time 1787466327.29 total 10.00 last 112.47) - /unlink 2019 (time 1787466337.29 total 20.01 last 89.35) - /unlink 3039 (time 1787466347.30 total 30.01 last 101.96) - /unlink 4015 (time 1787466357.30 total 40.01 last 97.59) total: 4500 /unlink in 45.49 seconds: 98.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 570 (time 1787466379.32 total 10.01 last 56.95) - open/close 1250 (time 1787466389.33 total 20.02 last 67.91) - open/close 1881 (time 1787466399.33 total 30.03 last 63.06) - open/close 2469 (time 1787466409.34 total 40.04 last 58.75) - open/close 3036 (time 1787466419.35 total 50.04 last 56.68) - open/close 3652 (time 1787466429.35 total 60.04 last 61.57) - open/close 4306 (time 1787466439.36 total 70.05 last 65.38) total: 4500 open/close in 73.04 seconds: 61.61 ops/second - /unlink 1270 (time 1787466458.71 total 10.00 last 126.99) - /unlink 2485 (time 1787466468.72 total 20.01 last 121.41) - /unlink 3797 (time 1787466478.72 total 30.01 last 131.16) total: 4500 /unlink in 35.42 seconds: 127.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 656 (time 1787466499.97 total 10.00 last 65.59) - open/close 1236 (time 1787466509.98 total 20.01 last 57.98) - open/close 1758 (time 1787466519.98 total 30.01 last 52.16) - open/close 2312 (time 1787466530.00 total 40.03 last 55.31) - open/close 2889 (time 1787466540.01 total 50.04 last 57.64) - open/close 3428 (time 1787466550.01 total 60.04 last 53.90) - open/close 3968 (time 1787466560.02 total 70.05 last 53.93) total: 4500 open/close in 79.54 seconds: 56.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1079 (time 1787466585.18 total 10.03 last 107.55) - /unlink 2026 (time 1787466595.18 total 20.04 last 94.67) - /unlink 3073 (time 1787466605.19 total 30.04 last 104.65) - /unlink 4240 (time 1787466615.20 total 40.05 last 116.58) total: 4500 /unlink in 42.61 seconds: 105.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 579 (time 1787466634.38 total 10.08 last 57.42) - open/close 1131 (time 1787466644.39 total 20.09 last 55.16) - open/close 1763 (time 1787466654.39 total 30.09 last 63.17) - open/close 2381 (time 1787466664.41 total 40.11 last 61.72) - open/close 2970 (time 1787466674.41 total 50.11 last 58.87) - open/close 3753 (time 1787466684.41 total 60.11 last 78.29) - open/close 4408 (time 1787466694.42 total 70.12 last 65.49) total: 4500 open/close in 71.58 seconds: 62.87 ops/second - /unlink 1195 (time 1787466710.78 total 10.00 last 119.45) - /unlink 2466 (time 1787466720.79 total 20.01 last 126.99) - /unlink 3814 (time 1787466730.79 total 30.01 last 134.79) total: 4500 /unlink in 36.32 seconds: 123.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 601 (time 1787466753.45 total 10.01 last 60.06) - open/close 1207 (time 1787466763.46 total 20.02 last 60.52) - open/close 1817 (time 1787466773.48 total 30.04 last 60.90) - open/close 2359 (time 1787466783.50 total 40.05 last 54.10) - open/close 2847 (time 1787466793.50 total 50.06 last 48.78) - open/close 3384 (time 1787466803.51 total 60.07 last 53.66) - open/close 3892 (time 1787466813.51 total 70.07 last 50.79) - open/close 4426 (time 1787466823.53 total 80.08 last 53.33) total: 4500 open/close in 81.47 seconds: 55.23 ops/second - /unlink 1085 (time 1787466841.15 total 10.02 last 108.30) - /unlink 1933 (time 1787466851.15 total 20.02 last 84.76) - /unlink 3073 (time 1787466861.15 total 30.02 last 114.00) - /unlink 4380 (time 1787466871.16 total 40.02 last 130.69) total: 4500 /unlink in 41.45 seconds: 108.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 628 (time 1787466888.38 total 10.00 last 62.78) - open/close 1284 (time 1787466898.40 total 20.02 last 65.50) - open/close 1988 (time 1787466908.41 total 30.03 last 70.29) - open/close 2609 (time 1787466918.42 total 40.04 last 62.07) - open/close 3195 (time 1787466928.43 total 50.05 last 58.56) - open/close 3634 (time 1787466938.44 total 60.06 last 43.85) - open/close 4088 (time 1787466948.45 total 70.07 last 45.33) total: 4500 open/close in 78.24 seconds: 57.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 971 (time 1787466973.01 total 10.00 last 97.09) - /unlink 2121 (time 1787466983.03 total 20.02 last 114.79) - /unlink 3274 (time 1787466993.03 total 30.02 last 115.26) - /unlink 4427 (time 1787467003.04 total 40.03 last 115.18) total: 4500 /unlink in 40.75 seconds: 110.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 400 (time 1787467019.83 total 10.01 last 39.98) - open/close 980 (time 1787467029.83 total 20.01 last 57.96) - open/close 1636 (time 1787467039.84 total 30.02 last 65.56) - open/close 2276 (time 1787467049.85 total 40.03 last 63.93) - open/close 2929 (time 1787467059.87 total 50.04 last 65.20) - open/close 3555 (time 1787467069.87 total 60.05 last 62.56) - open/close 4176 (time 1787467079.87 total 70.05 last 62.09) total: 4500 open/close in 75.58 seconds: 59.54 ops/second - /unlink 1139 (time 1787467101.14 total 10.01 last 113.81) - /unlink 2319 (time 1787467111.14 total 20.01 last 117.98) - /unlink 3597 (time 1787467121.15 total 30.01 last 127.76) total: 4500 /unlink in 37.02 seconds: 121.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 544 (time 1787467144.48 total 10.02 last 54.30) - open/close 1085 (time 1787467154.49 total 20.03 last 54.04) - open/close 1759 (time 1787467164.50 total 30.04 last 67.33) - open/close 2380 (time 1787467174.51 total 40.04 last 62.08) - open/close 2992 (time 1787467184.51 total 50.05 last 61.18) - open/close 3756 (time 1787467194.52 total 60.06 last 76.32) - open/close 4425 (time 1787467204.52 total 70.06 last 66.87) total: 4500 open/close in 71.90 seconds: 62.59 ops/second - /unlink 1231 (time 1787467222.41 total 10.01 last 123.00) - /unlink 2495 (time 1787467232.42 total 20.01 last 126.33) - /unlink 3903 (time 1787467242.43 total 30.02 last 140.66) total: 4500 /unlink in 34.90 seconds: 128.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 665 (time 1787467263.60 total 10.01 last 66.43) - open/close 1366 (time 1787467273.60 total 20.01 last 70.09) - open/close 2047 (time 1787467283.63 total 30.04 last 67.93) - open/close 2701 (time 1787467293.68 total 40.09 last 65.07) - open/close 3239 (time 1787467303.68 total 50.09 last 53.76) - open/close 3867 (time 1787467313.69 total 60.10 last 62.75) - open/close 4439 (time 1787467323.69 total 70.10 last 57.20) total: 4500 open/close in 71.20 seconds: 63.21 ops/second - /unlink 984 (time 1787467341.06 total 10.00 last 98.35) - /unlink 2292 (time 1787467351.07 total 20.01 last 130.67) - /unlink 3525 (time 1787467361.08 total 30.02 last 123.23) total: 4500 /unlink in 37.13 seconds: 121.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 620 (time 1787467384.07 total 10.00 last 61.97) - open/close 1309 (time 1787467394.09 total 20.02 last 68.78) - open/close 1854 (time 1787467404.09 total 30.02 last 54.49) - open/close 2392 (time 1787467414.10 total 40.03 last 53.75) - open/close 2888 (time 1787467424.11 total 50.04 last 49.55) - open/close 3438 (time 1787467434.12 total 60.05 last 54.96) - open/close 3961 (time 1787467444.13 total 70.06 last 52.26) total: 4500 open/close in 79.45 seconds: 56.64 ops/second - /unlink 1098 (time 1787467469.22 total 10.01 last 109.64) - /unlink 2440 (time 1787467479.24 total 20.03 last 134.04) - /unlink 3551 (time 1787467489.24 total 30.03 last 111.02) total: 4500 /unlink in 37.17 seconds: 121.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 545 (time 1787467514.31 total 10.00 last 54.49) - open/close 1077 (time 1787467524.32 total 20.01 last 53.14) - open/close 1715 (time 1787467534.33 total 30.02 last 63.75) - open/close 2248 (time 1787467544.34 total 40.04 last 53.22) - open/close 2723 (time 1787467554.38 total 50.07 last 47.32) - open/close 3214 (time 1787467564.39 total 60.08 last 49.06) - open/close 3745 (time 1787467574.41 total 70.10 last 52.98) - open/close 4344 (time 1787467584.43 total 80.12 last 59.82) total: 4500 open/close in 82.86 seconds: 54.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1191 (time 1787467603.45 total 10.01 last 119.02) - /unlink 2080 (time 1787467613.45 total 20.01 last 88.88) - /unlink 3119 (time 1787467623.45 total 30.01 last 103.89) - /unlink 4090 (time 1787467633.46 total 40.01 last 97.07) total: 4500 /unlink in 43.36 seconds: 103.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 706 (time 1787467652.94 total 10.00 last 70.58) - open/close 1301 (time 1787467662.95 total 20.01 last 59.46) - open/close 1985 (time 1787467672.97 total 30.03 last 68.24) - open/close 2498 (time 1787467682.97 total 40.03 last 51.30) - open/close 2947 (time 1787467692.97 total 50.04 last 44.90) - open/close 3544 (time 1787467702.97 total 60.04 last 59.70) - open/close 4198 (time 1787467712.99 total 70.05 last 65.30) total: 4500 open/close in 74.07 seconds: 60.75 ops/second conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11712:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7735:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11718:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1787467730.log kill changelog reader rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg625-server.1787467730.log.hLZR0T" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg625-server.1787467730.log.BWDMpj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95487 Terminated coproc COPROC $LFS changelog --follow $service /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95491 Terminated __test_135_file_thread "$service" lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg625-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg625-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg625-server unloading modules via unload_modules_local on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing unload_modules_local oleg625-server: modules unloaded.