== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:51:36 (1785883896) Stopping clients: oleg223-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg223-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg223-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg223-server oleg223-server: oleg223-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg223-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg223-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg223-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 880 (time 1785884184.75 total 10.00 last 88.00) - open/close 1702 (time 1785884194.75 total 20.00 last 82.17) - open/close 2242 (time 1785884204.76 total 30.02 last 53.92) - open/close 2727 (time 1785884214.80 total 40.05 last 48.33) - open/close 3130 (time 1785884224.82 total 50.07 last 40.21) - open/close 3788 (time 1785884234.83 total 60.09 last 65.73) total: 4500 open/close in 70.07 seconds: 64.22 ops/second - /unlink 1038 (time 1785884260.57 total 10.00 last 103.76) - /unlink 1952 (time 1785884270.61 total 20.04 last 91.09) - /unlink 2910 (time 1785884280.63 total 30.05 last 95.63) - /unlink 3913 (time 1785884290.64 total 40.07 last 100.15) total: 4500 /unlink in 46.36 seconds: 97.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 583 (time 1785884313.05 total 10.01 last 58.26) - open/close 1032 (time 1785884323.05 total 20.01 last 44.89) - open/close 1582 (time 1785884333.05 total 30.01 last 54.99) - open/close 2264 (time 1785884343.06 total 40.02 last 68.16) - open/close 2904 (time 1785884353.08 total 50.04 last 63.87) - open/close 3393 (time 1785884363.08 total 60.05 last 48.86) - open/close 3840 (time 1785884373.09 total 70.05 last 44.67) - open/close 4193 (time 1785884383.10 total 80.06 last 35.26) total: 4500 open/close in 88.53 seconds: 50.83 ops/second - /unlink 847 (time 1785884409.59 total 10.00 last 84.66) - /unlink 1840 (time 1785884419.60 total 20.01 last 99.23) - /unlink 2748 (time 1785884429.61 total 30.02 last 90.73) - /unlink 3763 (time 1785884439.61 total 40.02 last 101.48) - /unlink 4464 (time 1785884449.62 total 50.04 last 70.00) total: 4500 /unlink in 50.57 seconds: 88.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 582 (time 1785884468.75 total 10.02 last 58.11) - open/close 1062 (time 1785884478.76 total 20.03 last 47.95) - open/close 1474 (time 1785884488.77 total 30.03 last 41.18) - open/close 1875 (time 1785884498.79 total 40.05 last 40.02) - open/close 2211 (time 1785884508.80 total 50.06 last 33.58) - open/close 2475 (time 1785884520.56 total 61.82 last 22.44) - open/close 2884 (time 1785884530.56 total 71.82 last 40.89) - open/close 3488 (time 1785884540.59 total 81.85 last 60.27) - open/close 4015 (time 1785884550.59 total 91.85 last 52.67) total: 4500 open/close in 101.48 seconds: 44.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 871 (time 1785884577.02 total 10.02 last 86.95) - /unlink 1572 (time 1785884587.03 total 20.02 last 70.05) - /unlink 2504 (time 1785884597.04 total 30.03 last 93.11) - /unlink 3604 (time 1785884607.05 total 40.04 last 109.93) total: 4500 /unlink in 47.73 seconds: 94.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 464 (time 1785884630.77 total 10.01 last 46.37) - open/close 911 (time 1785884640.81 total 20.05 last 44.50) - open/close 1345 (time 1785884650.82 total 30.06 last 43.38) - open/close 2052 (time 1785884660.82 total 40.06 last 70.65) - open/close 2598 (time 1785884670.84 total 50.08 last 54.53) - open/close 3426 (time 1785884680.85 total 60.10 last 82.65) - open/close 3998 (time 1785884690.86 total 70.10 last 57.16) total: 4500 open/close in 78.90 seconds: 57.03 ops/second - /unlink 1555 (time 1785884716.12 total 10.00 last 155.49) - /unlink 2609 (time 1785884726.12 total 20.00 last 105.37) - /unlink 3939 (time 1785884736.13 total 30.02 last 132.84) total: 4500 /unlink in 35.16 seconds: 127.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 515 (time 1785884758.28 total 10.01 last 51.44) - open/close 999 (time 1785884768.29 total 20.02 last 48.35) - open/close 1691 (time 1785884778.30 total 30.03 last 69.17) - open/close 2214 (time 1785884788.31 total 40.04 last 52.21) - open/close 2793 (time 1785884798.32 total 50.05 last 57.89) - open/close 3454 (time 1785884808.32 total 60.05 last 66.08) - open/close 4091 (time 1785884818.33 total 70.06 last 63.64) total: 4500 open/close in 74.96 seconds: 60.03 ops/second - /unlink 1191 (time 1785884838.83 total 10.01 last 118.98) - /unlink 2673 (time 1785884848.83 total 20.01 last 148.16) - /unlink 3976 (time 1785884858.83 total 30.01 last 130.29) total: 4500 /unlink in 36.02 seconds: 124.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 855 (time 1785884880.88 total 10.01 last 85.42) - open/close 1454 (time 1785884890.90 total 20.03 last 59.78) - open/close 2059 (time 1785884900.92 total 30.04 last 60.42) - open/close 2558 (time 1785884910.92 total 40.05 last 49.86) - open/close 3132 (time 1785884920.93 total 50.06 last 57.35) - open/close 3902 (time 1785884930.95 total 60.08 last 76.86) - open/close 4451 (time 1785884940.97 total 70.10 last 54.78) total: 4500 open/close in 71.06 seconds: 63.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1528 (time 1785884958.75 total 10.00 last 152.75) - /unlink 2444 (time 1785884968.77 total 20.02 last 91.43) - /unlink 3081 (time 1785884978.79 total 30.05 last 63.54) - /unlink 4120 (time 1785884988.80 total 40.05 last 103.86) total: 4500 /unlink in 41.48 seconds: 108.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 450 (time 1785885006.47 total 10.01 last 44.97) - open/close 977 (time 1785885016.49 total 20.03 last 52.60) - open/close 1480 (time 1785885026.49 total 30.03 last 50.26) - open/close 2029 (time 1785885036.51 total 40.05 last 54.82) - open/close 2634 (time 1785885046.52 total 50.06 last 60.44) - open/close 3050 (time 1785885056.60 total 60.14 last 41.25) - open/close 3474 (time 1785885066.62 total 70.16 last 42.31) - open/close 3994 (time 1785885076.63 total 80.17 last 51.96) total: 4500 open/close in 89.73 seconds: 50.15 ops/second - /unlink 1129 (time 1785885102.14 total 10.00 last 112.86) - /unlink 2132 (time 1785885112.15 total 20.01 last 100.23) - /unlink 3118 (time 1785885122.16 total 30.02 last 98.56) - /unlink 3942 (time 1785885132.16 total 40.02 last 82.38) total: 4500 /unlink in 46.36 seconds: 97.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 464 (time 1785885155.32 total 10.00 last 46.39) - open/close 872 (time 1785885165.32 total 20.00 last 40.79) - open/close 1376 (time 1785885175.34 total 30.02 last 50.30) - open/close 2105 (time 1785885185.35 total 40.04 last 72.77) - open/close 2552 (time 1785885195.38 total 50.07 last 44.59) - open/close 3069 (time 1785885205.40 total 60.08 last 51.60) - open/close 3553 (time 1785885215.40 total 70.09 last 48.38) - open/close 4063 (time 1785885225.41 total 80.09 last 50.99) total: 4500 open/close in 87.83 seconds: 51.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1180 (time 1785885248.87 total 10.01 last 117.88) - /unlink 2268 (time 1785885258.87 total 20.01 last 108.79) - /unlink 3460 (time 1785885268.87 total 30.01 last 119.20) - /unlink 4391 (time 1785885278.89 total 40.03 last 92.96) total: 4500 /unlink in 41.73 seconds: 107.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 372 (time 1785885297.65 total 10.02 last 37.13) - open/close 814 (time 1785885307.67 total 20.03 last 44.13) - open/close 1304 (time 1785885317.68 total 30.04 last 48.95) - open/close 1879 (time 1785885327.70 total 40.06 last 57.40) - open/close 2422 (time 1785885337.73 total 50.10 last 54.11) - open/close 2936 (time 1785885347.76 total 60.12 last 51.28) - open/close 3403 (time 1785885357.76 total 70.13 last 46.66) - open/close 4139 (time 1785885367.76 total 80.13 last 73.60) total: 4500 open/close in 85.79 seconds: 52.45 ops/second - /unlink 1146 (time 1785885389.28 total 10.00 last 114.55) - /unlink 2205 (time 1785885399.29 total 20.01 last 105.81) - /unlink 3040 (time 1785885409.30 total 30.03 last 83.40) - /unlink 3991 (time 1785885419.31 total 40.03 last 95.03) total: 4500 /unlink in 48.56 seconds: 92.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 809 (time 1785885447.23 total 10.00 last 80.86) - open/close 1438 (time 1785885457.25 total 20.02 last 62.79) - open/close 2028 (time 1785885467.26 total 30.03 last 58.93) - open/close 2503 (time 1785885477.27 total 40.04 last 47.46) - open/close 2894 (time 1785885487.28 total 50.05 last 39.06) - open/close 3301 (time 1785885497.28 total 60.06 last 40.68) - open/close 3887 (time 1785885507.29 total 70.06 last 58.57) - open/close 4478 (time 1785885517.29 total 80.07 last 59.07) total: 4500 open/close in 80.55 seconds: 55.86 ops/second - /unlink 1064 (time 1785885534.11 total 10.01 last 106.31) - /unlink 2092 (time 1785885544.11 total 20.01 last 102.75) - /unlink 3080 (time 1785885554.12 total 30.02 last 98.77) - /unlink 4219 (time 1785885564.12 total 40.02 last 113.88) total: 4500 /unlink in 41.43 seconds: 108.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 927 (time 1785885581.43 total 10.01 last 92.62) - open/close 1580 (time 1785885591.44 total 20.02 last 65.21) - open/close 2147 (time 1785885601.44 total 30.02 last 56.70) - open/close 2665 (time 1785885611.48 total 40.06 last 51.62) - open/close 3123 (time 1785885621.49 total 50.07 last 45.76) - open/close 3670 (time 1785885631.49 total 60.07 last 54.66) - open/close 4144 (time 1785885641.50 total 70.08 last 47.39) total: 4500 open/close in 76.41 seconds: 58.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1101 (time 1785885663.58 total 10.01 last 110.02) - /unlink 2379 (time 1785885673.58 total 20.01 last 127.79) - /unlink 4250 (time 1785885683.59 total 30.02 last 186.96) total: 4500 /unlink in 31.60 seconds: 142.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 507 (time 1785885702.24 total 10.00 last 50.69) - open/close 1057 (time 1785885712.25 total 20.01 last 54.93) - open/close 1537 (time 1785885722.27 total 30.03 last 47.94) - open/close 2182 (time 1785885732.27 total 40.03 last 64.49) - open/close 2719 (time 1785885742.28 total 50.04 last 53.65) - open/close 3483 (time 1785885752.29 total 60.06 last 76.26) - open/close 4164 (time 1785885762.30 total 70.06 last 68.08) total: 4500 open/close in 74.25 seconds: 60.61 ops/second - /unlink 1223 (time 1785885781.07 total 10.01 last 122.19) - /unlink 2342 (time 1785885791.07 total 20.01 last 111.89) - /unlink 3858 (time 1785885801.07 total 30.01 last 151.55) total: 4500 /unlink in 36.61 seconds: 122.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 553 (time 1785885823.63 total 10.01 last 55.23) - open/close 1181 (time 1785885833.63 total 20.01 last 62.80) - open/close 1854 (time 1785885843.67 total 30.05 last 67.03) - open/close 2478 (time 1785885853.67 total 40.05 last 62.39) - open/close 3036 (time 1785885863.67 total 50.05 last 55.79) - open/close 3515 (time 1785885873.69 total 60.07 last 47.81) - open/close 3894 (time 1785885883.70 total 70.08 last 37.88) - open/close 4317 (time 1785885893.70 total 80.08 last 42.28) total: 4500 open/close in 83.49 seconds: 53.90 ops/second - /unlink 1026 (time 1785885913.17 total 10.00 last 102.55) - /unlink 1879 (time 1785885923.18 total 20.01 last 85.27) - /unlink 2938 (time 1785885933.19 total 30.02 last 105.77) - /unlink 3890 (time 1785885943.19 total 40.02 last 95.17) total: 4500 /unlink in 48.26 seconds: 93.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11704:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7735:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11710:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1785885966.log /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 95484 Terminated __test_135_file_thread "$service" kill changelog reader rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg223-server.1785885966.log.XUq4pZ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg223-server.1785885966.log.HCynI6" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95480 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local oleg223-server: modules unloaded.