== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:01:45 (1785315705) Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg421-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg421-server oleg421-server: oleg421-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg421-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg421-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 550 (time 1785315988.06 total 10.02 last 54.89) - open/close 988 (time 1785315998.06 total 20.02 last 43.79) - open/close 1469 (time 1785316008.09 total 30.06 last 47.93) - open/close 2020 (time 1785316018.10 total 40.06 last 55.08) - open/close 2569 (time 1785316028.11 total 50.07 last 54.85) - open/close 3071 (time 1785316038.11 total 60.08 last 50.16) - open/close 3619 (time 1785316048.13 total 70.09 last 54.72) - open/close 4170 (time 1785316058.14 total 80.10 last 55.04) total: 4500 open/close in 87.52 seconds: 51.42 ops/second - /unlink 916 (time 1785316083.38 total 10.01 last 91.51) - /unlink 1646 (time 1785316093.41 total 20.04 last 72.78) - /unlink 2415 (time 1785316103.43 total 30.06 last 76.77) - /unlink 3459 (time 1785316113.44 total 40.07 last 104.26) - /unlink 4154 (time 1785316123.45 total 50.08 last 69.44) total: 4500 /unlink in 55.22 seconds: 81.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 427 (time 1785316147.19 total 10.02 last 42.62) - open/close 1003 (time 1785316157.27 total 20.09 last 57.19) - open/close 1478 (time 1785316167.27 total 30.09 last 47.48) - open/close 1951 (time 1785316177.27 total 40.10 last 47.27) - open/close 2479 (time 1785316187.29 total 50.12 last 52.71) - open/close 3022 (time 1785316197.31 total 60.14 last 54.20) - open/close 3547 (time 1785316207.36 total 70.18 last 52.26) - open/close 3958 (time 1785316217.36 total 80.18 last 41.09) total: 4500 open/close in 87.77 seconds: 51.27 ops/second - /unlink 952 (time 1785316240.32 total 10.01 last 95.09) - /unlink 1912 (time 1785316250.36 total 20.05 last 95.60) - /unlink 2981 (time 1785316260.36 total 30.06 last 106.86) - /unlink 3968 (time 1785316270.38 total 40.07 last 98.52) total: 4500 /unlink in 44.30 seconds: 101.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 568 (time 1785316291.53 total 10.00 last 56.80) - open/close 1318 (time 1785316301.54 total 20.01 last 74.91) - open/close 1903 (time 1785316311.54 total 30.01 last 58.49) - open/close 2398 (time 1785316321.56 total 40.04 last 49.38) - open/close 2977 (time 1785316331.57 total 50.05 last 57.85) - open/close 3662 (time 1785316341.61 total 60.09 last 68.23) - open/close 4178 (time 1785316351.64 total 70.12 last 51.44) total: 4500 open/close in 77.80 seconds: 57.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1236 (time 1785316376.88 total 10.00 last 123.58) - /unlink 2124 (time 1785316386.89 total 20.01 last 88.68) - /unlink 3123 (time 1785316396.90 total 30.02 last 99.86) - /unlink 4028 (time 1785316406.90 total 40.02 last 90.45) total: 4500 /unlink in 46.06 seconds: 97.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 559 (time 1785316429.18 total 10.02 last 55.78) - open/close 1072 (time 1785316439.19 total 20.03 last 51.28) - open/close 1506 (time 1785316449.20 total 30.04 last 43.33) - open/close 1997 (time 1785316459.22 total 40.06 last 49.00) - open/close 2588 (time 1785316469.22 total 50.06 last 59.10) - open/close 3522 (time 1785316479.22 total 60.06 last 93.36) - open/close 3989 (time 1785316489.24 total 70.08 last 46.61) total: 4500 open/close in 79.60 seconds: 56.53 ops/second - /unlink 435 (time 1785316516.78 total 10.03 last 43.35) - /unlink 1062 (time 1785316526.78 total 20.04 last 62.68) - /unlink 1927 (time 1785316536.79 total 30.04 last 86.49) - /unlink 2814 (time 1785316546.79 total 40.04 last 88.68) - /unlink 3792 (time 1785316556.79 total 50.05 last 97.75) total: 4500 /unlink in 55.18 seconds: 81.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 568 (time 1785316579.43 total 10.02 last 56.68) - open/close 1013 (time 1785316589.44 total 20.03 last 44.46) - open/close 1557 (time 1785316599.44 total 30.04 last 54.36) - open/close 2161 (time 1785316609.46 total 40.05 last 60.30) - open/close 2648 (time 1785316619.47 total 50.07 last 48.63) - open/close 3072 (time 1785316629.51 total 60.10 last 42.26) - open/close 3601 (time 1785316639.54 total 70.14 last 52.72) - open/close 4282 (time 1785316649.55 total 80.15 last 68.02) total: 4500 open/close in 83.49 seconds: 53.90 ops/second - /unlink 883 (time 1785316669.05 total 10.00 last 88.27) - /unlink 1739 (time 1785316679.06 total 20.02 last 85.44) - /unlink 2703 (time 1785316689.07 total 30.03 last 96.31) - /unlink 3418 (time 1785316699.09 total 40.05 last 71.39) - /unlink 4347 (time 1785316709.09 total 50.05 last 92.88) total: 4500 /unlink in 51.00 seconds: 88.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 407 (time 1785316726.51 total 10.03 last 40.59) - open/close 1040 (time 1785316736.51 total 20.03 last 63.29) - open/close 1768 (time 1785316746.52 total 30.03 last 72.78) - open/close 2442 (time 1785316756.53 total 40.04 last 67.33) - open/close 3129 (time 1785316766.53 total 50.04 last 68.69) - open/close 3623 (time 1785316776.54 total 60.05 last 49.35) - open/close 4338 (time 1785316786.56 total 70.08 last 71.34) total: 4500 open/close in 72.38 seconds: 62.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1517 (time 1785316805.54 total 10.02 last 151.41) - /unlink 2510 (time 1785316815.54 total 20.02 last 99.28) - /unlink 3782 (time 1785316825.55 total 30.03 last 127.06) total: 4500 /unlink in 35.93 seconds: 125.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 528 (time 1785316847.93 total 10.02 last 52.69) - open/close 1067 (time 1785316857.95 total 20.04 last 53.81) - open/close 1597 (time 1785316867.98 total 30.07 last 52.85) - open/close 2075 (time 1785316878.00 total 40.08 last 47.72) - open/close 2691 (time 1785316888.01 total 50.09 last 61.55) - open/close 3288 (time 1785316898.01 total 60.10 last 59.66) - open/close 3726 (time 1785316908.02 total 70.10 last 43.79) - open/close 4140 (time 1785316918.02 total 80.11 last 41.39) total: 4500 open/close in 89.63 seconds: 50.21 ops/second conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7448:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11704:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7824:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7887:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7687:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11710:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1785316941.log kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11250: 95594 Terminated __test_135_file_thread "$service" /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11243: 95590 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg421-server.1785316941.log.M4vX2I" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg421-server.1785316941.log.fbgKNj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: modules unloaded.