== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:16:07 (1773612967) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg118-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg118-server oleg118-server: oleg118-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 551 (time 1773613171.16 total 10.01 last 55.04) - open/close 1128 (time 1773613181.16 total 20.01 last 57.69) - open/close 1748 (time 1773613191.17 total 30.03 last 61.92) - open/close 2333 (time 1773613201.17 total 40.03 last 58.48) - open/close 2899 (time 1773613211.18 total 50.04 last 56.55) - open/close 3427 (time 1773613221.19 total 60.05 last 52.74) - open/close 3990 (time 1773613231.20 total 70.06 last 56.24) total: 4500 open/close in 78.80 seconds: 57.11 ops/second - /unlink 1009 (time 1773613255.01 total 10.02 last 100.74) - /unlink 2216 (time 1773613265.01 total 20.02 last 120.68) - /unlink 3380 (time 1773613275.02 total 30.02 last 116.34) total: 4500 /unlink in 39.19 seconds: 114.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 604 (time 1773613298.94 total 10.01 last 60.34) - open/close 1210 (time 1773613308.95 total 20.01 last 60.59) - open/close 1742 (time 1773613318.96 total 30.03 last 53.13) - open/close 2352 (time 1773613328.96 total 40.03 last 60.98) - open/close 2969 (time 1773613338.96 total 50.03 last 61.69) - open/close 3592 (time 1773613348.98 total 60.05 last 62.19) - open/close 4168 (time 1773613358.99 total 70.06 last 57.54) - open/close 4461 (time 1773613369.01 total 80.08 last 29.25) total: 4500 open/close in 81.75 seconds: 55.04 ops/second conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7386:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11634:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7762:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7825:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7625:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11640:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1773613389.log kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11177: 94556 Terminated __test_135_file_thread "$service" rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg118-server.1773613389.log.CI90lV" failed: Operation not permitted (1) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11170: 94552 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg118-server.1773613389.log.Qg7AYG" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: modules unloaded.