== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:08:08 (1788854888) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg118-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg118-server oleg118-server: oleg118-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: kmemleak disabled oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 771 (time 1788855123.08 total 10.01 last 77.00) - open/close 1604 (time 1788855133.10 total 20.03 last 83.16) - open/close 2272 (time 1788855143.14 total 30.07 last 66.56) - open/close 2818 (time 1788855153.14 total 40.07 last 54.59) - open/close 3338 (time 1788855163.14 total 50.07 last 51.98) - open/close 3847 (time 1788855173.14 total 60.07 last 50.89) total: 4500 open/close in 68.77 seconds: 65.43 ops/second - /unlink 1418 (time 1788855196.90 total 10.00 last 141.79) - /unlink 2855 (time 1788855206.90 total 20.00 last 143.70) - /unlink 4172 (time 1788855216.90 total 30.00 last 131.67) total: 4500 /unlink in 33.31 seconds: 135.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 567 (time 1788855235.17 total 10.01 last 56.62) - open/close 1140 (time 1788855245.18 total 20.02 last 57.27) - open/close 1756 (time 1788855255.18 total 30.02 last 61.59) - open/close 2170 (time 1788855265.18 total 40.02 last 41.38) - open/close 2621 (time 1788855275.20 total 50.04 last 45.05) - open/close 3242 (time 1788855285.20 total 60.04 last 62.10) - open/close 3885 (time 1788855295.20 total 70.04 last 64.28) total: 4500 open/close in 78.14 seconds: 57.59 ops/second - /unlink 1266 (time 1788855318.65 total 10.01 last 126.43) - /unlink 2220 (time 1788855328.66 total 20.02 last 95.32) - /unlink 3098 (time 1788855338.66 total 30.02 last 87.78) - /unlink 4093 (time 1788855348.66 total 40.02 last 99.50) total: 4500 /unlink in 44.69 seconds: 100.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 473 (time 1788855370.30 total 10.01 last 47.26) - open/close 1177 (time 1788855380.30 total 20.01 last 70.37) - open/close 2042 (time 1788855390.32 total 30.03 last 86.40) - open/close 2747 (time 1788855400.32 total 40.03 last 70.50) - open/close 3396 (time 1788855410.32 total 50.03 last 64.90) - open/close 4016 (time 1788855420.33 total 60.04 last 61.94) total: 4500 open/close in 67.04 seconds: 67.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1231 (time 1788855441.86 total 10.01 last 123.00) - /unlink 2254 (time 1788855451.86 total 20.01 last 102.30) - /unlink 3232 (time 1788855461.88 total 30.03 last 97.64) - /unlink 4065 (time 1788855471.88 total 40.03 last 83.29) total: 4500 /unlink in 44.70 seconds: 100.68 ops/second conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7524:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11725:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7901:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7964:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7763:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11731:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1788855490.log kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11351: 95368 Terminated __test_135_file_thread "$service" rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg118-server.1788855490.log.DaELTK" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg118-server.1788855490.log.pTKGCJ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95364 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: kmemleak disabled modules unloaded.