== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:49:11 (1783147751) Stopping clients: oleg405-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg405-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg405-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg405-server oleg405-server: oleg405-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg405-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg405-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog total: 4500 open/close in 9.44 seconds: 476.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #12998 total: 4500 /unlink in 5.14 seconds: 875.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 total: 4500 open/close in 9.68 seconds: 464.91 ops/second total: 4500 /unlink in 5.52 seconds: 815.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 4419 (time 1783147895.08 total 10.00 last 441.89) total: 4500 open/close in 10.19 seconds: 441.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 total: 4500 /unlink in 5.33 seconds: 844.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 total: 4500 open/close in 9.44 seconds: 476.47 ops/second total: 4500 /unlink in 5.11 seconds: 881.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 total: 4500 open/close in 9.70 seconds: 463.97 ops/second total: 4500 /unlink in 5.32 seconds: 846.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 total: 4500 open/close in 9.87 seconds: 455.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 total: 4500 /unlink in 5.31 seconds: 847.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 total: 4500 open/close in 9.81 seconds: 458.61 ops/second total: 4500 /unlink in 5.57 seconds: 807.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 total: 4500 open/close in 9.70 seconds: 463.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 total: 4500 /unlink in 5.40 seconds: 832.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 total: 4500 open/close in 9.64 seconds: 466.84 ops/second total: 4500 /unlink in 5.18 seconds: 868.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 total: 4500 open/close in 9.75 seconds: 461.77 ops/second total: 4500 /unlink in 6.36 seconds: 707.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 3614 (time 1783148036.14 total 10.03 last 360.40) total: 4500 open/close in 12.06 seconds: 373.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 total: 4500 /unlink in 8.13 seconds: 553.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 2723 (time 1783148059.30 total 10.00 last 272.21) total: 4500 open/close in 15.37 seconds: 292.79 ops/second total: 4500 /unlink in 7.30 seconds: 616.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 2572 (time 1783148085.25 total 10.00 last 257.14) total: 4500 open/close in 16.85 seconds: 267.06 ops/second total: 4500 /unlink in 8.45 seconds: 532.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 - open/close 3520 (time 1783148113.78 total 10.00 last 351.89) total: 4500 open/close in 13.14 seconds: 342.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #246998 total: 4500 /unlink in 8.94 seconds: 503.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 2396 (time 1783148139.35 total 10.00 last 239.56) total: 4500 open/close in 18.35 seconds: 245.27 ops/second total: 4500 /unlink in 7.50 seconds: 600.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 3194 (time 1783148168.64 total 10.02 last 318.87) total: 4500 open/close in 14.11 seconds: 318.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 total: 4500 /unlink in 7.41 seconds: 607.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 2530 (time 1783148193.15 total 10.01 last 252.87) - open/close 4296 (time 1783148203.16 total 20.01 last 176.50) total: 4500 open/close in 21.10 seconds: 213.24 ops/second - /unlink 2784 (time 1783148216.31 total 10.00 last 278.35) - /unlink 3299 (time 1783148226.32 total 20.01 last 51.44) - /unlink 3799 (time 1783148236.33 total 30.03 last 49.92) - /unlink 4110 (time 1783148246.51 total 40.21 last 30.56) - /unlink 4488 (time 1783148256.57 total 50.26 last 37.58) total: 4500 /unlink in 50.68 seconds: 88.80 ops/second conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11678:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7686:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11684:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1783148274.log /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 93997 Terminated __test_135_file_thread "$service" kill changelog reader rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg405-server.1783148274.log.6OcKIK" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg405-server.1783148274.log.nlrWmj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 93993 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: modules unloaded.