== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:10:10 (1788271810) Stopping clients: oleg138-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg138-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg138-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg138-server oleg138-server: oleg138-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg138-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg138-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg138-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 876 (time 1788272000.69 total 10.01 last 87.50) - open/close 1798 (time 1788272010.70 total 20.02 last 92.16) - open/close 2693 (time 1788272020.70 total 30.02 last 89.43) - open/close 3682 (time 1788272030.71 total 40.03 last 98.84) total: 4500 open/close in 49.01 seconds: 91.82 ops/second - /unlink 1845 (time 1788272053.25 total 10.00 last 184.45) - /unlink 3533 (time 1788272063.25 total 20.00 last 168.77) total: 4500 /unlink in 25.62 seconds: 175.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 919 (time 1788272082.58 total 10.00 last 91.89) - open/close 1769 (time 1788272092.59 total 20.01 last 84.94) - open/close 2778 (time 1788272102.60 total 30.02 last 100.81) - open/close 3797 (time 1788272112.60 total 40.02 last 101.85) total: 4500 open/close in 47.52 seconds: 94.69 ops/second - /unlink 1801 (time 1788272133.93 total 10.00 last 180.10) - /unlink 3299 (time 1788272143.93 total 20.00 last 149.79) total: 4500 /unlink in 27.10 seconds: 166.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 930 (time 1788272164.86 total 10.01 last 92.92) - open/close 1806 (time 1788272174.87 total 20.01 last 87.58) - open/close 2696 (time 1788272184.89 total 30.03 last 88.82) - open/close 3580 (time 1788272194.89 total 40.04 last 88.33) total: 4500 open/close in 49.69 seconds: 90.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1843 (time 1788272218.10 total 10.00 last 184.25) - /unlink 3789 (time 1788272228.11 total 20.01 last 194.51) total: 4500 /unlink in 23.72 seconds: 189.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 1024 (time 1788272246.15 total 10.00 last 102.37) - open/close 1976 (time 1788272256.16 total 20.01 last 95.12) - open/close 2962 (time 1788272266.16 total 30.01 last 98.58) - open/close 4026 (time 1788272276.16 total 40.02 last 106.38) total: 4500 open/close in 44.50 seconds: 101.13 ops/second - /unlink 1650 (time 1788272293.70 total 10.01 last 164.76) - /unlink 2630 (time 1788272303.71 total 20.02 last 97.97) - /unlink 3605 (time 1788272313.71 total 30.02 last 97.46) - /unlink 4203 (time 1788272323.71 total 40.02 last 59.79) total: 4500 /unlink in 44.19 seconds: 101.83 ops/second conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11712:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7735:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11718:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1788272341.log /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 12451: 96370 Terminated $LUSTRE/tests/createmany $* /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 94639 Terminated __test_135_file_thread "$service" kill changelog reader rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg138-server.1788272341.log.Dj0MtO" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg138-server.1788272341.log.P0jMdv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 94635 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg138-server unloading modules via unload_modules_local on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing unload_modules_local oleg138-server: modules unloaded.