== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:32:04 (1788748324) Stopping clients: oleg101-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg101-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg101-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg101-server oleg101-server: oleg101-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: kmemleak disabled oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: detected 4 online CPUs by sysfs oleg101-server: MODOPTS_LIBCFS= oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg101-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg101-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 911 (time 1788748530.93 total 10.02 last 90.95) - open/close 1769 (time 1788748540.94 total 20.02 last 85.76) - open/close 2543 (time 1788748550.94 total 30.03 last 77.35) - open/close 3120 (time 1788748560.95 total 40.04 last 57.64) - open/close 3840 (time 1788748570.95 total 50.04 last 71.99) total: 4500 open/close in 57.77 seconds: 77.89 ops/second - /unlink 1543 (time 1788748592.65 total 10.00 last 154.25) - /unlink 3101 (time 1788748602.65 total 20.01 last 155.68) total: 4500 /unlink in 29.86 seconds: 150.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 816 (time 1788748627.80 total 10.02 last 81.41) - open/close 1720 (time 1788748637.81 total 20.03 last 90.36) - open/close 2462 (time 1788748647.84 total 30.06 last 73.99) - open/close 3391 (time 1788748657.84 total 40.06 last 92.87) - open/close 4338 (time 1788748667.84 total 50.06 last 94.70) total: 4500 open/close in 51.83 seconds: 86.82 ops/second - /unlink 1565 (time 1788748683.91 total 10.01 last 156.42) - /unlink 2965 (time 1788748693.92 total 20.01 last 139.89) - /unlink 4340 (time 1788748703.92 total 30.02 last 137.45) total: 4500 /unlink in 31.25 seconds: 144.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 811 (time 1788748719.56 total 10.01 last 81.03) - open/close 1545 (time 1788748729.57 total 20.02 last 73.33) - open/close 2312 (time 1788748739.58 total 30.03 last 76.64) - open/close 3203 (time 1788748749.58 total 40.03 last 89.07) - open/close 3949 (time 1788748759.59 total 50.04 last 74.51) total: 4500 open/close in 57.51 seconds: 78.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1624 (time 1788748781.64 total 10.00 last 162.34) - /unlink 3052 (time 1788748791.64 total 20.00 last 142.80) total: 4500 /unlink in 27.83 seconds: 161.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 907 (time 1788748813.81 total 10.00 last 90.67) - open/close 1797 (time 1788748823.82 total 20.01 last 88.93) - open/close 2697 (time 1788748833.82 total 30.02 last 89.93) - open/close 3699 (time 1788748843.83 total 40.03 last 100.10) total: 4500 open/close in 49.24 seconds: 91.38 ops/second - /unlink 1554 (time 1788748867.02 total 10.00 last 155.35) - /unlink 3347 (time 1788748877.02 total 20.01 last 179.20) total: 4500 /unlink in 27.69 seconds: 162.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 985 (time 1788748899.01 total 10.01 last 98.45) - open/close 1912 (time 1788748909.01 total 20.01 last 92.67) - open/close 2810 (time 1788748919.01 total 30.01 last 89.77) - open/close 3698 (time 1788748929.02 total 40.01 last 88.79) total: 4500 open/close in 48.66 seconds: 92.47 ops/second - /unlink 1802 (time 1788748951.30 total 10.00 last 180.14) - /unlink 3592 (time 1788748961.30 total 20.01 last 178.88) total: 4500 /unlink in 27.47 seconds: 163.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 892 (time 1788748983.97 total 10.00 last 89.17) - open/close 1776 (time 1788748993.98 total 20.01 last 88.37) - open/close 2695 (time 1788749003.98 total 30.01 last 91.88) - open/close 3653 (time 1788749013.98 total 40.01 last 95.78) total: 4500 open/close in 48.84 seconds: 92.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1786 (time 1788749035.99 total 10.00 last 178.55) - /unlink 3633 (time 1788749045.99 total 20.01 last 184.57) total: 4500 /unlink in 24.85 seconds: 181.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 927 (time 1788749065.41 total 10.01 last 92.65) - open/close 1801 (time 1788749075.41 total 20.01 last 87.37) - open/close 2773 (time 1788749085.42 total 30.02 last 97.07) - open/close 3905 (time 1788749095.43 total 40.03 last 113.09) total: 4500 open/close in 46.40 seconds: 96.98 ops/second - /unlink 1833 (time 1788749114.84 total 10.00 last 183.27) - /unlink 3692 (time 1788749124.84 total 20.00 last 185.85) total: 4500 /unlink in 24.39 seconds: 184.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 884 (time 1788749142.90 total 10.01 last 88.34) - open/close 1932 (time 1788749152.91 total 20.01 last 104.73) - open/close 2849 (time 1788749162.91 total 30.02 last 91.66) - open/close 3729 (time 1788749172.92 total 40.03 last 87.93) total: 4500 open/close in 49.12 seconds: 91.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 2047 (time 1788749194.97 total 10.00 last 204.69) - /unlink 3712 (time 1788749204.98 total 20.01 last 166.39) total: 4500 /unlink in 24.64 seconds: 182.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1023 (time 1788749222.60 total 10.00 last 102.27) - open/close 1991 (time 1788749232.62 total 20.02 last 96.68) - open/close 3065 (time 1788749242.63 total 30.02 last 107.31) - open/close 4060 (time 1788749252.63 total 40.03 last 99.45) total: 4500 open/close in 44.90 seconds: 100.22 ops/second - /unlink 2017 (time 1788749270.56 total 10.00 last 201.62) - /unlink 3916 (time 1788749280.56 total 20.01 last 189.83) total: 4500 /unlink in 22.52 seconds: 199.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1071 (time 1788749295.64 total 10.01 last 106.98) - open/close 2065 (time 1788749305.64 total 20.02 last 99.35) - open/close 3089 (time 1788749315.64 total 30.02 last 102.40) - open/close 4184 (time 1788749325.66 total 40.03 last 109.34) total: 4500 open/close in 43.00 seconds: 104.66 ops/second - /unlink 1871 (time 1788749341.53 total 10.00 last 187.08) - /unlink 3673 (time 1788749351.53 total 20.01 last 180.12) total: 4500 /unlink in 24.23 seconds: 185.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1073 (time 1788749369.87 total 10.01 last 107.17) - open/close 2230 (time 1788749379.87 total 20.01 last 115.69) - open/close 3449 (time 1788749389.87 total 30.02 last 121.84) total: 4500 open/close in 39.06 seconds: 115.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2218 (time 1788749411.57 total 10.01 last 221.59) - /unlink 4497 (time 1788749421.57 total 20.01 last 227.85) total: 4500 /unlink in 20.02 seconds: 224.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1097 (time 1788749434.32 total 10.00 last 109.66) - open/close 2197 (time 1788749444.32 total 20.01 last 109.92) - open/close 3339 (time 1788749454.33 total 30.01 last 114.17) - open/close 4383 (time 1788749464.33 total 40.02 last 104.37) total: 4500 open/close in 41.05 seconds: 109.63 ops/second - /unlink 2167 (time 1788749478.04 total 10.00 last 216.60) - /unlink 4415 (time 1788749488.04 total 20.01 last 224.68) total: 4500 /unlink in 20.45 seconds: 220.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1197 (time 1788749500.96 total 10.01 last 119.58) - open/close 2376 (time 1788749510.97 total 20.01 last 117.86) - open/close 3613 (time 1788749520.97 total 30.02 last 123.68) total: 4500 open/close in 37.20 seconds: 120.95 ops/second - /unlink 1593 (time 1788749542.22 total 10.00 last 159.23) - /unlink 2707 (time 1788749552.22 total 20.01 last 111.38) - /unlink 3794 (time 1788749562.23 total 30.02 last 108.59) - /unlink 4436 (time 1788749572.24 total 40.02 last 64.18) total: 4500 /unlink in 41.17 seconds: 109.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7553:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11760:test_135() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7930:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7993:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7792:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:11766:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_135.*.1788749589.log /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1: 95230 Terminated __test_135_file_thread "$service" kill changelog reader rsync: chown "/tmp/testlogs/.conf-sanity.test_135.debug_log.oleg101-server.1788749589.log.ubbEC4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_135.dmesg.oleg101-server.1788749589.log.u5nIEy" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11377: 95226 Terminated coproc COPROC $LFS changelog --follow $service lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg101-server unloading modules via unload_modules_local on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local oleg101-server: kmemleak disabled modules unloaded.