== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:24:19 (1763083459) Stopping clients: oleg345-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg345-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg345-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg345-server oleg345-server: oleg345-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg345-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg345-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 904 (time 1763083606.68 total 10.01 last 90.35) - open/close 1919 (time 1763083616.68 total 20.01 last 101.46) - open/close 3118 (time 1763083626.68 total 30.01 last 119.84) - open/close 4141 (time 1763083636.70 total 40.03 last 102.16) total: 4500 open/close in 43.47 seconds: 103.51 ops/second - /unlink 1874 (time 1763083652.66 total 10.00 last 187.40) - /unlink 3792 (time 1763083662.66 total 20.00 last 191.74) total: 4500 /unlink in 23.94 seconds: 188.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 841 (time 1763083679.56 total 10.01 last 84.02) - open/close 1749 (time 1763083689.57 total 20.02 last 90.73) - open/close 2687 (time 1763083699.58 total 30.02 last 93.75) - open/close 3690 (time 1763083709.58 total 40.02 last 100.30) total: 4500 open/close in 48.61 seconds: 92.57 ops/second - /unlink 1884 (time 1763083730.60 total 10.00 last 188.38) - /unlink 3884 (time 1763083740.60 total 20.00 last 199.99) total: 4500 /unlink in 23.10 seconds: 194.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 971 (time 1763083756.18 total 10.01 last 97.05) - open/close 1941 (time 1763083766.19 total 20.01 last 96.94) - open/close 2984 (time 1763083776.19 total 30.01 last 104.29) - open/close 4253 (time 1763083786.19 total 40.01 last 126.89) total: 4500 open/close in 42.66 seconds: 105.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1803 (time 1763083801.47 total 10.00 last 180.25) - /unlink 3865 (time 1763083811.47 total 20.00 last 206.16) total: 4500 /unlink in 22.79 seconds: 197.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 1320 (time 1763083826.36 total 10.01 last 131.91) - open/close 2513 (time 1763083836.37 total 20.01 last 119.21) - open/close 3581 (time 1763083846.37 total 30.02 last 106.76) total: 4500 open/close in 38.74 seconds: 116.16 ops/second - /unlink 2619 (time 1763083867.04 total 10.00 last 261.90) total: 4500 /unlink in 18.15 seconds: 247.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 1413 (time 1763083887.49 total 10.00 last 141.25) - open/close 2784 (time 1763083897.50 total 20.01 last 136.98) - open/close 4086 (time 1763083907.50 total 30.02 last 130.14) total: 4500 open/close in 33.47 seconds: 134.43 ops/second - /unlink 2313 (time 1763083922.99 total 10.00 last 231.29) total: 4500 /unlink in 19.90 seconds: 226.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 1087 (time 1763083945.29 total 10.02 last 108.53) - open/close 2013 (time 1763083955.31 total 20.03 last 92.49) - open/close 2981 (time 1763083965.31 total 30.04 last 96.72) - open/close 3946 (time 1763083975.32 total 40.05 last 96.40) total: 4500 open/close in 45.73 seconds: 98.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1629 (time 1763083993.53 total 10.01 last 162.79) - /unlink 3354 (time 1763084003.53 total 20.01 last 172.48) total: 4500 /unlink in 26.62 seconds: 169.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 1024 (time 1763084022.75 total 10.00 last 102.40) - open/close 2277 (time 1763084032.75 total 20.01 last 125.20) - open/close 3505 (time 1763084042.76 total 30.01 last 122.74) total: 4500 open/close in 38.43 seconds: 117.09 ops/second - /unlink 2110 (time 1763084063.43 total 10.00 last 210.96) - /unlink 4249 (time 1763084073.44 total 20.01 last 213.82) total: 4500 /unlink in 21.28 seconds: 211.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 1224 (time 1763084086.85 total 10.00 last 122.34) - open/close 2336 (time 1763084096.86 total 20.01 last 111.14) - open/close 3576 (time 1763084106.86 total 30.01 last 123.99) total: 4500 open/close in 37.83 seconds: 118.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 3049 (time 1763084126.50 total 10.00 last 304.77) total: 4500 /unlink in 15.04 seconds: 299.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1795 (time 1763084143.33 total 10.00 last 179.49) - open/close 3743 (time 1763084153.34 total 20.01 last 194.70) total: 4500 open/close in 24.93 seconds: 180.53 ops/second - /unlink 2799 (time 1763084170.11 total 10.00 last 279.86) total: 4500 /unlink in 16.24 seconds: 277.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1424 (time 1763084188.30 total 10.00 last 142.36) - open/close 2596 (time 1763084198.30 total 20.00 last 117.18) - open/close 4002 (time 1763084208.30 total 30.01 last 140.59) total: 4500 open/close in 33.43 seconds: 134.60 ops/second - /unlink 2888 (time 1763084223.58 total 10.00 last 288.79) total: 4500 /unlink in 15.61 seconds: 288.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1290 (time 1763084241.12 total 10.00 last 128.95) - open/close 2758 (time 1763084251.12 total 20.01 last 146.73) - open/close 4473 (time 1763084261.12 total 30.01 last 171.46) total: 4500 open/close in 30.18 seconds: 149.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 2551 (time 1763084273.30 total 10.00 last 255.08) total: 4500 /unlink in 17.38 seconds: 258.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1239 (time 1763084292.61 total 10.01 last 123.83) - open/close 2387 (time 1763084302.62 total 20.01 last 114.73) - open/close 3597 (time 1763084312.63 total 30.02 last 120.87) total: 4500 open/close in 37.97 seconds: 118.53 ops/second - /unlink 2433 (time 1763084332.74 total 10.00 last 243.24) total: 4500 /unlink in 17.85 seconds: 252.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 2033 (time 1763084352.49 total 10.00 last 203.23) - open/close 4178 (time 1763084362.49 total 20.01 last 214.45) total: 4500 open/close in 22.04 seconds: 204.22 ops/second - /unlink 3835 (time 1763084376.43 total 10.00 last 383.47) total: 4500 /unlink in 11.92 seconds: 377.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 2380 (time 1763084390.18 total 10.00 last 237.99) total: 4500 open/close in 18.80 seconds: 239.35 ops/second total: 4500 /unlink in 8.28 seconds: 543.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 2453 (time 1763084420.20 total 10.00 last 245.20) - open/close 4465 (time 1763084430.20 total 20.01 last 201.14) total: 4500 open/close in 20.15 seconds: 223.31 ops/second total: 4500 /unlink in 9.48 seconds: 474.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 2304 (time 1763084452.98 total 10.00 last 230.36) total: 4500 open/close in 18.02 seconds: 249.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 4259 (time 1763084472.63 total 10.00 last 425.88) total: 4500 /unlink in 10.50 seconds: 428.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1723 (time 1763084484.79 total 10.00 last 172.22) - open/close 3894 (time 1763084494.79 total 20.01 last 217.09) total: 4500 open/close in 22.64 seconds: 198.80 ops/second - /unlink 4257 (time 1763084509.05 total 10.00 last 425.60) total: 4500 /unlink in 10.55 seconds: 426.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 2578 (time 1763084521.32 total 10.00 last 257.71) total: 4500 open/close in 18.42 seconds: 244.33 ops/second - /unlink 2569 (time 1763084541.62 total 10.00 last 256.87) total: 4500 /unlink in 17.01 seconds: 264.49 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93892 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg345-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.