== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:24:24 (1767324264) Stopping clients: oleg217-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg217-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg217-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Stopping /mnt/lustre-ost2 (opts:-f) on oleg217-server oleg217-server: oleg217-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format ost1: /dev/vde Format ost2: /dev/vdf Format mds1: /dev/vdc Format mds2: /dev/vdd start mds service on oleg217-server pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 start mds service on oleg217-server pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg217-server pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 979 (time 1767324429.61 total 10.00 last 97.85) - open/close 1998 (time 1767324439.61 total 20.01 last 101.86) - open/close 3088 (time 1767324449.62 total 30.02 last 108.88) - open/close 4134 (time 1767324459.63 total 40.02 last 104.54) total: 4500 open/close in 43.40 seconds: 103.68 ops/second - /unlink 1827 (time 1767324475.34 total 10.00 last 182.65) - /unlink 3828 (time 1767324485.34 total 20.01 last 200.04) total: 4500 /unlink in 23.33 seconds: 192.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 1186 (time 1767324500.93 total 10.00 last 118.59) - open/close 2336 (time 1767324510.93 total 20.01 last 114.94) - open/close 3382 (time 1767324520.95 total 30.02 last 104.43) - open/close 4430 (time 1767324530.96 total 40.03 last 104.71) total: 4500 open/close in 40.65 seconds: 110.70 ops/second - /unlink 1765 (time 1767324544.51 total 10.00 last 176.44) - /unlink 3814 (time 1767324554.51 total 20.01 last 204.82) total: 4500 /unlink in 23.40 seconds: 192.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 1063 (time 1767324570.55 total 10.00 last 106.25) - open/close 2085 (time 1767324580.56 total 20.01 last 102.18) - open/close 3175 (time 1767324590.56 total 30.01 last 108.99) - open/close 4355 (time 1767324600.57 total 40.02 last 117.89) total: 4500 open/close in 41.11 seconds: 109.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 2366 (time 1767324613.90 total 10.00 last 236.60) total: 4500 /unlink in 17.93 seconds: 251.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 1345 (time 1767324633.92 total 10.01 last 134.38) - open/close 2536 (time 1767324643.92 total 20.01 last 119.05) - open/close 3590 (time 1767324653.93 total 30.02 last 105.32) total: 4500 open/close in 39.96 seconds: 112.61 ops/second - /unlink 1759 (time 1767324676.49 total 10.00 last 175.84) - /unlink 3471 (time 1767324686.49 total 20.00 last 171.18) total: 4500 /unlink in 25.68 seconds: 175.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 948 (time 1767324704.88 total 10.01 last 94.69) - open/close 1875 (time 1767324714.88 total 20.02 last 92.67) - open/close 3000 (time 1767324724.89 total 30.02 last 112.43) - open/close 4154 (time 1767324734.89 total 40.02 last 115.39) total: 4500 open/close in 43.43 seconds: 103.62 ops/second - /unlink 2063 (time 1767324750.63 total 10.00 last 206.28) - /unlink 4059 (time 1767324760.64 total 20.00 last 199.54) total: 4500 /unlink in 22.33 seconds: 201.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 1182 (time 1767324775.46 total 10.01 last 118.12) - open/close 2560 (time 1767324785.47 total 20.02 last 137.68) - open/close 4044 (time 1767324795.47 total 30.02 last 148.30) total: 4500 open/close in 32.99 seconds: 136.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 2976 (time 1767324810.31 total 10.00 last 297.58) total: 4500 /unlink in 16.16 seconds: 278.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 1137 (time 1767324828.97 total 10.00 last 113.67) - open/close 2211 (time 1767324838.98 total 20.00 last 107.39) - open/close 3288 (time 1767324848.98 total 30.01 last 107.63) - open/close 4316 (time 1767324858.99 total 40.01 last 102.75) total: 4500 open/close in 41.72 seconds: 107.87 ops/second - /unlink 2069 (time 1767324873.39 total 10.00 last 206.89) - /unlink 4050 (time 1767324883.39 total 20.00 last 198.08) total: 4500 /unlink in 22.27 seconds: 202.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 956 (time 1767324898.27 total 10.01 last 95.53) - open/close 2009 (time 1767324908.28 total 20.01 last 105.23) - open/close 2959 (time 1767324918.28 total 30.02 last 94.92) - open/close 3998 (time 1767324928.29 total 40.02 last 103.87) total: 4500 open/close in 44.76 seconds: 100.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 2220 (time 1767324945.93 total 10.01 last 221.88) - /unlink 4369 (time 1767324955.93 total 20.01 last 214.81) total: 4500 /unlink in 20.73 seconds: 217.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 1382 (time 1767324969.12 total 10.00 last 138.19) - open/close 2592 (time 1767324979.12 total 20.01 last 120.94) - open/close 3929 (time 1767324989.13 total 30.01 last 133.62) total: 4500 open/close in 34.07 seconds: 132.08 ops/second - /unlink 2498 (time 1767325005.54 total 10.00 last 249.69) total: 4500 /unlink in 18.64 seconds: 241.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 1406 (time 1767325026.52 total 10.00 last 140.57) - open/close 2831 (time 1767325036.53 total 20.01 last 142.35) - open/close 4394 (time 1767325046.54 total 30.02 last 156.13) total: 4500 open/close in 30.58 seconds: 147.16 ops/second - /unlink 2985 (time 1767325058.99 total 10.00 last 298.35) total: 4500 /unlink in 16.65 seconds: 270.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 1697 (time 1767325077.94 total 10.00 last 169.66) - open/close 3137 (time 1767325087.95 total 20.01 last 143.88) total: 4500 open/close in 28.48 seconds: 158.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 3074 (time 1767325108.30 total 10.01 last 307.19) total: 4500 /unlink in 16.57 seconds: 271.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1409 (time 1767325127.00 total 10.01 last 140.82) - open/close 2962 (time 1767325137.00 total 20.01 last 155.28) - open/close 4200 (time 1767325147.01 total 30.01 last 123.74) total: 4500 open/close in 31.99 seconds: 140.68 ops/second - /unlink 2533 (time 1767325161.01 total 10.01 last 253.03) total: 4500 /unlink in 17.50 seconds: 257.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1477 (time 1767325180.52 total 10.00 last 147.63) - open/close 2868 (time 1767325190.52 total 20.01 last 139.08) - open/close 4150 (time 1767325200.52 total 30.01 last 128.11) total: 4500 open/close in 32.43 seconds: 138.75 ops/second - /unlink 2706 (time 1767325214.95 total 10.00 last 270.50) total: 4500 /unlink in 17.92 seconds: 251.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1180 (time 1767325234.98 total 10.00 last 117.98) - open/close 2388 (time 1767325244.99 total 20.01 last 120.71) - open/close 3573 (time 1767325255.00 total 30.02 last 118.36) total: 4500 open/close in 38.62 seconds: 116.53 ops/second - /unlink 2843 (time 1767325275.70 total 10.00 last 284.27) total: 4500 /unlink in 16.77 seconds: 268.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 1251 (time 1767325294.38 total 10.01 last 125.00) - open/close 2583 (time 1767325304.38 total 20.01 last 133.14) - open/close 3859 (time 1767325314.38 total 30.01 last 127.57) total: 4500 open/close in 34.99 seconds: 128.61 ops/second - /unlink 2591 (time 1767325331.25 total 10.00 last 259.04) total: 4500 /unlink in 19.86 seconds: 226.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1176 (time 1767325353.66 total 10.00 last 117.60) - open/close 2301 (time 1767325363.67 total 20.01 last 112.43) - open/close 3521 (time 1767325373.67 total 30.01 last 121.94) total: 4500 open/close in 37.28 seconds: 120.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2562 (time 1767325393.09 total 10.01 last 256.07) total: 4500 /unlink in 18.15 seconds: 247.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1448 (time 1767325413.25 total 10.00 last 144.77) - open/close 2729 (time 1767325423.26 total 20.01 last 128.06) - open/close 3965 (time 1767325433.26 total 30.01 last 123.58) total: 4500 open/close in 34.57 seconds: 130.17 ops/second - /unlink 2632 (time 1767325450.11 total 10.00 last 263.13) total: 4500 /unlink in 18.40 seconds: 244.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1277 (time 1767325470.76 total 10.00 last 127.69) - open/close 2544 (time 1767325480.77 total 20.01 last 126.59) total: 4500 open/close in 29.77 seconds: 151.13 ops/second - /unlink 2591 (time 1767325502.55 total 10.00 last 259.04) total: 4500 /unlink in 17.84 seconds: 252.22 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11271: 111461 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg217-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local oleg217-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.