== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:58:44 (1766415524) Stopping clients: oleg413-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg413-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg413-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg413-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg413-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg413-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg413-server oleg413-server: oleg413-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing load_modules_local oleg413-server: Loading modules from /home/green/git/lustre-release/lustre oleg413-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg413-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg413-server: detected 4 online CPUs by sysfs oleg413-server: MODOPTS_LIBCFS= oleg413-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg413-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg413-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg413-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 760 (time 1766415686.60 total 10.00 last 75.97) - open/close 1513 (time 1766415696.61 total 20.01 last 75.25) - open/close 2309 (time 1766415706.62 total 30.02 last 79.51) - open/close 3083 (time 1766415716.62 total 40.03 last 77.36) - open/close 3898 (time 1766415726.63 total 50.03 last 81.47) total: 4500 open/close in 57.06 seconds: 78.87 ops/second - /unlink 1378 (time 1766415746.55 total 10.00 last 137.76) - /unlink 2766 (time 1766415756.55 total 20.01 last 138.75) - /unlink 4046 (time 1766415766.56 total 30.01 last 127.94) total: 4500 /unlink in 33.35 seconds: 134.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 703 (time 1766415783.63 total 10.00 last 70.30) - open/close 1392 (time 1766415793.64 total 20.01 last 68.86) - open/close 2114 (time 1766415803.65 total 30.02 last 72.11) - open/close 2747 (time 1766415813.66 total 40.03 last 63.24) - open/close 3463 (time 1766415823.67 total 50.04 last 71.53) - open/close 4163 (time 1766415833.67 total 60.04 last 69.98) total: 4500 open/close in 64.74 seconds: 69.51 ops/second - /unlink 1335 (time 1766415852.17 total 10.00 last 133.49) - /unlink 2713 (time 1766415862.17 total 20.00 last 137.78) - /unlink 4071 (time 1766415872.17 total 30.01 last 135.74) total: 4500 /unlink in 32.83 seconds: 137.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 707 (time 1766415888.55 total 10.02 last 70.57) - open/close 1387 (time 1766415898.56 total 20.03 last 67.95) - open/close 2164 (time 1766415908.56 total 30.03 last 77.67) - open/close 2947 (time 1766415918.56 total 40.03 last 78.27) - open/close 3668 (time 1766415928.57 total 50.04 last 72.08) - open/close 4383 (time 1766415938.57 total 60.04 last 71.46) total: 4500 open/close in 61.71 seconds: 72.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1266 (time 1766415954.02 total 10.01 last 126.53) - /unlink 2531 (time 1766415964.03 total 20.01 last 126.41) - /unlink 3931 (time 1766415974.03 total 30.01 last 139.97) total: 4500 /unlink in 34.14 seconds: 131.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 746 (time 1766415992.10 total 10.01 last 74.56) - open/close 1493 (time 1766416002.12 total 20.02 last 74.62) - open/close 2254 (time 1766416012.12 total 30.02 last 76.08) - open/close 3074 (time 1766416022.13 total 40.03 last 81.93) - open/close 3852 (time 1766416032.14 total 50.04 last 77.73) total: 4500 open/close in 58.00 seconds: 77.59 ops/second - /unlink 1420 (time 1766416053.01 total 10.00 last 141.96) - /unlink 2861 (time 1766416063.02 total 20.01 last 143.97) - /unlink 4363 (time 1766416073.03 total 30.02 last 150.09) total: 4500 /unlink in 30.89 seconds: 145.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 770 (time 1766416086.99 total 10.01 last 76.95) - open/close 1533 (time 1766416097.00 total 20.01 last 76.25) - open/close 2336 (time 1766416107.00 total 30.02 last 80.28) - open/close 3180 (time 1766416117.00 total 40.02 last 84.39) - open/close 4010 (time 1766416127.01 total 50.02 last 82.94) total: 4500 open/close in 54.83 seconds: 82.08 ops/second - /unlink 1555 (time 1766416144.54 total 10.00 last 155.46) - /unlink 3297 (time 1766416154.54 total 20.00 last 174.17) total: 4500 /unlink in 27.10 seconds: 166.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 754 (time 1766416174.96 total 10.01 last 75.36) - open/close 1564 (time 1766416184.98 total 20.02 last 80.85) - open/close 2318 (time 1766416194.98 total 30.03 last 75.37) - open/close 3085 (time 1766416204.99 total 40.04 last 76.61) - open/close 3898 (time 1766416215.00 total 50.05 last 81.22) total: 4500 open/close in 57.40 seconds: 78.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1640 (time 1766416235.57 total 10.01 last 163.86) - /unlink 2984 (time 1766416245.58 total 20.01 last 134.33) - /unlink 4390 (time 1766416255.59 total 30.03 last 140.38) total: 4500 /unlink in 30.97 seconds: 145.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 757 (time 1766416270.77 total 10.01 last 75.61) - open/close 1509 (time 1766416280.78 total 20.02 last 75.13) - open/close 2274 (time 1766416290.79 total 30.03 last 76.47) - open/close 3082 (time 1766416300.79 total 40.03 last 80.75) - open/close 3932 (time 1766416310.80 total 50.04 last 84.96) total: 4500 open/close in 57.42 seconds: 78.37 ops/second - /unlink 1451 (time 1766416331.15 total 10.00 last 145.05) - /unlink 2968 (time 1766416341.15 total 20.01 last 151.67) - /unlink 4367 (time 1766416351.15 total 30.01 last 139.85) total: 4500 /unlink in 31.10 seconds: 144.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 728 (time 1766416365.82 total 10.01 last 72.74) - open/close 1450 (time 1766416375.83 total 20.02 last 72.13) - open/close 2194 (time 1766416385.83 total 30.02 last 74.40) - open/close 2976 (time 1766416395.84 total 40.02 last 78.14) - open/close 3798 (time 1766416405.84 total 50.03 last 82.17) total: 4500 open/close in 59.34 seconds: 75.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1490 (time 1766416428.31 total 10.00 last 148.96) - /unlink 3017 (time 1766416438.31 total 20.01 last 152.67) - /unlink 4296 (time 1766416448.31 total 30.01 last 127.88) total: 4500 /unlink in 31.54 seconds: 142.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 853 (time 1766416463.20 total 10.00 last 85.29) - open/close 1765 (time 1766416473.21 total 20.01 last 91.11) - open/close 2562 (time 1766416483.22 total 30.02 last 79.64) - open/close 3419 (time 1766416493.23 total 40.03 last 85.62) - open/close 4254 (time 1766416503.24 total 50.04 last 83.41) total: 4500 open/close in 53.21 seconds: 84.57 ops/second - /unlink 1451 (time 1766416519.30 total 10.01 last 144.95) - /unlink 2878 (time 1766416529.30 total 20.01 last 142.66) - /unlink 4280 (time 1766416539.30 total 30.02 last 140.13) total: 4500 /unlink in 31.55 seconds: 142.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 794 (time 1766416553.98 total 10.00 last 79.36) - open/close 1636 (time 1766416563.99 total 20.01 last 84.13) - open/close 2450 (time 1766416573.99 total 30.02 last 81.37) - open/close 3289 (time 1766416584.00 total 40.02 last 83.85) - open/close 4056 (time 1766416594.00 total 50.03 last 76.67) total: 4500 open/close in 55.38 seconds: 81.26 ops/second - /unlink 1393 (time 1766416612.56 total 10.00 last 139.23) - /unlink 2907 (time 1766416622.56 total 20.01 last 151.35) - /unlink 4430 (time 1766416632.57 total 30.02 last 152.18) total: 4500 /unlink in 30.57 seconds: 147.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 829 (time 1766416646.02 total 10.01 last 82.85) - open/close 1650 (time 1766416656.03 total 20.02 last 82.02) - open/close 2449 (time 1766416666.04 total 30.02 last 79.84) - open/close 3336 (time 1766416676.05 total 40.04 last 88.58) - open/close 4213 (time 1766416686.06 total 50.05 last 87.60) total: 4500 open/close in 53.68 seconds: 83.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1490 (time 1766416702.82 total 10.00 last 149.00) - /unlink 2891 (time 1766416712.82 total 20.01 last 140.01) - /unlink 4274 (time 1766416722.84 total 30.02 last 138.15) total: 4500 /unlink in 31.67 seconds: 142.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 797 (time 1766416737.69 total 10.02 last 79.54) - open/close 1615 (time 1766416747.69 total 20.02 last 81.77) - open/close 2405 (time 1766416757.70 total 30.03 last 78.98) - open/close 3192 (time 1766416767.70 total 40.03 last 78.67) - open/close 3995 (time 1766416777.70 total 50.03 last 80.28) total: 4500 open/close in 57.14 seconds: 78.75 ops/second - /unlink 1434 (time 1766416798.37 total 10.01 last 143.32) - /unlink 2760 (time 1766416808.37 total 20.01 last 132.52) - /unlink 4123 (time 1766416818.38 total 30.02 last 136.22) total: 4500 /unlink in 32.69 seconds: 137.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 848 (time 1766416834.79 total 10.01 last 84.71) - open/close 1603 (time 1766416844.80 total 20.03 last 75.37) - open/close 2390 (time 1766416854.81 total 30.03 last 78.68) - open/close 3192 (time 1766416864.81 total 40.04 last 80.16) - open/close 4045 (time 1766416874.81 total 50.04 last 85.28) total: 4500 open/close in 55.55 seconds: 81.01 ops/second - /unlink 1498 (time 1766416893.27 total 10.00 last 149.79) - /unlink 3002 (time 1766416903.27 total 20.00 last 150.35) total: 4500 /unlink in 28.53 seconds: 157.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 905 (time 1766416924.68 total 10.02 last 90.34) - open/close 1927 (time 1766416934.68 total 20.03 last 102.12) - open/close 3012 (time 1766416944.68 total 30.03 last 108.49) - open/close 4050 (time 1766416954.69 total 40.03 last 103.77) total: 4500 open/close in 43.40 seconds: 103.70 ops/second - /unlink 2090 (time 1766416970.35 total 10.00 last 208.95) - /unlink 4054 (time 1766416980.35 total 20.01 last 196.32) total: 4500 /unlink in 22.95 seconds: 196.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 968 (time 1766416995.86 total 10.00 last 96.78) - open/close 2005 (time 1766417005.87 total 20.01 last 103.61) - open/close 3120 (time 1766417015.88 total 30.01 last 111.46) - open/close 4180 (time 1766417025.88 total 40.02 last 105.96) total: 4500 open/close in 42.87 seconds: 104.97 ops/second - /unlink 1837 (time 1766417041.00 total 10.00 last 183.66) - /unlink 3875 (time 1766417051.00 total 20.01 last 203.73) total: 4500 /unlink in 22.63 seconds: 198.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1044 (time 1766417065.79 total 10.01 last 104.33) - open/close 2180 (time 1766417075.79 total 20.01 last 113.54) - open/close 3342 (time 1766417085.79 total 30.01 last 116.19) total: 4500 open/close in 38.48 seconds: 116.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2309 (time 1766417106.09 total 10.01 last 230.77) - /unlink 4033 (time 1766417116.10 total 20.01 last 172.34) total: 4500 /unlink in 22.27 seconds: 202.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1095 (time 1766417130.55 total 10.01 last 109.42) - open/close 2271 (time 1766417140.56 total 20.02 last 117.48) - open/close 3400 (time 1766417150.57 total 30.02 last 112.87) total: 4500 open/close in 38.69 seconds: 116.30 ops/second - /unlink 2354 (time 1766417171.18 total 10.00 last 235.33) - /unlink 4378 (time 1766417181.19 total 20.01 last 202.26) total: 4500 /unlink in 20.70 seconds: 217.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1376 (time 1766417193.99 total 10.00 last 137.60) - open/close 2772 (time 1766417203.99 total 20.01 last 139.50) - open/close 3937 (time 1766417213.99 total 30.01 last 116.49) total: 4500 open/close in 34.07 seconds: 132.09 ops/second - /unlink 2713 (time 1766417229.88 total 10.00 last 271.28) total: 4500 /unlink in 17.47 seconds: 257.64 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11101: 94043 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg413-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg413-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg413-server stop mds service on oleg413-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg413-server stop mds service on oleg413-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg413-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing unload_modules_local oleg413-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg413-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.