== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:13:15 (1767319995) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) Stopping client oleg241-client.virtnet /mnt/lustre opts:-f Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server waited 0 for 11 ST obdfilter lustre-OST0001 lustre-OST0001_UUID 5 waited 2 for 11 ST obdfilter lustre-OST0001 lustre-OST0001_UUID 5 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 oleg241-server: oleg241-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg241-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/vdc Format ost1: /dev/vde Format mds2: /dev/vdd Format ost2: /dev/vdf start mds service on oleg241-server pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 start mds service on oleg241-server pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg241-server pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 813 (time 1767320236.90 total 10.01 last 81.20) - open/close 1492 (time 1767320246.90 total 20.01 last 67.89) - open/close 2306 (time 1767320256.90 total 30.01 last 81.39) - open/close 3063 (time 1767320266.91 total 40.02 last 75.64) - open/close 3901 (time 1767320276.93 total 50.04 last 83.65) total: 4500 open/close in 57.22 seconds: 78.65 ops/second - /unlink 1486 (time 1767320297.92 total 10.00 last 148.59) - /unlink 3061 (time 1767320307.92 total 20.00 last 157.45) total: 4500 /unlink in 29.13 seconds: 154.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 793 (time 1767320331.92 total 10.01 last 79.24) - open/close 1529 (time 1767320341.93 total 20.02 last 73.54) - open/close 2325 (time 1767320351.93 total 30.02 last 79.57) - open/close 3076 (time 1767320361.94 total 40.03 last 75.06) - open/close 3837 (time 1767320371.94 total 50.03 last 76.08) total: 4500 open/close in 57.82 seconds: 77.83 ops/second - /unlink 1326 (time 1767320393.96 total 10.00 last 132.58) - /unlink 2714 (time 1767320403.96 total 20.00 last 138.78) - /unlink 4017 (time 1767320413.96 total 30.01 last 130.24) total: 4500 /unlink in 33.49 seconds: 134.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 799 (time 1767320431.95 total 10.01 last 79.83) - open/close 1573 (time 1767320441.95 total 20.01 last 77.38) - open/close 2458 (time 1767320451.95 total 30.02 last 88.46) - open/close 3216 (time 1767320461.96 total 40.02 last 75.76) - open/close 3997 (time 1767320471.97 total 50.03 last 78.05) total: 4500 open/close in 56.56 seconds: 79.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1245 (time 1767320492.59 total 10.00 last 124.46) - /unlink 2765 (time 1767320502.59 total 20.01 last 151.91) - /unlink 4305 (time 1767320512.59 total 30.01 last 153.97) total: 4500 /unlink in 31.53 seconds: 142.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 632 (time 1767320528.99 total 10.02 last 63.06) - open/close 1260 (time 1767320538.99 total 20.03 last 62.78) - open/close 1968 (time 1767320549.00 total 30.04 last 70.72) - open/close 2583 (time 1767320559.03 total 40.06 last 61.36) - open/close 3181 (time 1767320569.03 total 50.06 last 59.78) - open/close 3887 (time 1767320579.04 total 60.07 last 70.55) total: 4500 open/close in 68.13 seconds: 66.05 ops/second - /unlink 1188 (time 1767320601.64 total 10.00 last 118.79) - /unlink 2612 (time 1767320611.65 total 20.00 last 142.37) - /unlink 3829 (time 1767320621.65 total 30.00 last 121.69) total: 4500 /unlink in 35.71 seconds: 126.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 713 (time 1767320641.86 total 10.02 last 71.18) - open/close 1384 (time 1767320651.86 total 20.02 last 67.04) - open/close 2151 (time 1767320661.89 total 30.05 last 76.51) - open/close 2909 (time 1767320671.90 total 40.06 last 75.72) - open/close 3609 (time 1767320681.90 total 50.06 last 69.98) - open/close 4313 (time 1767320691.90 total 60.06 last 70.39) total: 4500 open/close in 62.48 seconds: 72.03 ops/second - /unlink 1246 (time 1767320709.23 total 10.00 last 124.59) - /unlink 2642 (time 1767320719.24 total 20.01 last 139.53) - /unlink 4075 (time 1767320729.24 total 30.01 last 143.27) total: 4500 /unlink in 33.37 seconds: 134.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 713 (time 1767320747.65 total 10.02 last 71.13) - open/close 1449 (time 1767320757.66 total 20.03 last 73.53) - open/close 2221 (time 1767320767.66 total 30.04 last 77.19) - open/close 3009 (time 1767320777.68 total 40.05 last 78.70) - open/close 3764 (time 1767320787.68 total 50.05 last 75.49) - open/close 4456 (time 1767320797.68 total 60.05 last 69.17) total: 4500 open/close in 60.56 seconds: 74.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1230 (time 1767320812.22 total 10.01 last 122.91) - /unlink 2488 (time 1767320822.22 total 20.01 last 125.78) - /unlink 3929 (time 1767320832.22 total 30.01 last 144.08) total: 4500 /unlink in 33.91 seconds: 132.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 768 (time 1767320851.00 total 10.00 last 76.78) - open/close 1419 (time 1767320861.02 total 20.03 last 64.95) - open/close 2084 (time 1767320871.04 total 30.04 last 66.42) - open/close 2759 (time 1767320881.04 total 40.04 last 67.46) - open/close 3530 (time 1767320891.05 total 50.05 last 77.05) - open/close 4243 (time 1767320901.05 total 60.06 last 71.25) total: 4500 open/close in 63.52 seconds: 70.84 ops/second - /unlink 1248 (time 1767320919.72 total 10.00 last 124.75) - /unlink 2492 (time 1767320929.72 total 20.01 last 124.37) - /unlink 4021 (time 1767320939.72 total 30.01 last 152.83) total: 4500 /unlink in 33.33 seconds: 135.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 821 (time 1767320957.45 total 10.01 last 82.02) - open/close 1574 (time 1767320967.46 total 20.02 last 75.22) - open/close 2254 (time 1767320977.46 total 30.02 last 67.97) - open/close 2861 (time 1767320987.47 total 40.03 last 60.65) - open/close 3558 (time 1767320997.47 total 50.04 last 69.68) - open/close 4368 (time 1767321007.49 total 60.05 last 80.86) total: 4500 open/close in 61.89 seconds: 72.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1273 (time 1767321023.89 total 10.00 last 127.27) - /unlink 2612 (time 1767321033.89 total 20.01 last 133.84) - /unlink 3942 (time 1767321043.90 total 30.01 last 132.92) total: 4500 /unlink in 34.15 seconds: 131.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 694 (time 1767321063.19 total 10.00 last 69.40) - open/close 1393 (time 1767321073.19 total 20.00 last 69.88) - open/close 2186 (time 1767321083.20 total 30.01 last 79.25) - open/close 2921 (time 1767321093.20 total 40.01 last 73.48) - open/close 3642 (time 1767321103.22 total 50.03 last 71.96) - open/close 4364 (time 1767321113.22 total 60.03 last 72.19) total: 4500 open/close in 61.69 seconds: 72.95 ops/second - /unlink 1268 (time 1767321129.14 total 10.00 last 126.75) - /unlink 2615 (time 1767321139.15 total 20.01 last 134.60) - /unlink 4052 (time 1767321149.16 total 30.01 last 143.65) total: 4500 /unlink in 33.12 seconds: 135.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 564 (time 1767321168.12 total 10.01 last 56.36) - open/close 1256 (time 1767321178.13 total 20.02 last 69.12) - open/close 2032 (time 1767321188.13 total 30.02 last 77.57) - open/close 2713 (time 1767321198.13 total 40.02 last 68.09) - open/close 3394 (time 1767321208.15 total 50.03 last 68.02) - open/close 4177 (time 1767321218.15 total 60.04 last 78.24) total: 4500 open/close in 64.72 seconds: 69.53 ops/second - /unlink 1408 (time 1767321237.62 total 10.00 last 140.74) - /unlink 2808 (time 1767321247.62 total 20.00 last 140.00) - /unlink 4346 (time 1767321257.62 total 30.01 last 153.75) total: 4500 /unlink in 31.12 seconds: 144.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 736 (time 1767321273.17 total 10.01 last 73.56) - open/close 1425 (time 1767321283.18 total 20.01 last 68.86) - open/close 2111 (time 1767321293.19 total 30.02 last 68.54) - open/close 2822 (time 1767321303.19 total 40.02 last 71.07) - open/close 3505 (time 1767321313.20 total 50.04 last 68.22) - open/close 4126 (time 1767321323.21 total 60.05 last 62.04) total: 4500 open/close in 65.69 seconds: 68.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1222 (time 1767321344.22 total 10.00 last 122.19) - /unlink 2485 (time 1767321354.22 total 20.00 last 126.30) - /unlink 3697 (time 1767321364.22 total 30.00 last 121.20) total: 4500 /unlink in 37.38 seconds: 120.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 784 (time 1767321387.76 total 10.01 last 78.33) - open/close 1517 (time 1767321397.76 total 20.01 last 73.26) - open/close 2085 (time 1767321407.79 total 30.04 last 56.64) - open/close 2681 (time 1767321417.79 total 40.05 last 59.58) - open/close 3415 (time 1767321427.80 total 50.05 last 73.35) - open/close 4215 (time 1767321437.80 total 60.05 last 80.00) total: 4500 open/close in 63.59 seconds: 70.77 ops/second - /unlink 1359 (time 1767321455.28 total 10.01 last 135.80) - /unlink 2714 (time 1767321465.29 total 20.01 last 135.43) - /unlink 4114 (time 1767321475.29 total 30.01 last 139.97) total: 4500 /unlink in 32.90 seconds: 136.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 861 (time 1767321492.55 total 10.00 last 86.06) - open/close 1660 (time 1767321502.55 total 20.01 last 79.89) - open/close 2423 (time 1767321512.57 total 30.02 last 76.18) - open/close 3308 (time 1767321522.57 total 40.02 last 88.48) - open/close 4177 (time 1767321532.59 total 50.04 last 86.72) total: 4500 open/close in 55.03 seconds: 81.78 ops/second - /unlink 1324 (time 1767321551.77 total 10.01 last 132.27) - /unlink 2730 (time 1767321561.77 total 20.01 last 140.58) - /unlink 4023 (time 1767321571.78 total 30.02 last 129.21) total: 4500 /unlink in 33.55 seconds: 134.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 761 (time 1767321589.54 total 10.00 last 76.09) - open/close 1554 (time 1767321599.55 total 20.01 last 79.24) - open/close 2353 (time 1767321609.56 total 30.02 last 79.82) - open/close 3106 (time 1767321619.56 total 40.02 last 75.26) - open/close 3846 (time 1767321629.57 total 50.03 last 73.93) total: 4500 open/close in 58.55 seconds: 76.86 ops/second - /unlink 1549 (time 1767321651.91 total 10.00 last 154.88) - /unlink 2997 (time 1767321661.91 total 20.00 last 144.75) - /unlink 4061 (time 1767321671.91 total 30.01 last 106.37) total: 4500 /unlink in 34.67 seconds: 129.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 717 (time 1767321692.90 total 10.00 last 71.68) - open/close 1460 (time 1767321702.91 total 20.01 last 74.22) - open/close 2211 (time 1767321712.92 total 30.02 last 75.06) - open/close 2911 (time 1767321722.93 total 40.04 last 69.88) - open/close 3693 (time 1767321732.94 total 50.04 last 78.15) - open/close 4454 (time 1767321742.95 total 60.05 last 76.05) total: 4500 open/close in 60.61 seconds: 74.25 ops/second - /unlink 1417 (time 1767321757.47 total 10.00 last 141.66) - /unlink 2788 (time 1767321767.47 total 20.00 last 137.08) - /unlink 4142 (time 1767321777.48 total 30.01 last 135.37) total: 4500 /unlink in 32.53 seconds: 138.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 782 (time 1767321794.51 total 10.00 last 78.18) - open/close 1519 (time 1767321804.52 total 20.01 last 73.67) - open/close 2251 (time 1767321814.52 total 30.01 last 73.14) - open/close 3071 (time 1767321824.53 total 40.02 last 81.94) - open/close 3803 (time 1767321834.54 total 50.03 last 73.14) total: 4500 open/close in 59.47 seconds: 75.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1406 (time 1767321858.48 total 10.00 last 140.56) - /unlink 2777 (time 1767321868.48 total 20.00 last 137.08) - /unlink 4243 (time 1767321878.48 total 30.01 last 146.52) total: 4500 /unlink in 32.24 seconds: 139.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 750 (time 1767321894.79 total 10.01 last 74.96) - open/close 1515 (time 1767321904.80 total 20.01 last 76.45) - open/close 2338 (time 1767321914.81 total 30.03 last 82.17) - open/close 3025 (time 1767321924.82 total 40.03 last 68.67) - open/close 3714 (time 1767321934.82 total 50.04 last 68.85) - open/close 4374 (time 1767321944.82 total 60.04 last 65.99) total: 4500 open/close in 61.56 seconds: 73.10 ops/second - /unlink 1430 (time 1767321960.30 total 10.00 last 142.98) - /unlink 2817 (time 1767321970.30 total 20.00 last 138.66) - /unlink 4185 (time 1767321980.30 total 30.01 last 136.75) total: 4500 /unlink in 32.14 seconds: 140.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 676 (time 1767321997.46 total 10.00 last 67.60) - open/close 1324 (time 1767322007.47 total 20.01 last 64.72) - open/close 1927 (time 1767322017.48 total 30.02 last 60.24) - open/close 2551 (time 1767322027.49 total 40.03 last 62.37) - open/close 3216 (time 1767322037.50 total 50.04 last 66.44) - open/close 4008 (time 1767322047.51 total 60.05 last 79.11) total: 4500 open/close in 66.65 seconds: 67.52 ops/second - /unlink 1253 (time 1767322069.19 total 10.00 last 125.27) - /unlink 2308 (time 1767322079.21 total 20.02 last 105.34) - /unlink 3512 (time 1767322089.21 total 30.02 last 120.40) total: 4500 /unlink in 38.74 seconds: 116.15 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11271: 113187 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg241-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg241-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.