== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:25:25 (1786832725) Stopping clients: oleg460-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg460-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg460-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg460-server oleg460-server: oleg460-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg460-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg460-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 761 (time 1786832947.49 total 10.00 last 76.06) - open/close 1463 (time 1786832957.50 total 20.01 last 70.14) - open/close 2225 (time 1786832967.53 total 30.05 last 75.92) - open/close 2999 (time 1786832977.53 total 40.05 last 77.40) - open/close 3714 (time 1786832987.55 total 50.07 last 71.40) - open/close 4277 (time 1786832997.56 total 60.08 last 56.24) total: 4500 open/close in 63.88 seconds: 70.45 ops/second - /unlink 1413 (time 1786833017.35 total 10.00 last 141.26) - /unlink 2714 (time 1786833027.35 total 20.01 last 130.04) - /unlink 4158 (time 1786833037.35 total 30.01 last 144.39) total: 4500 /unlink in 32.20 seconds: 139.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 766 (time 1786833055.36 total 10.01 last 76.51) - open/close 1529 (time 1786833065.37 total 20.02 last 76.23) - open/close 2180 (time 1786833075.38 total 30.03 last 65.04) - open/close 2851 (time 1786833085.39 total 40.04 last 67.01) - open/close 3580 (time 1786833095.40 total 50.05 last 72.88) - open/close 4163 (time 1786833105.40 total 60.05 last 58.30) total: 4500 open/close in 64.59 seconds: 69.67 ops/second - /unlink 1311 (time 1786833123.98 total 10.00 last 131.07) - /unlink 2459 (time 1786833133.98 total 20.00 last 114.78) - /unlink 3440 (time 1786833143.99 total 30.02 last 97.93) total: 4500 /unlink in 39.46 seconds: 114.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 715 (time 1786833168.07 total 10.01 last 71.39) - open/close 1482 (time 1786833178.08 total 20.03 last 76.62) - open/close 2113 (time 1786833188.10 total 30.05 last 62.94) - open/close 2773 (time 1786833198.12 total 40.06 last 65.92) - open/close 3304 (time 1786833208.15 total 50.10 last 52.93) - open/close 4081 (time 1786833218.16 total 60.10 last 77.64) total: 4500 open/close in 67.04 seconds: 67.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1373 (time 1786833239.74 total 10.00 last 137.29) - /unlink 2665 (time 1786833249.74 total 20.01 last 129.14) - /unlink 4036 (time 1786833259.74 total 30.01 last 137.07) total: 4500 /unlink in 33.35 seconds: 134.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 763 (time 1786833277.40 total 10.00 last 76.27) - open/close 1562 (time 1786833287.40 total 20.01 last 79.89) - open/close 2261 (time 1786833297.41 total 30.02 last 69.80) - open/close 2999 (time 1786833307.43 total 40.04 last 73.67) - open/close 3651 (time 1786833317.43 total 50.04 last 65.19) - open/close 4369 (time 1786833327.44 total 60.04 last 71.78) total: 4500 open/close in 62.01 seconds: 72.57 ops/second - /unlink 1195 (time 1786833344.40 total 10.02 last 119.31) - /unlink 2505 (time 1786833354.41 total 20.02 last 130.97) - /unlink 3871 (time 1786833364.41 total 30.02 last 136.59) total: 4500 /unlink in 35.08 seconds: 128.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 602 (time 1786833384.65 total 10.01 last 60.14) - open/close 1158 (time 1786833394.65 total 20.01 last 55.58) - open/close 1864 (time 1786833404.66 total 30.02 last 70.53) - open/close 2629 (time 1786833414.66 total 40.03 last 76.48) - open/close 3305 (time 1786833424.66 total 50.03 last 67.60) - open/close 4123 (time 1786833434.67 total 60.03 last 81.80) total: 4500 open/close in 65.25 seconds: 68.96 ops/second - /unlink 1475 (time 1786833453.76 total 10.00 last 147.46) - /unlink 2813 (time 1786833463.76 total 20.00 last 133.80) - /unlink 4223 (time 1786833473.77 total 30.01 last 140.95) total: 4500 /unlink in 32.03 seconds: 140.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 635 (time 1786833490.23 total 10.01 last 63.41) - open/close 1271 (time 1786833500.25 total 20.03 last 63.52) - open/close 2006 (time 1786833510.25 total 30.03 last 73.44) - open/close 2726 (time 1786833520.26 total 40.04 last 71.96) - open/close 3532 (time 1786833530.27 total 50.05 last 80.51) - open/close 4362 (time 1786833540.28 total 60.06 last 82.96) total: 4500 open/close in 61.73 seconds: 72.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1392 (time 1786833556.62 total 10.00 last 139.18) - /unlink 2808 (time 1786833566.62 total 20.01 last 141.54) - /unlink 4199 (time 1786833576.63 total 30.01 last 139.00) total: 4500 /unlink in 32.27 seconds: 139.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 819 (time 1786833593.21 total 10.01 last 81.82) - open/close 1505 (time 1786833603.21 total 20.02 last 68.55) - open/close 2118 (time 1786833613.22 total 30.03 last 61.23) - open/close 2728 (time 1786833623.26 total 40.06 last 60.81) - open/close 3382 (time 1786833633.26 total 50.06 last 65.37) - open/close 4106 (time 1786833643.26 total 60.07 last 72.38) total: 4500 open/close in 65.87 seconds: 68.32 ops/second - /unlink 1327 (time 1786833663.86 total 10.00 last 132.65) - /unlink 2661 (time 1786833673.86 total 20.01 last 133.38) - /unlink 4110 (time 1786833683.86 total 30.01 last 144.87) total: 4500 /unlink in 32.65 seconds: 137.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 739 (time 1786833701.31 total 10.01 last 73.81) - open/close 1528 (time 1786833711.32 total 20.02 last 78.86) - open/close 2284 (time 1786833721.32 total 30.02 last 75.58) - open/close 3018 (time 1786833731.33 total 40.03 last 73.34) - open/close 3657 (time 1786833741.35 total 50.04 last 63.80) - open/close 4330 (time 1786833751.35 total 60.05 last 67.29) total: 4500 open/close in 62.78 seconds: 71.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1136 (time 1786833768.72 total 10.00 last 113.58) - /unlink 2318 (time 1786833778.72 total 20.00 last 118.19) - /unlink 3485 (time 1786833788.72 total 30.01 last 116.66) total: 4500 /unlink in 38.24 seconds: 117.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 758 (time 1786833811.36 total 10.00 last 75.78) - open/close 1551 (time 1786833821.37 total 20.01 last 79.24) - open/close 2275 (time 1786833831.37 total 30.02 last 72.34) - open/close 3005 (time 1786833841.37 total 40.02 last 72.99) - open/close 3760 (time 1786833851.38 total 50.02 last 75.49) total: 4500 open/close in 59.90 seconds: 75.12 ops/second - /unlink 1128 (time 1786833876.68 total 10.00 last 112.75) - /unlink 2515 (time 1786833886.68 total 20.01 last 138.67) - /unlink 3890 (time 1786833896.68 total 30.01 last 137.47) total: 4500 /unlink in 34.46 seconds: 130.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 764 (time 1786833915.58 total 10.01 last 76.31) - open/close 1432 (time 1786833925.59 total 20.02 last 66.73) - open/close 2148 (time 1786833935.60 total 30.03 last 71.56) - open/close 2868 (time 1786833945.62 total 40.04 last 71.87) - open/close 3580 (time 1786833955.63 total 50.05 last 71.13) - open/close 4343 (time 1786833965.63 total 60.06 last 76.28) total: 4500 open/close in 62.00 seconds: 72.59 ops/second - /unlink 1248 (time 1786833981.88 total 10.01 last 124.70) - /unlink 2493 (time 1786833991.88 total 20.01 last 124.48) - /unlink 3725 (time 1786834001.88 total 30.01 last 123.19) total: 4500 /unlink in 35.83 seconds: 125.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 701 (time 1786834021.88 total 10.02 last 69.93) - open/close 1456 (time 1786834031.90 total 20.04 last 75.37) - open/close 2069 (time 1786834041.91 total 30.05 last 61.23) - open/close 2764 (time 1786834051.92 total 40.07 last 69.42) - open/close 3418 (time 1786834061.93 total 50.08 last 65.33) - open/close 4188 (time 1786834071.94 total 60.09 last 76.93) total: 4500 open/close in 65.05 seconds: 69.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1314 (time 1786834091.64 total 10.00 last 131.40) - /unlink 2556 (time 1786834101.65 total 20.00 last 124.16) - /unlink 3759 (time 1786834111.66 total 30.02 last 120.12) total: 4500 /unlink in 36.19 seconds: 124.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 700 (time 1786834132.21 total 10.00 last 69.98) - open/close 1369 (time 1786834142.21 total 20.01 last 66.84) - open/close 2067 (time 1786834152.22 total 30.02 last 69.74) - open/close 2810 (time 1786834162.23 total 40.03 last 74.24) - open/close 3615 (time 1786834172.24 total 50.03 last 80.44) - open/close 4379 (time 1786834182.24 total 60.04 last 76.36) total: 4500 open/close in 61.97 seconds: 72.62 ops/second - /unlink 1068 (time 1786834198.35 total 10.01 last 106.75) - /unlink 2348 (time 1786834208.35 total 20.01 last 127.95) - /unlink 3418 (time 1786834218.36 total 30.01 last 106.97) total: 4500 /unlink in 37.92 seconds: 118.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 716 (time 1786834241.50 total 10.02 last 71.45) - open/close 1442 (time 1786834251.51 total 20.03 last 72.52) - open/close 2175 (time 1786834261.52 total 30.04 last 73.22) - open/close 2854 (time 1786834271.55 total 40.07 last 67.72) - open/close 3511 (time 1786834281.55 total 50.07 last 65.68) - open/close 4288 (time 1786834291.55 total 60.07 last 77.70) total: 4500 open/close in 63.19 seconds: 71.22 ops/second - /unlink 1320 (time 1786834309.20 total 10.00 last 131.95) - /unlink 2631 (time 1786834319.21 total 20.01 last 131.07) - /unlink 3916 (time 1786834329.21 total 30.01 last 128.41) total: 4500 /unlink in 34.66 seconds: 129.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 648 (time 1786834348.49 total 10.01 last 64.73) - open/close 1325 (time 1786834358.50 total 20.02 last 67.64) - open/close 2018 (time 1786834368.51 total 30.03 last 69.23) - open/close 2719 (time 1786834378.51 total 40.03 last 70.07) - open/close 3468 (time 1786834388.52 total 50.04 last 74.85) - open/close 4245 (time 1786834398.53 total 60.05 last 77.64) total: 4500 open/close in 64.00 seconds: 70.32 ops/second - /unlink 1298 (time 1786834417.23 total 10.01 last 129.73) - /unlink 2503 (time 1786834427.24 total 20.01 last 120.39) - /unlink 3572 (time 1786834437.25 total 30.03 last 106.76) total: 4500 /unlink in 39.59 seconds: 113.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 536 (time 1786834463.39 total 10.01 last 53.52) - open/close 1177 (time 1786834473.40 total 20.02 last 64.05) - open/close 1854 (time 1786834483.41 total 30.03 last 67.64) - open/close 2622 (time 1786834493.42 total 40.04 last 76.75) - open/close 3275 (time 1786834503.43 total 50.05 last 65.25) - open/close 4061 (time 1786834513.43 total 60.05 last 78.59) total: 4500 open/close in 66.20 seconds: 67.97 ops/second - /unlink 1293 (time 1786834533.80 total 10.00 last 129.30) - /unlink 2537 (time 1786834543.80 total 20.00 last 124.34) - /unlink 3788 (time 1786834553.80 total 30.01 last 125.09) total: 4500 /unlink in 36.27 seconds: 124.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 696 (time 1786834574.80 total 10.01 last 69.52) - open/close 1369 (time 1786834584.85 total 20.06 last 66.96) - open/close 2051 (time 1786834594.86 total 30.07 last 68.14) - open/close 2775 (time 1786834604.88 total 40.10 last 72.21) - open/close 3460 (time 1786834614.88 total 50.10 last 68.49) - open/close 4247 (time 1786834624.90 total 60.11 last 78.58) total: 4500 open/close in 63.55 seconds: 70.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1356 (time 1786834642.98 total 10.00 last 135.53) - /unlink 2668 (time 1786834652.98 total 20.01 last 131.17) - /unlink 4041 (time 1786834662.99 total 30.01 last 137.22) total: 4500 /unlink in 33.10 seconds: 135.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 775 (time 1786834680.11 total 10.00 last 77.49) - open/close 1499 (time 1786834690.13 total 20.02 last 72.25) - open/close 2217 (time 1786834700.13 total 30.03 last 71.77) - open/close 3007 (time 1786834710.14 total 40.03 last 78.97) - open/close 3764 (time 1786834720.14 total 50.04 last 75.65) total: 4500 open/close in 59.72 seconds: 75.35 ops/second - /unlink 1510 (time 1786834744.28 total 10.00 last 150.93) - /unlink 2825 (time 1786834754.28 total 20.01 last 131.40) - /unlink 4070 (time 1786834764.28 total 30.01 last 124.49) total: 4500 /unlink in 33.41 seconds: 134.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 733 (time 1786834782.26 total 10.01 last 73.25) - open/close 1427 (time 1786834792.27 total 20.02 last 69.34) - open/close 2171 (time 1786834802.27 total 30.02 last 74.39) - open/close 2926 (time 1786834812.28 total 40.03 last 75.41) - open/close 3688 (time 1786834822.29 total 50.03 last 76.16) - open/close 4438 (time 1786834832.30 total 60.04 last 74.94) total: 4500 open/close in 60.94 seconds: 73.85 ops/second - /unlink 1374 (time 1786834846.81 total 10.01 last 137.32) - /unlink 2724 (time 1786834856.82 total 20.01 last 134.93) - /unlink 4182 (time 1786834866.82 total 30.01 last 145.79) total: 4500 /unlink in 32.16 seconds: 139.91 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95370 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: modules unloaded.