== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:22:46 (1786422166) Stopping clients: oleg301-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg301-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg301-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg301-server oleg301-server: oleg301-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg301-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg301-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 730 (time 1786422388.22 total 10.01 last 72.93) - open/close 1429 (time 1786422398.23 total 20.02 last 69.84) - open/close 2177 (time 1786422408.24 total 30.02 last 74.78) - open/close 2896 (time 1786422418.27 total 40.06 last 71.64) - open/close 3696 (time 1786422428.27 total 50.06 last 79.99) - open/close 4409 (time 1786422438.28 total 60.07 last 71.22) total: 4500 open/close in 61.37 seconds: 73.33 ops/second - /unlink 1264 (time 1786422454.07 total 10.00 last 126.39) - /unlink 2518 (time 1786422464.07 total 20.00 last 125.37) - /unlink 3721 (time 1786422474.07 total 30.00 last 120.29) total: 4500 /unlink in 36.99 seconds: 121.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 672 (time 1786422497.59 total 10.09 last 66.60) - open/close 1401 (time 1786422507.60 total 20.09 last 72.87) - open/close 2154 (time 1786422517.60 total 30.10 last 75.29) - open/close 2866 (time 1786422527.60 total 40.10 last 71.18) - open/close 3688 (time 1786422537.62 total 50.11 last 82.10) - open/close 4436 (time 1786422547.62 total 60.12 last 74.76) total: 4500 open/close in 60.90 seconds: 73.89 ops/second - /unlink 1389 (time 1786422562.65 total 10.00 last 138.86) - /unlink 2805 (time 1786422572.65 total 20.01 last 141.54) - /unlink 4049 (time 1786422582.65 total 30.01 last 124.38) total: 4500 /unlink in 33.87 seconds: 132.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 713 (time 1786422602.20 total 10.01 last 71.26) - open/close 1463 (time 1786422612.28 total 20.08 last 74.47) - open/close 2168 (time 1786422622.30 total 30.10 last 70.35) - open/close 2898 (time 1786422632.30 total 40.10 last 72.99) - open/close 3533 (time 1786422642.30 total 50.10 last 63.47) - open/close 4227 (time 1786422652.31 total 60.11 last 69.34) total: 4500 open/close in 63.71 seconds: 70.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1349 (time 1786422670.92 total 10.00 last 134.84) - /unlink 2620 (time 1786422681.22 total 20.31 last 123.37) - /unlink 3879 (time 1786422691.27 total 30.36 last 125.27) total: 4500 /unlink in 35.24 seconds: 127.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 711 (time 1786422710.98 total 10.00 last 71.08) - open/close 1486 (time 1786422720.99 total 20.01 last 77.43) - open/close 2222 (time 1786422731.00 total 30.02 last 73.55) - open/close 2908 (time 1786422741.01 total 40.03 last 68.55) - open/close 3572 (time 1786422751.01 total 50.03 last 66.39) - open/close 4247 (time 1786422761.01 total 60.03 last 67.49) total: 4500 open/close in 64.08 seconds: 70.22 ops/second - /unlink 1374 (time 1786422779.97 total 10.04 last 136.84) - /unlink 2529 (time 1786422789.97 total 20.04 last 115.49) - /unlink 3686 (time 1786422799.97 total 30.04 last 115.67) total: 4500 /unlink in 36.45 seconds: 123.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 740 (time 1786422822.05 total 10.01 last 73.95) - open/close 1314 (time 1786422832.06 total 20.01 last 57.36) - open/close 2010 (time 1786422842.07 total 30.02 last 69.56) - open/close 2622 (time 1786422852.11 total 40.06 last 60.96) - open/close 3269 (time 1786422862.11 total 50.06 last 64.68) - open/close 3861 (time 1786422872.18 total 60.13 last 58.79) - open/close 4436 (time 1786422882.18 total 70.13 last 57.50) total: 4500 open/close in 70.89 seconds: 63.48 ops/second - /unlink 1253 (time 1786422898.76 total 10.01 last 125.24) - /unlink 2396 (time 1786422908.76 total 20.01 last 114.28) - /unlink 3535 (time 1786422918.76 total 30.01 last 113.84) total: 4500 /unlink in 38.38 seconds: 117.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 782 (time 1786422942.94 total 10.03 last 77.98) - open/close 1539 (time 1786422952.95 total 20.04 last 75.64) - open/close 2240 (time 1786422962.96 total 30.05 last 70.01) - open/close 3040 (time 1786422972.97 total 40.06 last 79.94) - open/close 3800 (time 1786422982.98 total 50.07 last 75.93) total: 4500 open/close in 59.60 seconds: 75.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1318 (time 1786423007.90 total 10.00 last 131.74) - /unlink 2469 (time 1786423017.91 total 20.01 last 115.07) - /unlink 3728 (time 1786423027.95 total 30.05 last 125.34) total: 4500 /unlink in 36.57 seconds: 123.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 749 (time 1786423049.63 total 10.01 last 74.84) - open/close 1503 (time 1786423059.64 total 20.02 last 75.34) - open/close 2263 (time 1786423069.64 total 30.02 last 75.98) - open/close 2987 (time 1786423079.64 total 40.02 last 72.40) - open/close 3724 (time 1786423089.64 total 50.02 last 73.68) total: 4500 open/close in 59.91 seconds: 75.12 ops/second - /unlink 1455 (time 1786423113.74 total 10.00 last 145.45) - /unlink 2720 (time 1786423123.74 total 20.00 last 126.49) - /unlink 3881 (time 1786423133.74 total 30.01 last 116.06) total: 4500 /unlink in 35.28 seconds: 127.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 782 (time 1786423154.42 total 10.01 last 78.16) - open/close 1533 (time 1786423164.43 total 20.01 last 75.08) - open/close 2365 (time 1786423174.44 total 30.02 last 83.09) - open/close 3102 (time 1786423184.45 total 40.03 last 73.65) - open/close 3879 (time 1786423194.53 total 50.11 last 77.07) total: 4500 open/close in 57.95 seconds: 77.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1209 (time 1786423216.74 total 10.00 last 120.88) - /unlink 2452 (time 1786423226.75 total 20.01 last 124.14) - /unlink 3654 (time 1786423236.75 total 30.02 last 120.17) total: 4500 /unlink in 37.31 seconds: 120.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 768 (time 1786423259.02 total 10.01 last 76.74) - open/close 1521 (time 1786423269.03 total 20.02 last 75.22) - open/close 2307 (time 1786423279.05 total 30.03 last 78.50) - open/close 3028 (time 1786423289.05 total 40.03 last 72.10) - open/close 3871 (time 1786423299.05 total 50.04 last 84.27) total: 4500 open/close in 58.90 seconds: 76.40 ops/second - /unlink 1236 (time 1786423322.52 total 10.00 last 123.55) - /unlink 2584 (time 1786423332.53 total 20.01 last 134.71) - /unlink 3816 (time 1786423342.54 total 30.02 last 123.09) total: 4500 /unlink in 35.57 seconds: 126.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 725 (time 1786423363.56 total 10.01 last 72.45) - open/close 1486 (time 1786423373.56 total 20.01 last 76.09) - open/close 2198 (time 1786423383.57 total 30.02 last 71.16) - open/close 2904 (time 1786423393.57 total 40.02 last 70.56) - open/close 3610 (time 1786423403.57 total 50.02 last 70.58) - open/close 4333 (time 1786423413.58 total 60.02 last 72.30) total: 4500 open/close in 62.47 seconds: 72.03 ops/second - /unlink 1136 (time 1786423429.81 total 10.00 last 113.56) - /unlink 2312 (time 1786423439.82 total 20.02 last 117.43) - /unlink 3383 (time 1786423449.82 total 30.02 last 107.08) total: 4500 /unlink in 39.18 seconds: 114.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 708 (time 1786423474.62 total 10.01 last 70.75) - open/close 1440 (time 1786423484.62 total 20.01 last 73.14) - open/close 2201 (time 1786423494.64 total 30.03 last 76.01) - open/close 2900 (time 1786423504.66 total 40.05 last 69.72) - open/close 3655 (time 1786423514.66 total 50.05 last 75.48) - open/close 4351 (time 1786423524.67 total 60.06 last 69.58) total: 4500 open/close in 61.98 seconds: 72.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1408 (time 1786423541.39 total 10.01 last 140.66) - /unlink 2672 (time 1786423551.39 total 20.01 last 126.40) - /unlink 4067 (time 1786423561.39 total 30.01 last 139.44) total: 4500 /unlink in 33.40 seconds: 134.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 749 (time 1786423580.50 total 10.00 last 74.88) - open/close 1491 (time 1786423590.50 total 20.01 last 74.17) - open/close 2244 (time 1786423600.50 total 30.01 last 75.29) - open/close 3075 (time 1786423610.51 total 40.02 last 83.04) - open/close 3823 (time 1786423620.52 total 50.02 last 74.74) total: 4500 open/close in 59.92 seconds: 75.10 ops/second - /unlink 1278 (time 1786423644.53 total 10.00 last 127.75) - /unlink 2572 (time 1786423654.54 total 20.02 last 129.25) - /unlink 3968 (time 1786423664.54 total 30.02 last 139.57) total: 4500 /unlink in 34.19 seconds: 131.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 771 (time 1786423683.63 total 10.01 last 77.06) - open/close 1489 (time 1786423693.64 total 20.02 last 71.71) - open/close 2242 (time 1786423703.65 total 30.03 last 75.22) - open/close 2997 (time 1786423713.66 total 40.03 last 75.47) - open/close 3742 (time 1786423723.67 total 50.05 last 74.41) total: 4500 open/close in 59.93 seconds: 75.09 ops/second - /unlink 1212 (time 1786423748.35 total 10.00 last 121.18) - /unlink 2492 (time 1786423758.35 total 20.00 last 127.99) - /unlink 3773 (time 1786423768.36 total 30.01 last 128.03) total: 4500 /unlink in 35.39 seconds: 127.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 757 (time 1786423788.45 total 10.01 last 75.61) - open/close 1545 (time 1786423798.46 total 20.02 last 78.71) - open/close 2285 (time 1786423808.47 total 30.03 last 73.92) - open/close 3059 (time 1786423818.48 total 40.05 last 77.31) - open/close 3762 (time 1786423828.53 total 50.10 last 69.92) total: 4500 open/close in 59.40 seconds: 75.75 ops/second - /unlink 1372 (time 1786423852.35 total 10.00 last 137.17) - /unlink 2841 (time 1786423862.35 total 20.00 last 146.89) - /unlink 4181 (time 1786423872.36 total 30.01 last 133.94) total: 4500 /unlink in 32.46 seconds: 138.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 835 (time 1786423889.66 total 10.01 last 83.44) - open/close 1585 (time 1786423899.66 total 20.02 last 74.94) - open/close 2399 (time 1786423909.67 total 30.02 last 81.37) - open/close 3212 (time 1786423919.67 total 40.02 last 81.26) - open/close 4033 (time 1786423929.75 total 50.11 last 81.43) total: 4500 open/close in 55.84 seconds: 80.58 ops/second - /unlink 1256 (time 1786423949.74 total 10.00 last 125.59) - /unlink 2508 (time 1786423959.75 total 20.00 last 125.15) - /unlink 3875 (time 1786423969.75 total 30.01 last 136.65) total: 4500 /unlink in 35.12 seconds: 128.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 805 (time 1786423989.42 total 10.00 last 80.47) - open/close 1601 (time 1786423999.43 total 20.01 last 79.54) - open/close 2387 (time 1786424009.44 total 30.02 last 78.50) - open/close 3247 (time 1786424019.44 total 40.03 last 85.98) - open/close 4071 (time 1786424029.49 total 50.08 last 81.99) total: 4500 open/close in 55.14 seconds: 81.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1438 (time 1786424048.42 total 10.00 last 143.73) - /unlink 2757 (time 1786424058.44 total 20.02 last 131.64) - /unlink 4090 (time 1786424068.44 total 30.03 last 133.25) total: 4500 /unlink in 32.87 seconds: 136.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 885 (time 1786424085.45 total 10.01 last 88.45) - open/close 1715 (time 1786424095.46 total 20.01 last 82.94) - open/close 2546 (time 1786424105.46 total 30.02 last 83.07) - open/close 3424 (time 1786424115.47 total 40.02 last 87.73) - open/close 4163 (time 1786424125.47 total 50.03 last 73.86) total: 4500 open/close in 54.29 seconds: 82.89 ops/second - /unlink 1300 (time 1786424143.92 total 10.00 last 129.94) - /unlink 2678 (time 1786424153.92 total 20.01 last 137.75) - /unlink 3972 (time 1786424163.93 total 30.01 last 129.35) total: 4500 /unlink in 33.63 seconds: 133.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 757 (time 1786424182.06 total 10.00 last 75.66) - open/close 1534 (time 1786424192.06 total 20.01 last 77.65) - open/close 2338 (time 1786424202.07 total 30.02 last 80.34) - open/close 3152 (time 1786424212.08 total 40.03 last 81.34) - open/close 3955 (time 1786424222.08 total 50.03 last 80.24) total: 4500 open/close in 57.45 seconds: 78.33 ops/second - /unlink 1311 (time 1786424243.50 total 10.00 last 131.09) - /unlink 2438 (time 1786424253.50 total 20.00 last 112.70) - /unlink 3841 (time 1786424263.50 total 30.00 last 140.28) total: 4500 /unlink in 35.00 seconds: 128.56 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95267 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: modules unloaded.