== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:25:39 (1786677939) Stopping clients: oleg215-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg215-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg215-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg215-server oleg215-server: oleg215-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg215-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg215-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 851 (time 1786678139.26 total 10.00 last 85.09) - open/close 1793 (time 1786678149.26 total 20.00 last 94.19) - open/close 2718 (time 1786678159.27 total 30.01 last 92.42) - open/close 3662 (time 1786678169.27 total 40.02 last 94.33) total: 4500 open/close in 49.09 seconds: 91.67 ops/second - /unlink 1696 (time 1786678192.88 total 10.00 last 169.54) - /unlink 3295 (time 1786678202.88 total 20.00 last 159.90) total: 4500 /unlink in 27.14 seconds: 165.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 1005 (time 1786678223.46 total 10.01 last 100.43) - open/close 2000 (time 1786678233.47 total 20.01 last 99.49) - open/close 2881 (time 1786678243.47 total 30.01 last 88.04) - open/close 3847 (time 1786678253.48 total 40.02 last 96.55) total: 4500 open/close in 48.50 seconds: 92.79 ops/second - /unlink 1107 (time 1786678279.30 total 10.01 last 110.62) - /unlink 2391 (time 1786678289.31 total 20.01 last 128.30) - /unlink 4169 (time 1786678299.31 total 30.02 last 177.73) total: 4500 /unlink in 31.42 seconds: 143.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 776 (time 1786678315.50 total 10.01 last 77.52) - open/close 1626 (time 1786678325.51 total 20.02 last 84.93) - open/close 2531 (time 1786678335.51 total 30.02 last 90.50) - open/close 3390 (time 1786678345.52 total 40.03 last 85.83) - open/close 4261 (time 1786678355.53 total 50.04 last 87.03) total: 4500 open/close in 52.62 seconds: 85.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1578 (time 1786678373.07 total 10.00 last 157.80) - /unlink 3270 (time 1786678383.07 total 20.01 last 169.10) total: 4500 /unlink in 27.72 seconds: 162.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 903 (time 1786678405.60 total 10.00 last 90.27) - open/close 1666 (time 1786678415.61 total 20.01 last 76.23) - open/close 2567 (time 1786678425.62 total 30.02 last 90.07) - open/close 3549 (time 1786678435.62 total 40.02 last 98.13) - open/close 4359 (time 1786678445.64 total 50.04 last 80.87) total: 4500 open/close in 51.81 seconds: 86.86 ops/second - /unlink 1425 (time 1786678461.62 total 10.00 last 142.50) - /unlink 2930 (time 1786678471.63 total 20.00 last 150.44) - /unlink 4178 (time 1786678481.63 total 30.01 last 124.75) total: 4500 /unlink in 33.48 seconds: 134.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 950 (time 1786678500.44 total 10.00 last 94.99) - open/close 1895 (time 1786678510.45 total 20.01 last 94.41) - open/close 2731 (time 1786678520.45 total 30.01 last 83.57) - open/close 3618 (time 1786678530.46 total 40.02 last 88.66) total: 4500 open/close in 49.24 seconds: 91.38 ops/second - /unlink 1524 (time 1786678553.82 total 10.00 last 152.35) - /unlink 3063 (time 1786678563.82 total 20.01 last 153.86) total: 4500 /unlink in 28.76 seconds: 156.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 968 (time 1786678587.38 total 10.01 last 96.73) - open/close 1905 (time 1786678597.40 total 20.03 last 93.52) - open/close 2789 (time 1786678607.42 total 30.05 last 88.18) - open/close 3698 (time 1786678617.43 total 40.06 last 90.82) total: 4500 open/close in 48.34 seconds: 93.09 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1795 (time 1786678639.94 total 10.00 last 179.42) - /unlink 3419 (time 1786678649.94 total 20.01 last 162.36) total: 4500 /unlink in 27.08 seconds: 166.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 900 (time 1786678671.22 total 10.00 last 90.00) - open/close 1838 (time 1786678681.22 total 20.00 last 93.79) - open/close 2838 (time 1786678691.22 total 30.00 last 99.99) - open/close 3851 (time 1786678701.23 total 40.01 last 101.26) total: 4500 open/close in 47.84 seconds: 94.07 ops/second - /unlink 1418 (time 1786678723.00 total 10.00 last 141.78) - /unlink 2607 (time 1786678733.00 total 20.00 last 118.89) - /unlink 4173 (time 1786678743.00 total 30.01 last 156.54) total: 4500 /unlink in 32.15 seconds: 139.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 634 (time 1786678759.76 total 10.01 last 63.34) - open/close 1315 (time 1786678769.77 total 20.02 last 68.05) - open/close 2181 (time 1786678779.77 total 30.02 last 86.56) - open/close 3028 (time 1786678789.78 total 40.03 last 84.66) - open/close 3897 (time 1786678799.78 total 50.03 last 86.85) total: 4500 open/close in 56.34 seconds: 79.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1749 (time 1786678820.04 total 10.00 last 174.83) - /unlink 3489 (time 1786678830.05 total 20.01 last 173.83) total: 4500 /unlink in 25.96 seconds: 173.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 799 (time 1786678850.88 total 10.01 last 79.84) - open/close 1691 (time 1786678860.91 total 20.04 last 88.91) - open/close 2578 (time 1786678870.92 total 30.04 last 88.66) - open/close 3485 (time 1786678880.92 total 40.04 last 90.70) - open/close 4414 (time 1786678890.93 total 50.06 last 92.79) total: 4500 open/close in 50.81 seconds: 88.57 ops/second - /unlink 1841 (time 1786678905.43 total 10.00 last 184.03) - /unlink 3101 (time 1786678915.43 total 20.01 last 125.91) total: 4500 /unlink in 29.35 seconds: 153.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 717 (time 1786678940.26 total 10.01 last 71.63) - open/close 1513 (time 1786678950.28 total 20.04 last 79.39) - open/close 2295 (time 1786678960.29 total 30.04 last 78.15) - open/close 3050 (time 1786678970.31 total 40.07 last 75.32) - open/close 3905 (time 1786678980.32 total 50.07 last 85.42) total: 4500 open/close in 58.45 seconds: 76.99 ops/second - /unlink 1264 (time 1786679002.89 total 10.02 last 126.15) - /unlink 2744 (time 1786679012.90 total 20.02 last 147.93) - /unlink 3953 (time 1786679022.90 total 30.03 last 120.84) total: 4500 /unlink in 34.90 seconds: 128.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 845 (time 1786679042.78 total 10.00 last 84.48) - open/close 1596 (time 1786679052.83 total 20.05 last 74.77) - open/close 2241 (time 1786679062.84 total 30.06 last 64.41) - open/close 3065 (time 1786679072.84 total 40.06 last 82.40) - open/close 3810 (time 1786679082.85 total 50.07 last 74.42) total: 4500 open/close in 59.48 seconds: 75.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1533 (time 1786679107.68 total 10.00 last 153.28) - /unlink 3164 (time 1786679117.68 total 20.00 last 163.06) total: 4500 /unlink in 29.48 seconds: 152.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 864 (time 1786679144.22 total 10.02 last 86.20) - open/close 1643 (time 1786679154.23 total 20.04 last 77.79) - open/close 2453 (time 1786679164.24 total 30.05 last 80.94) - open/close 3124 (time 1786679174.25 total 40.06 last 67.03) - open/close 3866 (time 1786679184.28 total 50.09 last 73.97) total: 4500 open/close in 57.01 seconds: 78.93 ops/second - /unlink 1672 (time 1786679206.15 total 10.01 last 167.00) - /unlink 3140 (time 1786679216.16 total 20.02 last 146.62) total: 4500 /unlink in 29.55 seconds: 152.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 951 (time 1786679240.94 total 10.00 last 95.07) - open/close 1755 (time 1786679250.94 total 20.00 last 80.38) - open/close 2595 (time 1786679260.94 total 30.01 last 83.96) - open/close 3399 (time 1786679270.94 total 40.01 last 80.38) - open/close 4152 (time 1786679280.95 total 50.02 last 75.23) total: 4500 open/close in 54.55 seconds: 82.49 ops/second - /unlink 1819 (time 1786679299.67 total 10.00 last 181.90) - /unlink 3346 (time 1786679309.67 total 20.00 last 152.70) total: 4500 /unlink in 27.48 seconds: 163.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 962 (time 1786679331.72 total 10.01 last 96.14) - open/close 1922 (time 1786679341.76 total 20.04 last 95.66) - open/close 2879 (time 1786679351.76 total 30.05 last 95.63) - open/close 3755 (time 1786679361.77 total 40.05 last 87.55) total: 4500 open/close in 48.47 seconds: 92.84 ops/second - /unlink 1778 (time 1786679384.08 total 10.00 last 177.76) - /unlink 3341 (time 1786679394.08 total 20.00 last 156.29) total: 4500 /unlink in 27.94 seconds: 161.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 938 (time 1786679417.22 total 10.00 last 93.80) - open/close 1935 (time 1786679427.24 total 20.02 last 99.48) - open/close 2781 (time 1786679437.25 total 30.03 last 84.57) - open/close 3754 (time 1786679447.26 total 40.03 last 97.24) total: 4500 open/close in 48.82 seconds: 92.18 ops/second - /unlink 1693 (time 1786679470.34 total 10.00 last 169.26) - /unlink 3467 (time 1786679480.34 total 20.01 last 177.34) total: 4500 /unlink in 26.35 seconds: 170.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 736 (time 1786679501.84 total 10.02 last 73.46) - open/close 1512 (time 1786679511.85 total 20.03 last 77.53) - open/close 2323 (time 1786679521.86 total 30.04 last 81.02) - open/close 3178 (time 1786679531.87 total 40.05 last 85.39) - open/close 3998 (time 1786679541.87 total 50.05 last 81.98) total: 4500 open/close in 55.94 seconds: 80.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1563 (time 1786679561.83 total 10.00 last 156.26) - /unlink 3068 (time 1786679571.85 total 20.02 last 150.30) total: 4500 /unlink in 29.56 seconds: 152.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 845 (time 1786679596.37 total 10.02 last 84.37) - open/close 1724 (time 1786679606.37 total 20.02 last 87.87) - open/close 2614 (time 1786679616.39 total 30.04 last 88.85) - open/close 3590 (time 1786679626.40 total 40.05 last 97.47) - open/close 4484 (time 1786679636.41 total 50.06 last 89.31) total: 4500 open/close in 50.29 seconds: 89.49 ops/second - /unlink 1538 (time 1786679651.27 total 10.00 last 153.77) - /unlink 3200 (time 1786679661.27 total 20.00 last 166.17) total: 4500 /unlink in 28.94 seconds: 155.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 905 (time 1786679685.25 total 10.01 last 90.43) - open/close 1866 (time 1786679695.26 total 20.01 last 96.05) - open/close 2745 (time 1786679705.26 total 30.02 last 87.82) - open/close 3623 (time 1786679715.28 total 40.04 last 87.65) - open/close 4446 (time 1786679725.29 total 50.04 last 82.28) total: 4500 open/close in 50.88 seconds: 88.45 ops/second - /unlink 1355 (time 1786679740.68 total 10.01 last 135.43) - /unlink 3043 (time 1786679750.69 total 20.01 last 168.76) - /unlink 4395 (time 1786679760.69 total 30.01 last 135.13) total: 4500 /unlink in 31.22 seconds: 144.16 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95178 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: modules unloaded.