== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:43:43 (1786736623) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg260-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg260-server oleg260-server: oleg260-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg260-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg260-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 839 (time 1786736830.71 total 10.01 last 83.85) - open/close 1550 (time 1786736840.71 total 20.01 last 71.05) - open/close 2247 (time 1786736850.72 total 30.03 last 69.61) - open/close 3108 (time 1786736860.73 total 40.03 last 86.07) - open/close 3944 (time 1786736870.74 total 50.04 last 83.54) total: 4500 open/close in 56.29 seconds: 79.95 ops/second - /unlink 1571 (time 1786736891.08 total 10.01 last 157.02) - /unlink 2988 (time 1786736901.08 total 20.01 last 141.68) - /unlink 4375 (time 1786736911.09 total 30.01 last 138.64) total: 4500 /unlink in 31.19 seconds: 144.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 738 (time 1786736927.69 total 10.01 last 73.73) - open/close 1389 (time 1786736937.70 total 20.02 last 65.03) - open/close 2206 (time 1786736947.70 total 30.02 last 81.68) - open/close 3108 (time 1786736957.71 total 40.03 last 90.13) - open/close 3967 (time 1786736967.72 total 50.04 last 85.86) total: 4500 open/close in 55.77 seconds: 80.69 ops/second - /unlink 1073 (time 1786736989.38 total 10.01 last 107.21) - /unlink 2582 (time 1786736999.39 total 20.02 last 150.78) - /unlink 4261 (time 1786737009.39 total 30.02 last 167.77) total: 4500 /unlink in 31.77 seconds: 141.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 991 (time 1786737026.90 total 10.01 last 99.02) - open/close 1790 (time 1786737036.92 total 20.02 last 79.78) - open/close 2533 (time 1786737046.95 total 30.06 last 74.04) - open/close 3316 (time 1786737056.95 total 40.06 last 78.29) - open/close 4181 (time 1786737066.95 total 50.06 last 86.49) total: 4500 open/close in 53.78 seconds: 83.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1325 (time 1786737085.33 total 10.00 last 132.48) - /unlink 2872 (time 1786737095.33 total 20.00 last 154.66) - /unlink 4418 (time 1786737105.34 total 30.01 last 154.49) total: 4500 /unlink in 30.62 seconds: 146.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 838 (time 1786737120.28 total 10.00 last 83.79) - open/close 1643 (time 1786737130.29 total 20.01 last 80.45) - open/close 2406 (time 1786737140.32 total 30.04 last 76.09) - open/close 3190 (time 1786737150.33 total 40.05 last 78.31) - open/close 3977 (time 1786737160.34 total 50.06 last 78.60) total: 4500 open/close in 56.63 seconds: 79.46 ops/second - /unlink 1613 (time 1786737180.54 total 10.00 last 161.30) - /unlink 3181 (time 1786737190.54 total 20.00 last 156.78) total: 4500 /unlink in 29.95 seconds: 150.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 841 (time 1786737215.33 total 10.01 last 84.02) - open/close 1627 (time 1786737225.35 total 20.04 last 78.40) - open/close 2402 (time 1786737235.36 total 30.05 last 77.42) - open/close 3131 (time 1786737245.37 total 40.05 last 72.88) - open/close 4043 (time 1786737255.38 total 50.06 last 91.10) total: 4500 open/close in 55.52 seconds: 81.05 ops/second - /unlink 1341 (time 1786737275.19 total 10.00 last 134.09) - /unlink 2704 (time 1786737285.20 total 20.01 last 136.25) - /unlink 4143 (time 1786737295.21 total 30.02 last 143.75) total: 4500 /unlink in 32.85 seconds: 136.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 788 (time 1786737314.04 total 10.00 last 78.78) - open/close 1526 (time 1786737324.05 total 20.01 last 73.75) - open/close 2439 (time 1786737334.05 total 30.01 last 91.29) - open/close 3583 (time 1786737344.05 total 40.01 last 114.38) - open/close 4346 (time 1786737354.06 total 50.01 last 76.28) total: 4500 open/close in 52.05 seconds: 86.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1513 (time 1786737371.50 total 10.00 last 151.30) - /unlink 3009 (time 1786737381.50 total 20.01 last 149.49) - /unlink 4498 (time 1786737391.51 total 30.01 last 148.83) total: 4500 /unlink in 30.02 seconds: 149.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 664 (time 1786737406.43 total 10.00 last 66.38) - open/close 1483 (time 1786737416.44 total 20.01 last 81.81) - open/close 2141 (time 1786737426.45 total 30.01 last 65.80) - open/close 2950 (time 1786737436.45 total 40.02 last 80.84) - open/close 3809 (time 1786737446.46 total 50.03 last 85.87) total: 4500 open/close in 58.61 seconds: 76.78 ops/second - /unlink 1463 (time 1786737469.38 total 10.01 last 146.17) - /unlink 2756 (time 1786737479.39 total 20.02 last 129.18) - /unlink 4263 (time 1786737489.39 total 30.02 last 150.68) total: 4500 /unlink in 31.71 seconds: 141.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 771 (time 1786737505.92 total 10.00 last 77.09) - open/close 1572 (time 1786737515.93 total 20.01 last 80.03) - open/close 2294 (time 1786737525.93 total 30.01 last 72.18) - open/close 3027 (time 1786737535.94 total 40.02 last 73.21) - open/close 3846 (time 1786737545.94 total 50.03 last 81.87) total: 4500 open/close in 57.90 seconds: 77.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1206 (time 1786737568.78 total 10.00 last 120.58) - /unlink 2442 (time 1786737578.78 total 20.00 last 123.59) - /unlink 3866 (time 1786737588.78 total 30.01 last 142.33) total: 4500 /unlink in 35.01 seconds: 128.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 690 (time 1786737608.79 total 10.01 last 68.96) - open/close 1273 (time 1786737618.79 total 20.01 last 58.28) - open/close 2152 (time 1786737628.80 total 30.02 last 87.79) - open/close 2922 (time 1786737638.80 total 40.02 last 76.98) - open/close 3794 (time 1786737648.81 total 50.03 last 87.19) total: 4500 open/close in 58.31 seconds: 77.18 ops/second - /unlink 1444 (time 1786737671.67 total 10.01 last 144.29) - /unlink 2723 (time 1786737681.67 total 20.01 last 127.89) - /unlink 4148 (time 1786737691.68 total 30.01 last 142.46) total: 4500 /unlink in 32.33 seconds: 139.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 842 (time 1786737709.26 total 10.01 last 84.15) - open/close 1647 (time 1786737719.27 total 20.01 last 80.48) - open/close 2515 (time 1786737729.27 total 30.02 last 86.72) - open/close 3462 (time 1786737739.28 total 40.02 last 94.65) - open/close 4459 (time 1786737749.28 total 50.03 last 99.66) total: 4500 open/close in 50.52 seconds: 89.07 ops/second - /unlink 1435 (time 1786737764.22 total 10.01 last 143.30) - /unlink 2997 (time 1786737774.22 total 20.02 last 156.16) total: 4500 /unlink in 29.80 seconds: 151.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 805 (time 1786737798.79 total 10.01 last 80.39) - open/close 1488 (time 1786737808.80 total 20.02 last 68.27) - open/close 2277 (time 1786737818.80 total 30.02 last 78.84) - open/close 3081 (time 1786737828.81 total 40.03 last 80.38) - open/close 3739 (time 1786737838.82 total 50.04 last 65.69) - open/close 4404 (time 1786737848.88 total 60.10 last 66.15) total: 4500 open/close in 61.31 seconds: 73.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1718 (time 1786737863.68 total 10.00 last 171.74) - /unlink 3289 (time 1786737873.69 total 20.01 last 157.05) total: 4500 /unlink in 28.23 seconds: 159.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 893 (time 1786737897.26 total 10.02 last 89.10) - open/close 1700 (time 1786737907.27 total 20.02 last 80.69) - open/close 2602 (time 1786737917.27 total 30.03 last 90.19) - open/close 3386 (time 1786737927.28 total 40.03 last 78.33) - open/close 4218 (time 1786737937.28 total 50.04 last 83.17) total: 4500 open/close in 54.12 seconds: 83.15 ops/second - /unlink 1476 (time 1786737956.43 total 10.01 last 147.43) - /unlink 2974 (time 1786737966.44 total 20.02 last 149.73) - /unlink 4455 (time 1786737976.44 total 30.02 last 148.07) total: 4500 /unlink in 30.42 seconds: 147.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 742 (time 1786737992.00 total 10.01 last 74.14) - open/close 1584 (time 1786738002.01 total 20.02 last 84.13) - open/close 2397 (time 1786738012.02 total 30.02 last 81.25) - open/close 3284 (time 1786738022.02 total 40.03 last 88.65) - open/close 3957 (time 1786738032.03 total 50.03 last 67.27) total: 4500 open/close in 59.24 seconds: 75.96 ops/second - /unlink 1271 (time 1786738056.20 total 10.00 last 127.09) - /unlink 2840 (time 1786738066.21 total 20.00 last 156.87) - /unlink 4409 (time 1786738076.21 total 30.01 last 156.80) total: 4500 /unlink in 30.82 seconds: 146.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 819 (time 1786738092.38 total 10.01 last 81.80) - open/close 1649 (time 1786738102.40 total 20.03 last 82.85) - open/close 2405 (time 1786738112.42 total 30.05 last 75.46) - open/close 3249 (time 1786738122.42 total 40.05 last 84.38) - open/close 4147 (time 1786738132.42 total 50.05 last 89.80) total: 4500 open/close in 54.27 seconds: 82.92 ops/second - /unlink 1432 (time 1786738151.05 total 10.01 last 143.13) - /unlink 2461 (time 1786738161.06 total 20.02 last 102.77) - /unlink 3938 (time 1786738171.07 total 30.03 last 147.56) total: 4500 /unlink in 35.68 seconds: 126.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 706 (time 1786738190.30 total 10.01 last 70.54) - open/close 1409 (time 1786738200.31 total 20.02 last 70.22) - open/close 1957 (time 1786738210.31 total 30.02 last 54.79) - open/close 2634 (time 1786738220.32 total 40.03 last 67.66) - open/close 3332 (time 1786738230.32 total 50.03 last 69.78) - open/close 4103 (time 1786738240.33 total 60.04 last 77.00) total: 4500 open/close in 65.35 seconds: 68.86 ops/second - /unlink 1398 (time 1786738260.73 total 10.01 last 139.66) - /unlink 2894 (time 1786738270.73 total 20.01 last 149.58) - /unlink 4392 (time 1786738280.74 total 30.02 last 149.61) total: 4500 /unlink in 31.68 seconds: 142.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 657 (time 1786738297.98 total 10.00 last 65.68) - open/close 1319 (time 1786738307.99 total 20.01 last 66.14) - open/close 1939 (time 1786738317.99 total 30.02 last 61.96) - open/close 2567 (time 1786738328.00 total 40.02 last 62.78) - open/close 3410 (time 1786738338.02 total 50.04 last 84.13) - open/close 4421 (time 1786738348.05 total 60.07 last 100.76) total: 4500 open/close in 61.02 seconds: 73.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1593 (time 1786738363.31 total 10.00 last 159.24) - /unlink 3092 (time 1786738373.31 total 20.01 last 149.88) - /unlink 4389 (time 1786738383.32 total 30.01 last 129.67) total: 4500 /unlink in 30.89 seconds: 145.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 755 (time 1786738399.50 total 10.00 last 75.48) - open/close 1504 (time 1786738409.52 total 20.02 last 74.76) - open/close 2299 (time 1786738419.52 total 30.03 last 79.47) - open/close 3164 (time 1786738429.54 total 40.04 last 86.37) - open/close 3966 (time 1786738439.54 total 50.04 last 80.20) total: 4500 open/close in 58.78 seconds: 76.56 ops/second - /unlink 1422 (time 1786738463.90 total 10.01 last 142.11) - /unlink 2875 (time 1786738473.90 total 20.01 last 145.28) total: 4500 /unlink in 29.77 seconds: 151.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 758 (time 1786738498.39 total 10.00 last 75.77) - open/close 1494 (time 1786738508.40 total 20.02 last 73.52) - open/close 2175 (time 1786738518.41 total 30.02 last 68.04) - open/close 2949 (time 1786738528.42 total 40.03 last 77.36) - open/close 3668 (time 1786738538.44 total 50.05 last 71.76) total: 4500 open/close in 60.01 seconds: 74.99 ops/second - /unlink 1338 (time 1786738562.33 total 10.00 last 133.78) - /unlink 2961 (time 1786738572.33 total 20.00 last 162.29) - /unlink 4457 (time 1786738582.34 total 30.01 last 149.49) total: 4500 /unlink in 30.55 seconds: 147.29 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95168 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: modules unloaded.