== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:10:42 (1786817442) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg118-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg118-server oleg118-server: oleg118-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 703 (time 1786817659.09 total 10.00 last 70.29) - open/close 1463 (time 1786817669.10 total 20.01 last 75.95) - open/close 2130 (time 1786817679.12 total 30.02 last 66.59) - open/close 2821 (time 1786817689.13 total 40.04 last 68.99) - open/close 3559 (time 1786817699.15 total 50.05 last 73.70) - open/close 4118 (time 1786817709.15 total 60.06 last 55.89) total: 4500 open/close in 66.31 seconds: 67.86 ops/second - /unlink 1359 (time 1786817729.63 total 10.00 last 135.86) - /unlink 2506 (time 1786817739.63 total 20.00 last 114.70) - /unlink 3788 (time 1786817749.63 total 30.00 last 128.19) total: 4500 /unlink in 34.73 seconds: 129.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 713 (time 1786817769.43 total 10.01 last 71.25) - open/close 1391 (time 1786817779.44 total 20.02 last 67.72) - open/close 2165 (time 1786817789.46 total 30.04 last 77.23) - open/close 2945 (time 1786817799.46 total 40.04 last 78.00) - open/close 3746 (time 1786817809.46 total 50.04 last 80.08) - open/close 4475 (time 1786817819.48 total 60.06 last 72.81) total: 4500 open/close in 60.39 seconds: 74.51 ops/second - /unlink 1457 (time 1786817834.93 total 10.00 last 145.70) - /unlink 2877 (time 1786817844.93 total 20.00 last 142.00) - /unlink 4335 (time 1786817854.93 total 30.01 last 145.71) total: 4500 /unlink in 31.25 seconds: 144.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 798 (time 1786817871.34 total 10.00 last 79.79) - open/close 1536 (time 1786817881.35 total 20.01 last 73.71) - open/close 2218 (time 1786817891.35 total 30.02 last 68.16) - open/close 2966 (time 1786817901.36 total 40.03 last 74.73) - open/close 3769 (time 1786817911.37 total 50.04 last 80.21) total: 4500 open/close in 60.01 seconds: 74.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1580 (time 1786817935.68 total 10.00 last 157.94) - /unlink 2737 (time 1786817945.68 total 20.01 last 115.68) - /unlink 3950 (time 1786817955.69 total 30.01 last 121.19) total: 4500 /unlink in 33.66 seconds: 133.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 691 (time 1786817974.98 total 10.02 last 68.96) - open/close 1442 (time 1786817984.98 total 20.02 last 75.10) - open/close 2198 (time 1786817994.98 total 30.02 last 75.57) - open/close 2915 (time 1786818004.98 total 40.03 last 71.69) - open/close 3578 (time 1786818015.01 total 50.05 last 66.15) - open/close 4281 (time 1786818025.01 total 60.05 last 70.30) total: 4500 open/close in 63.39 seconds: 70.99 ops/second - /unlink 1422 (time 1786818043.96 total 10.01 last 142.04) - /unlink 2817 (time 1786818053.97 total 20.01 last 139.47) - /unlink 4120 (time 1786818063.97 total 30.02 last 130.18) total: 4500 /unlink in 33.09 seconds: 135.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 570 (time 1786818082.85 total 10.00 last 56.99) - open/close 1198 (time 1786818092.86 total 20.01 last 62.74) - open/close 1928 (time 1786818102.86 total 30.01 last 72.99) - open/close 2616 (time 1786818112.87 total 40.02 last 68.77) - open/close 3380 (time 1786818122.88 total 50.03 last 76.33) - open/close 4174 (time 1786818132.89 total 60.03 last 79.34) total: 4500 open/close in 63.76 seconds: 70.58 ops/second - /unlink 1412 (time 1786818151.14 total 10.00 last 141.15) - /unlink 2819 (time 1786818161.15 total 20.01 last 140.62) - /unlink 4269 (time 1786818171.15 total 30.01 last 144.94) total: 4500 /unlink in 31.68 seconds: 142.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 812 (time 1786818187.60 total 10.00 last 81.16) - open/close 1648 (time 1786818197.61 total 20.01 last 83.53) - open/close 2394 (time 1786818207.62 total 30.02 last 74.54) - open/close 3155 (time 1786818217.63 total 40.03 last 76.01) - open/close 3995 (time 1786818227.63 total 50.04 last 83.96) total: 4500 open/close in 56.39 seconds: 79.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1542 (time 1786818248.31 total 10.01 last 154.06) - /unlink 2694 (time 1786818258.32 total 20.02 last 115.10) - /unlink 4132 (time 1786818268.32 total 30.02 last 143.79) total: 4500 /unlink in 32.60 seconds: 138.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 762 (time 1786818285.93 total 10.01 last 76.16) - open/close 1561 (time 1786818295.94 total 20.01 last 79.86) - open/close 2428 (time 1786818305.94 total 30.01 last 86.70) - open/close 3288 (time 1786818315.95 total 40.02 last 85.95) - open/close 4117 (time 1786818325.96 total 50.03 last 82.82) total: 4500 open/close in 55.06 seconds: 81.72 ops/second - /unlink 1407 (time 1786818345.41 total 10.00 last 140.68) - /unlink 2666 (time 1786818355.41 total 20.00 last 125.89) - /unlink 3545 (time 1786818365.42 total 30.01 last 87.83) total: 4500 /unlink in 38.05 seconds: 118.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 801 (time 1786818390.04 total 10.00 last 80.08) - open/close 1558 (time 1786818400.05 total 20.01 last 75.65) - open/close 2270 (time 1786818410.06 total 30.02 last 71.14) - open/close 3131 (time 1786818420.08 total 40.04 last 85.88) - open/close 3937 (time 1786818430.09 total 50.05 last 80.56) total: 4500 open/close in 58.16 seconds: 77.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1514 (time 1786818452.71 total 10.01 last 151.29) - /unlink 2862 (time 1786818462.71 total 20.01 last 134.72) - /unlink 4226 (time 1786818472.72 total 30.01 last 136.39) total: 4500 /unlink in 32.27 seconds: 139.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 803 (time 1786818490.89 total 10.02 last 80.12) - open/close 1496 (time 1786818500.90 total 20.03 last 69.22) - open/close 2223 (time 1786818510.91 total 30.04 last 72.63) - open/close 2726 (time 1786818520.92 total 40.05 last 50.27) - open/close 3307 (time 1786818530.92 total 50.05 last 58.08) - open/close 4107 (time 1786818540.93 total 60.06 last 79.97) total: 4500 open/close in 64.25 seconds: 70.04 ops/second - /unlink 1011 (time 1786818561.04 total 10.00 last 101.05) - /unlink 2229 (time 1786818571.04 total 20.01 last 121.79) - /unlink 3178 (time 1786818581.05 total 30.02 last 94.80) - /unlink 4316 (time 1786818591.05 total 40.02 last 113.72) total: 4500 /unlink in 41.65 seconds: 108.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 608 (time 1786818607.15 total 10.00 last 60.79) - open/close 1255 (time 1786818617.15 total 20.00 last 64.69) - open/close 2012 (time 1786818627.16 total 30.01 last 75.62) - open/close 2765 (time 1786818637.17 total 40.03 last 75.20) - open/close 3582 (time 1786818647.18 total 50.03 last 81.66) - open/close 4350 (time 1786818657.19 total 60.04 last 76.72) total: 4500 open/close in 61.61 seconds: 73.04 ops/second - /unlink 1515 (time 1786818673.99 total 10.00 last 151.46) - /unlink 2799 (time 1786818683.99 total 20.01 last 128.34) - /unlink 4316 (time 1786818693.99 total 30.01 last 151.67) total: 4500 /unlink in 31.38 seconds: 143.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 776 (time 1786818711.00 total 10.02 last 77.46) - open/close 1466 (time 1786818721.01 total 20.03 last 68.93) - open/close 2153 (time 1786818731.02 total 30.04 last 68.62) - open/close 2890 (time 1786818741.02 total 40.04 last 73.68) - open/close 3568 (time 1786818751.03 total 50.05 last 67.76) - open/close 4414 (time 1786818761.04 total 60.06 last 84.53) total: 4500 open/close in 61.08 seconds: 73.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1489 (time 1786818777.14 total 10.00 last 148.88) - /unlink 2739 (time 1786818787.15 total 20.00 last 124.98) - /unlink 4132 (time 1786818797.16 total 30.01 last 139.16) total: 4500 /unlink in 32.49 seconds: 138.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 675 (time 1786818815.09 total 10.00 last 67.49) - open/close 1403 (time 1786818825.09 total 20.01 last 72.76) - open/close 2047 (time 1786818835.10 total 30.01 last 64.35) - open/close 2752 (time 1786818845.10 total 40.02 last 70.48) - open/close 3446 (time 1786818855.11 total 50.03 last 69.33) - open/close 4224 (time 1786818865.12 total 60.03 last 77.74) total: 4500 open/close in 63.47 seconds: 70.90 ops/second - /unlink 1105 (time 1786818883.82 total 10.00 last 110.50) - /unlink 2033 (time 1786818893.84 total 20.02 last 92.61) - /unlink 3097 (time 1786818903.84 total 30.02 last 106.38) - /unlink 4242 (time 1786818913.85 total 40.03 last 114.40) total: 4500 /unlink in 42.50 seconds: 105.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 657 (time 1786818932.89 total 10.01 last 65.63) - open/close 1442 (time 1786818942.89 total 20.01 last 78.49) - open/close 2163 (time 1786818952.89 total 30.02 last 72.06) - open/close 2950 (time 1786818962.90 total 40.02 last 78.68) - open/close 3704 (time 1786818972.90 total 50.02 last 75.38) - open/close 4485 (time 1786818982.93 total 60.05 last 77.89) total: 4500 open/close in 60.29 seconds: 74.63 ops/second - /unlink 1465 (time 1786818996.94 total 10.00 last 146.45) - /unlink 2574 (time 1786819006.94 total 20.00 last 110.89) - /unlink 3742 (time 1786819016.94 total 30.01 last 116.78) total: 4500 /unlink in 36.90 seconds: 121.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 677 (time 1786819039.91 total 10.01 last 67.63) - open/close 1392 (time 1786819049.91 total 20.01 last 71.48) - open/close 2232 (time 1786819059.92 total 30.02 last 83.97) - open/close 2975 (time 1786819069.92 total 40.02 last 74.27) - open/close 3716 (time 1786819079.93 total 50.02 last 74.06) - open/close 4415 (time 1786819089.93 total 60.03 last 69.87) total: 4500 open/close in 60.96 seconds: 73.82 ops/second - /unlink 1418 (time 1786819105.59 total 10.00 last 141.78) - /unlink 2700 (time 1786819115.59 total 20.00 last 128.20) - /unlink 3746 (time 1786819125.60 total 30.00 last 104.58) total: 4500 /unlink in 35.92 seconds: 125.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 710 (time 1786819147.07 total 10.01 last 70.94) - open/close 1391 (time 1786819157.07 total 20.01 last 68.06) - open/close 2114 (time 1786819167.08 total 30.02 last 72.24) - open/close 2708 (time 1786819177.09 total 40.03 last 59.36) - open/close 3321 (time 1786819187.09 total 50.03 last 61.29) - open/close 3933 (time 1786819197.10 total 60.04 last 61.14) total: 4500 open/close in 68.34 seconds: 65.85 ops/second - /unlink 1172 (time 1786819220.43 total 10.00 last 117.16) - /unlink 2619 (time 1786819230.44 total 20.01 last 144.58) - /unlink 4024 (time 1786819240.44 total 30.01 last 140.49) total: 4500 /unlink in 33.93 seconds: 132.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 774 (time 1786819259.82 total 10.01 last 77.36) - open/close 1405 (time 1786819269.82 total 20.01 last 63.09) - open/close 2081 (time 1786819279.83 total 30.01 last 67.56) - open/close 2766 (time 1786819289.83 total 40.02 last 68.46) - open/close 3396 (time 1786819299.85 total 50.04 last 62.89) - open/close 4126 (time 1786819309.85 total 60.04 last 72.96) total: 4500 open/close in 64.80 seconds: 69.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1435 (time 1786819329.42 total 10.01 last 143.34) - /unlink 2670 (time 1786819339.43 total 20.02 last 123.42) - /unlink 3870 (time 1786819349.43 total 30.02 last 119.96) total: 4500 /unlink in 35.10 seconds: 128.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 557 (time 1786819370.84 total 10.01 last 55.65) - open/close 1242 (time 1786819380.85 total 20.02 last 68.46) - open/close 1852 (time 1786819390.85 total 30.02 last 60.98) - open/close 2369 (time 1786819400.86 total 40.02 last 51.69) - open/close 2976 (time 1786819410.86 total 50.03 last 60.67) - open/close 3579 (time 1786819420.87 total 60.03 last 60.27) - open/close 4268 (time 1786819430.87 total 70.04 last 68.88) total: 4500 open/close in 72.98 seconds: 61.66 ops/second - /unlink 1250 (time 1786819449.77 total 10.00 last 124.94) - /unlink 2408 (time 1786819459.77 total 20.01 last 115.76) - /unlink 3554 (time 1786819469.78 total 30.02 last 114.50) total: 4500 /unlink in 37.76 seconds: 119.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 565 (time 1786819494.79 total 10.00 last 56.50) - open/close 1356 (time 1786819504.80 total 20.01 last 79.00) - open/close 2055 (time 1786819514.81 total 30.02 last 69.84) - open/close 2676 (time 1786819524.82 total 40.03 last 62.04) - open/close 3424 (time 1786819534.83 total 50.04 last 74.76) - open/close 4028 (time 1786819544.83 total 60.04 last 60.39) total: 4500 open/close in 67.12 seconds: 67.05 ops/second - /unlink 1401 (time 1786819567.18 total 10.00 last 140.06) - /unlink 2682 (time 1786819577.18 total 20.01 last 128.06) - /unlink 4062 (time 1786819587.19 total 30.01 last 137.88) total: 4500 /unlink in 33.36 seconds: 134.89 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95313 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: modules unloaded.