== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:58:14 (1779994694) Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg418-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg418-server oleg418-server: oleg418-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg418-server: detected 4 online CPUs by sysfs oleg418-server: MODOPTS_LIBCFS= oleg418-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg418-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg418-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 663 (time 1779994889.36 total 10.01 last 66.23) - open/close 1284 (time 1779994899.36 total 20.01 last 62.09) - open/close 1913 (time 1779994909.36 total 30.01 last 62.88) - open/close 2540 (time 1779994919.37 total 40.03 last 62.63) - open/close 3162 (time 1779994929.38 total 50.04 last 62.13) - open/close 3776 (time 1779994939.39 total 60.04 last 61.36) - open/close 4388 (time 1779994949.42 total 70.07 last 61.02) total: 4500 open/close in 72.02 seconds: 62.49 ops/second - /unlink 1169 (time 1779994965.99 total 10.00 last 116.85) - /unlink 2307 (time 1779994975.99 total 20.01 last 113.79) - /unlink 3370 (time 1779994985.99 total 30.01 last 106.22) total: 4500 /unlink in 39.98 seconds: 112.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 601 (time 1779995009.89 total 10.04 last 59.85) - open/close 1161 (time 1779995019.91 total 20.05 last 55.92) - open/close 1724 (time 1779995029.92 total 30.07 last 56.21) - open/close 2356 (time 1779995039.95 total 40.09 last 63.06) - open/close 2931 (time 1779995049.95 total 50.10 last 57.46) - open/close 3528 (time 1779995059.96 total 60.11 last 59.64) - open/close 4179 (time 1779995069.97 total 70.11 last 65.07) total: 4500 open/close in 75.26 seconds: 59.80 ops/second - /unlink 1117 (time 1779995089.85 total 10.00 last 111.70) - /unlink 2218 (time 1779995099.85 total 20.00 last 110.07) - /unlink 3392 (time 1779995109.87 total 30.02 last 117.18) total: 4500 /unlink in 39.56 seconds: 113.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 637 (time 1779995135.15 total 10.01 last 63.65) - open/close 1279 (time 1779995145.16 total 20.02 last 64.15) - open/close 1935 (time 1779995155.17 total 30.02 last 65.54) - open/close 2574 (time 1779995165.19 total 40.05 last 63.74) - open/close 3150 (time 1779995175.24 total 50.10 last 57.32) - open/close 3791 (time 1779995185.25 total 60.11 last 64.06) - open/close 4397 (time 1779995195.25 total 70.11 last 60.57) total: 4500 open/close in 71.47 seconds: 62.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1111 (time 1779995210.47 total 10.00 last 111.10) - /unlink 1998 (time 1779995220.48 total 20.01 last 88.61) - /unlink 3190 (time 1779995230.48 total 30.01 last 119.17) - /unlink 4376 (time 1779995240.48 total 40.02 last 118.56) total: 4500 /unlink in 41.11 seconds: 109.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 581 (time 1779995257.09 total 10.02 last 57.97) - open/close 1180 (time 1779995267.10 total 20.04 last 59.81) - open/close 1684 (time 1779995277.12 total 30.06 last 50.30) - open/close 2168 (time 1779995287.14 total 40.07 last 48.32) - open/close 2706 (time 1779995297.15 total 50.09 last 53.73) - open/close 3366 (time 1779995307.15 total 60.09 last 65.99) - open/close 3973 (time 1779995317.15 total 70.09 last 60.69) total: 4500 open/close in 79.21 seconds: 56.81 ops/second - /unlink 1119 (time 1779995341.08 total 10.01 last 111.82) - /unlink 2239 (time 1779995351.08 total 20.01 last 112.00) - /unlink 3367 (time 1779995361.09 total 30.01 last 112.75) - /unlink 4485 (time 1779995371.10 total 40.02 last 111.70) total: 4500 /unlink in 40.21 seconds: 111.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 539 (time 1779995386.45 total 10.00 last 53.88) - open/close 1215 (time 1779995396.47 total 20.02 last 67.47) - open/close 1865 (time 1779995406.47 total 30.02 last 65.00) - open/close 2495 (time 1779995416.47 total 40.02 last 63.00) - open/close 3122 (time 1779995426.47 total 50.03 last 62.68) - open/close 3798 (time 1779995436.48 total 60.03 last 67.57) - open/close 4427 (time 1779995446.49 total 70.04 last 62.84) total: 4500 open/close in 71.34 seconds: 63.08 ops/second - /unlink 1194 (time 1779995462.39 total 10.01 last 119.28) - /unlink 2394 (time 1779995472.39 total 20.02 last 119.93) - /unlink 3424 (time 1779995482.40 total 30.02 last 102.97) - /unlink 4470 (time 1779995492.40 total 40.02 last 104.60) total: 4500 /unlink in 40.43 seconds: 111.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 523 (time 1779995508.15 total 10.02 last 52.21) - open/close 1015 (time 1779995518.16 total 20.03 last 49.14) - open/close 1571 (time 1779995528.17 total 30.04 last 55.53) - open/close 2194 (time 1779995538.18 total 40.05 last 62.24) - open/close 2879 (time 1779995548.19 total 50.06 last 68.46) - open/close 3526 (time 1779995558.19 total 60.06 last 64.66) - open/close 4207 (time 1779995568.20 total 70.06 last 68.10) total: 4500 open/close in 74.96 seconds: 60.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1215 (time 1779995587.56 total 10.00 last 121.46) - /unlink 2304 (time 1779995597.58 total 20.02 last 108.73) - /unlink 3492 (time 1779995607.58 total 30.02 last 118.77) - /unlink 4489 (time 1779995617.60 total 40.04 last 99.48) total: 4500 /unlink in 40.18 seconds: 111.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 397 (time 1779995634.98 total 10.01 last 39.66) - open/close 902 (time 1779995644.98 total 20.01 last 50.50) - open/close 1499 (time 1779995655.00 total 30.03 last 59.59) - open/close 2154 (time 1779995665.00 total 40.03 last 65.48) - open/close 2759 (time 1779995675.01 total 50.04 last 60.47) - open/close 3401 (time 1779995685.01 total 60.04 last 64.16) - open/close 3946 (time 1779995695.02 total 70.05 last 54.48) total: 4500 open/close in 80.02 seconds: 56.24 ops/second - /unlink 1037 (time 1779995719.73 total 10.02 last 103.53) - /unlink 2141 (time 1779995729.74 total 20.02 last 110.36) - /unlink 3354 (time 1779995739.79 total 30.07 last 120.66) total: 4500 /unlink in 40.06 seconds: 112.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 595 (time 1779995765.45 total 10.01 last 59.44) - open/close 1209 (time 1779995775.47 total 20.03 last 61.27) - open/close 1775 (time 1779995785.47 total 30.03 last 56.58) - open/close 2308 (time 1779995795.48 total 40.04 last 53.25) - open/close 2892 (time 1779995805.50 total 50.06 last 58.31) - open/close 3469 (time 1779995815.50 total 60.06 last 57.68) - open/close 4087 (time 1779995825.50 total 70.06 last 61.79) total: 4500 open/close in 76.65 seconds: 58.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1160 (time 1779995846.81 total 10.00 last 115.98) - /unlink 2349 (time 1779995856.81 total 20.00 last 118.89) - /unlink 3549 (time 1779995866.81 total 30.00 last 119.98) total: 4500 /unlink in 38.38 seconds: 117.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 663 (time 1779995890.43 total 10.03 last 66.10) - open/close 1182 (time 1779995900.45 total 20.05 last 51.79) - open/close 1775 (time 1779995910.45 total 30.06 last 59.27) - open/close 2441 (time 1779995920.47 total 40.08 last 66.46) - open/close 3027 (time 1779995930.48 total 50.08 last 58.58) - open/close 3680 (time 1779995940.48 total 60.09 last 65.25) - open/close 4259 (time 1779995950.49 total 70.10 last 57.85) total: 4500 open/close in 74.84 seconds: 60.13 ops/second - /unlink 961 (time 1779995970.71 total 10.01 last 96.01) - /unlink 1858 (time 1779995980.72 total 20.02 last 89.64) - /unlink 2760 (time 1779995990.73 total 30.03 last 90.10) - /unlink 3868 (time 1779996000.73 total 40.03 last 110.74) total: 4500 /unlink in 45.49 seconds: 98.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 567 (time 1779996020.83 total 10.00 last 56.70) - open/close 1203 (time 1779996030.84 total 20.01 last 63.52) - open/close 1754 (time 1779996040.84 total 30.01 last 55.09) - open/close 2331 (time 1779996050.86 total 40.03 last 57.61) - open/close 2934 (time 1779996060.86 total 50.03 last 60.30) - open/close 3584 (time 1779996070.86 total 60.03 last 64.98) - open/close 4219 (time 1779996080.88 total 70.05 last 63.40) total: 4500 open/close in 74.83 seconds: 60.14 ops/second - /unlink 1079 (time 1779996100.44 total 10.00 last 107.85) - /unlink 2238 (time 1779996110.45 total 20.01 last 115.81) - /unlink 3405 (time 1779996120.46 total 30.02 last 116.59) - /unlink 4494 (time 1779996130.47 total 40.04 last 108.73) total: 4500 /unlink in 40.11 seconds: 112.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 684 (time 1779996144.54 total 10.01 last 68.32) - open/close 1377 (time 1779996154.55 total 20.02 last 69.23) - open/close 1984 (time 1779996164.57 total 30.04 last 60.61) - open/close 2527 (time 1779996174.57 total 40.04 last 54.28) - open/close 2980 (time 1779996184.58 total 50.05 last 45.26) - open/close 3492 (time 1779996194.58 total 60.05 last 51.19) - open/close 3975 (time 1779996204.59 total 70.06 last 48.28) total: 4500 open/close in 78.77 seconds: 57.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 950 (time 1779996229.48 total 10.03 last 94.74) - /unlink 1828 (time 1779996239.48 total 20.03 last 87.79) - /unlink 2739 (time 1779996249.48 total 30.03 last 91.06) - /unlink 3593 (time 1779996259.49 total 40.04 last 85.36) total: 4500 /unlink in 48.18 seconds: 93.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 590 (time 1779996284.31 total 10.00 last 58.99) - open/close 1173 (time 1779996294.32 total 20.01 last 58.24) - open/close 1844 (time 1779996304.34 total 30.04 last 66.94) - open/close 2476 (time 1779996314.37 total 40.06 last 63.02) - open/close 3042 (time 1779996324.37 total 50.07 last 56.59) - open/close 3614 (time 1779996334.37 total 60.07 last 57.20) - open/close 4169 (time 1779996344.38 total 70.08 last 55.44) total: 4500 open/close in 75.61 seconds: 59.52 ops/second - /unlink 1062 (time 1779996365.65 total 10.00 last 106.17) - /unlink 2016 (time 1779996375.65 total 20.01 last 95.35) - /unlink 2935 (time 1779996385.66 total 30.02 last 91.84) - /unlink 3970 (time 1779996395.67 total 40.02 last 103.44) total: 4500 /unlink in 45.04 seconds: 99.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 455 (time 1779996416.94 total 10.04 last 45.32) - open/close 855 (time 1779996426.95 total 20.04 last 39.98) - open/close 1263 (time 1779996436.99 total 30.08 last 40.63) - open/close 1772 (time 1779996447.00 total 40.10 last 50.82) - open/close 2229 (time 1779996457.00 total 50.10 last 45.70) - open/close 2733 (time 1779996467.00 total 60.10 last 50.39) - open/close 3136 (time 1779996477.02 total 70.11 last 40.25) - open/close 3645 (time 1779996487.04 total 80.13 last 50.80) - open/close 4165 (time 1779996497.04 total 90.14 last 51.97) total: 4500 open/close in 95.92 seconds: 46.91 ops/second - /unlink 1033 (time 1779996517.88 total 10.01 last 103.24) - /unlink 2038 (time 1779996527.89 total 20.01 last 100.42) - /unlink 2936 (time 1779996537.90 total 30.02 last 89.76) - /unlink 3868 (time 1779996547.92 total 40.04 last 93.01) total: 4500 /unlink in 45.90 seconds: 98.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 605 (time 1779996570.07 total 10.01 last 60.43) - open/close 1160 (time 1779996580.08 total 20.02 last 55.47) - open/close 1760 (time 1779996590.09 total 30.03 last 59.94) - open/close 2360 (time 1779996600.09 total 40.03 last 59.97) - open/close 2869 (time 1779996610.11 total 50.05 last 50.80) - open/close 3351 (time 1779996620.12 total 60.05 last 48.19) - open/close 3935 (time 1779996630.13 total 70.07 last 58.31) total: 4500 open/close in 79.02 seconds: 56.95 ops/second - /unlink 1200 (time 1779996653.29 total 10.01 last 119.89) - /unlink 2333 (time 1779996663.30 total 20.02 last 113.21) - /unlink 3482 (time 1779996673.30 total 30.02 last 114.89) total: 4500 /unlink in 39.28 seconds: 114.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 582 (time 1779996697.78 total 10.01 last 58.17) - open/close 1183 (time 1779996707.80 total 20.03 last 59.97) - open/close 1848 (time 1779996717.81 total 30.04 last 66.44) - open/close 2455 (time 1779996727.81 total 40.04 last 60.68) - open/close 3020 (time 1779996737.81 total 50.04 last 56.49) - open/close 3533 (time 1779996747.82 total 60.04 last 51.28) - open/close 4095 (time 1779996757.82 total 70.05 last 56.16) total: 4500 open/close in 76.68 seconds: 58.68 ops/second - /unlink 1119 (time 1779996779.26 total 10.00 last 111.89) - /unlink 2269 (time 1779996789.26 total 20.00 last 114.99) - /unlink 3406 (time 1779996799.27 total 30.02 last 113.52) - /unlink 4461 (time 1779996809.27 total 40.02 last 105.50) total: 4500 /unlink in 40.60 seconds: 110.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 522 (time 1779996825.03 total 10.02 last 52.08) - open/close 973 (time 1779996835.06 total 20.05 last 44.99) - open/close 1371 (time 1779996845.09 total 30.08 last 39.66) - open/close 1968 (time 1779996855.09 total 40.09 last 59.67) - open/close 2487 (time 1779996865.11 total 50.11 last 51.80) - open/close 3100 (time 1779996875.13 total 60.12 last 61.23) - open/close 3736 (time 1779996885.14 total 70.13 last 63.51) - open/close 4362 (time 1779996895.16 total 80.15 last 62.50) total: 4500 open/close in 82.40 seconds: 54.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1145 (time 1779996912.00 total 10.00 last 114.48) - /unlink 2140 (time 1779996922.02 total 20.02 last 99.33) - /unlink 3103 (time 1779996932.03 total 30.03 last 96.21) - /unlink 4025 (time 1779996942.04 total 40.04 last 92.05) total: 4500 /unlink in 44.75 seconds: 100.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 538 (time 1779996962.24 total 10.02 last 53.68) - open/close 1078 (time 1779996972.24 total 20.03 last 53.97) - open/close 1653 (time 1779996982.25 total 30.04 last 57.45) - open/close 2139 (time 1779996992.27 total 40.05 last 48.52) - open/close 2748 (time 1779997002.28 total 50.06 last 60.83) - open/close 3284 (time 1779997012.29 total 60.08 last 53.54) - open/close 3893 (time 1779997022.30 total 70.09 last 60.83) - open/close 4416 (time 1779997032.31 total 80.09 last 52.26) total: 4500 open/close in 81.90 seconds: 54.95 ops/second - /unlink 1075 (time 1779997050.09 total 10.00 last 107.49) - /unlink 2173 (time 1779997060.11 total 20.01 last 109.66) - /unlink 3207 (time 1779997070.11 total 30.01 last 103.39) - /unlink 4296 (time 1779997080.11 total 40.02 last 108.87) total: 4500 /unlink in 42.24 seconds: 106.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 512 (time 1779997097.90 total 10.01 last 51.16) - open/close 1122 (time 1779997107.91 total 20.02 last 60.94) - open/close 1748 (time 1779997117.91 total 30.02 last 62.60) - open/close 2347 (time 1779997127.93 total 40.03 last 59.84) - open/close 3009 (time 1779997137.93 total 50.04 last 66.16) - open/close 3631 (time 1779997147.93 total 60.04 last 62.20) - open/close 4171 (time 1779997157.94 total 70.04 last 53.96) total: 4500 open/close in 75.04 seconds: 59.97 ops/second - /unlink 1135 (time 1779997178.00 total 10.01 last 113.43) - /unlink 2014 (time 1779997188.00 total 20.01 last 87.89) - /unlink 2855 (time 1779997198.00 total 30.01 last 84.06) - /unlink 3710 (time 1779997208.01 total 40.02 last 85.46) total: 4500 /unlink in 47.15 seconds: 95.44 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94132 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules via unload_modules_local on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local oleg418-server: modules unloaded.