== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:13 (1710101113) start mds service on oleg107-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 244 (time 1710101170.65 total 13.37 last 18.25) - open/close 1012 (time 1710101180.66 total 23.38 last 76.69) - open/close 1792 (time 1710101190.68 total 33.40 last 77.88) - open/close 2527 (time 1710101200.69 total 43.41 last 73.43) - open/close 3179 (time 1710101210.69 total 53.41 last 65.18) - open/close 3877 (time 1710101220.69 total 63.41 last 69.79) - open/close 4452 (time 1710101230.71 total 73.43 last 57.42) - open/close 5011 (time 1710101240.72 total 83.45 last 55.80) - open/close 5503 (time 1710101250.84 total 93.56 last 48.65) - open/close 6038 (time 1710101260.85 total 103.57 last 53.44) - open/close 6511 (time 1710101270.86 total 113.58 last 47.25) - open/close 6991 (time 1710101280.87 total 123.59 last 47.95) - open/close 7547 (time 1710101290.87 total 133.59 last 55.59) - open/close 8004 (time 1710101300.88 total 143.60 last 45.66) - open/close 8457 (time 1710101310.89 total 153.61 last 45.26) - open/close 8867 (time 1710101320.89 total 163.61 last 40.97) - open/close 9238 (time 1710101341.32 total 184.04 last 18.16) - open/close 9629 (time 1710101351.34 total 194.06 last 39.03) total: 10000 open/close in 203.34 seconds: 49.18 ops/second - unlinked 0 (time 1710101363 ; total 0 ; last 0) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 298 (time 1710101501.09 total 14.75 last 20.20) - open/close 909 (time 1710101519.94 total 33.60 last 32.41) - open/close 1683 (time 1710101529.95 total 43.62 last 77.29) - open/close 2356 (time 1710101539.97 total 53.63 last 67.22) - open/close 2547 (time 1710101553.71 total 67.37 last 13.90) - open/close 3252 (time 1710101563.72 total 77.38 last 70.45) - open/close 3864 (time 1710101573.72 total 87.38 last 61.18) - open/close 4440 (time 1710101583.79 total 97.45 last 57.20) - open/close 4971 (time 1710101593.79 total 107.46 last 53.08) - open/close 5529 (time 1710101603.80 total 117.47 last 55.73) - open/close 5623 (time 1710101616.88 total 130.54 last 7.19) - open/close 6109 (time 1710101626.88 total 140.55 last 48.59) - open/close 6609 (time 1710101636.89 total 150.56 last 49.94) - open/close 7081 (time 1710101646.94 total 160.60 last 46.99) - open/close 7517 (time 1710101656.95 total 170.62 last 43.55) - open/close 7971 (time 1710101666.96 total 180.62 last 45.37) - open/close 8411 (time 1710101676.99 total 190.65 last 43.86) - open/close 8893 (time 1710101687.01 total 200.67 last 48.11) - open/close 9047 (time 1710101701.27 total 214.94 last 10.79) - open/close 9539 (time 1710101711.30 total 224.97 last 49.06) - open/close 9940 (time 1710101721.32 total 234.98 last 40.03) total: 10000 open/close in 236.96 seconds: 42.20 ops/second - unlinked 0 (time 1710101726 ; total 0 ; last 0) total: 10000 unlinks in 132 seconds: 75.757576 unlinks/second - open/close 882 (time 1710101871.15 total 10.01 last 88.10) - open/close 1565 (time 1710101881.16 total 20.02 last 68.23) - open/close 2335 (time 1710101891.17 total 30.03 last 76.91) - open/close 3056 (time 1710101901.18 total 40.04 last 72.03) - open/close 3690 (time 1710101911.19 total 50.05 last 63.36) - open/close 4267 (time 1710101921.23 total 60.09 last 57.46) - open/close 4868 (time 1710101931.26 total 70.12 last 59.91) - open/close 5303 (time 1710101950.15 total 89.01 last 23.03) - open/close 5874 (time 1710101960.15 total 99.02 last 57.08) - open/close 6386 (time 1710101970.16 total 109.03 last 51.13) - open/close 6850 (time 1710101980.16 total 119.03 last 46.40) - open/close 7374 (time 1710101990.17 total 129.04 last 52.36) - open/close 7880 (time 1710102000.18 total 139.05 last 50.54) - open/close 8304 (time 1710102010.26 total 149.12 last 42.09) - open/close 8749 (time 1710102020.27 total 159.14 last 44.43) - open/close 8843 (time 1710102033.22 total 172.08 last 7.26) - open/close 9320 (time 1710102043.23 total 182.09 last 47.66) - open/close 9756 (time 1710102053.24 total 192.10 last 43.55) total: 10000 open/close in 198.15 seconds: 50.47 ops/second - unlinked 0 (time 1710102061 ; total 0 ; last 0) total: 10000 unlinks in 122 seconds: 81.967216 unlinks/second - open/close 841 (time 1710102196.66 total 10.01 last 84.06) - open/close 1561 (time 1710102206.68 total 20.03 last 71.84) - open/close 2198 (time 1710102216.69 total 30.03 last 63.67) - open/close 2867 (time 1710102226.69 total 40.04 last 66.89) - open/close 3508 (time 1710102236.69 total 50.04 last 64.09) - open/close 4060 (time 1710102246.69 total 60.04 last 55.19) - open/close 4572 (time 1710102256.69 total 70.04 last 51.20) - open/close 5142 (time 1710102266.71 total 80.06 last 56.89) - open/close 5679 (time 1710102276.72 total 90.07 last 53.66) - open/close 6204 (time 1710102286.73 total 100.08 last 52.42) - open/close 6751 (time 1710102296.76 total 110.11 last 54.56) - open/close 7298 (time 1710102306.79 total 120.13 last 54.55) - open/close 7841 (time 1710102316.81 total 130.15 last 54.19) - open/close 8320 (time 1710102326.83 total 140.18 last 47.77) - open/close 8746 (time 1710102336.86 total 150.20 last 42.50) - open/close 9192 (time 1710102346.86 total 160.21 last 44.60) - open/close 9702 (time 1710102356.87 total 170.22 last 50.95) total: 10000 open/close in 176.20 seconds: 56.75 ops/second - unlinked 0 (time 1710102365 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 817 (time 1710102497.00 total 10.00 last 81.68) - open/close 1442 (time 1710102516.30 total 29.30 last 32.38) - open/close 2102 (time 1710102526.30 total 39.31 last 65.98) - open/close 2214 (time 1710102538.78 total 51.79 last 8.97) - open/close 2835 (time 1710102548.80 total 61.80 last 62.00) - open/close 3457 (time 1710102558.81 total 71.82 last 62.10) - open/close 4092 (time 1710102568.81 total 81.82 last 63.49) - open/close 4691 (time 1710102578.82 total 91.83 last 59.86) - open/close 5278 (time 1710102588.84 total 101.84 last 58.61) - open/close 5783 (time 1710102598.84 total 111.84 last 50.48) - open/close 6312 (time 1710102608.85 total 121.85 last 52.85) - open/close 6774 (time 1710102618.86 total 131.86 last 46.16) - open/close 7208 (time 1710102628.87 total 141.88 last 43.35) - open/close 7709 (time 1710102638.88 total 151.89 last 50.03) - open/close 8162 (time 1710102648.89 total 161.90 last 45.27) - open/close 8593 (time 1710102658.90 total 171.91 last 43.04) - open/close 9015 (time 1710102668.91 total 181.91 last 42.17) - open/close 9397 (time 1710102678.92 total 191.92 last 38.17) - open/close 9824 (time 1710102688.94 total 201.94 last 42.62) total: 10000 open/close in 206.04 seconds: 48.53 ops/second - unlinked 0 (time 1710102696 ; total 1 ; last 1) total: 10000 unlinks in 120 seconds: 83.333336 unlinks/second - open/close 812 (time 1710102828.19 total 10.00 last 81.19) - open/close 1616 (time 1710102838.20 total 20.01 last 80.32) - open/close 2250 (time 1710102848.20 total 30.01 last 63.39) - open/close 2938 (time 1710102858.21 total 40.02 last 68.78) - open/close 3594 (time 1710102868.22 total 50.03 last 65.53) - open/close 4138 (time 1710102878.23 total 60.04 last 54.33) - open/close 4544 (time 1710102897.19 total 79.00 last 21.41) - open/close 5132 (time 1710102907.20 total 89.01 last 58.72) - open/close 5702 (time 1710102917.21 total 99.02 last 56.96) - open/close 6207 (time 1710102927.21 total 109.02 last 50.48) - open/close 6747 (time 1710102937.23 total 119.04 last 53.93) - open/close 7203 (time 1710102947.23 total 129.04 last 45.58) - open/close 7647 (time 1710102957.24 total 139.05 last 44.34) - open/close 8057 (time 1710102967.30 total 149.11 last 40.76) - open/close 8083 (time 1710102978.91 total 160.72 last 2.24) - open/close 8518 (time 1710102988.95 total 170.76 last 43.34) - open/close 8914 (time 1710102998.98 total 180.80 last 39.46) - open/close 9348 (time 1710103008.99 total 190.81 last 43.36) - open/close 9741 (time 1710103019.01 total 200.82 last 39.26) total: 10000 open/close in 206.52 seconds: 48.42 ops/second - unlinked 0 (time 1710103027 ; total 0 ; last 0) total: 10000 unlinks in 138 seconds: 72.463768 unlinks/second - open/close 936 (time 1710103177.80 total 10.00 last 93.59) - open/close 959 (time 1710103189.06 total 21.27 last 2.04) - open/close 1781 (time 1710103199.07 total 31.28 last 82.10) - open/close 2550 (time 1710103209.09 total 41.29 last 76.80) - open/close 2765 (time 1710103223.17 total 55.38 last 15.26) - open/close 3437 (time 1710103233.18 total 65.39 last 67.14) - open/close 4071 (time 1710103243.19 total 75.39 last 63.35) - open/close 4792 (time 1710103253.19 total 85.40 last 72.09) - open/close 5404 (time 1710103263.19 total 95.40 last 61.19) - open/close 6074 (time 1710103273.20 total 105.40 last 66.96) - open/close 6416 (time 1710103289.32 total 121.53 last 21.21) - open/close 6993 (time 1710103299.33 total 131.53 last 57.70) - open/close 7592 (time 1710103309.33 total 141.54 last 59.86) - open/close 8228 (time 1710103319.33 total 151.54 last 63.60) - open/close 8814 (time 1710103329.34 total 161.55 last 58.53) - open/close 9383 (time 1710103339.36 total 171.57 last 56.78) - open/close 9947 (time 1710103349.39 total 181.59 last 56.27) total: 10000 open/close in 182.45 seconds: 54.81 ops/second - unlinked 0 (time 1710103352 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 1086 (time 1710103463.93 total 10.00 last 108.59) - open/close 2089 (time 1710103473.94 total 20.01 last 100.24) - open/close 3050 (time 1710103483.94 total 30.01 last 96.08) - open/close 4028 (time 1710103493.94 total 40.01 last 97.80) - open/close 5037 (time 1710103503.94 total 50.01 last 100.86) - open/close 5979 (time 1710103513.95 total 60.02 last 94.14) - open/close 6935 (time 1710103523.96 total 70.03 last 95.51) - open/close 7887 (time 1710103533.96 total 80.03 last 95.16) - open/close 8803 (time 1710103543.97 total 90.04 last 91.53) - open/close 9673 (time 1710103553.97 total 100.04 last 86.99) total: 10000 open/close in 103.84 seconds: 96.31 ops/second - unlinked 0 (time 1710103559 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 2136 (time 1710103621.79 total 10.00 last 213.53) - open/close 3928 (time 1710103631.80 total 20.01 last 179.16) - open/close 5499 (time 1710103641.80 total 30.01 last 157.06) - open/close 6984 (time 1710103651.80 total 40.01 last 148.47) - open/close 8354 (time 1710103661.80 total 50.01 last 136.98) - open/close 9740 (time 1710103671.81 total 60.01 last 138.55) total: 10000 open/close in 62.05 seconds: 161.15 ops/second - unlinked 0 (time 1710103674 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 3716 (time 1710103719.75 total 10.00 last 371.48) - open/close 6399 (time 1710103729.75 total 20.00 last 268.26) - open/close 8829 (time 1710103739.75 total 30.01 last 242.91) total: 10000 open/close in 34.81 seconds: 287.26 ops/second - unlinked 0 (time 1710103745 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg107-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec oleg107-server: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg107-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock oleg107-server@tcp:/lustre /mnt/lustre On OST0, 17847 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server unloading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2