== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:37 (1710101137) start mds service on oleg111-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg111-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg111-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock oleg111-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 989 (time 1710101185.74 total 10.00 last 98.89) - open/close 1567 (time 1710101202.37 total 26.64 last 34.74) - open/close 2476 (time 1710101212.40 total 36.66 last 90.71) - open/close 3290 (time 1710101222.40 total 46.66 last 81.34) - open/close 3955 (time 1710101232.41 total 56.67 last 66.44) - open/close 4688 (time 1710101242.43 total 66.69 last 73.17) - open/close 5279 (time 1710101252.43 total 76.69 last 59.09) - open/close 5949 (time 1710101262.44 total 86.71 last 66.92) - open/close 6559 (time 1710101272.45 total 96.72 last 60.94) - open/close 6858 (time 1710101289.44 total 113.71 last 17.60) - open/close 7505 (time 1710101299.45 total 123.71 last 64.68) - open/close 8065 (time 1710101309.46 total 133.72 last 55.92) - open/close 8589 (time 1710101319.47 total 143.73 last 52.34) - open/close 9135 (time 1710101329.51 total 153.77 last 54.42) - open/close 9697 (time 1710101339.51 total 163.78 last 56.16) total: 10000 open/close in 169.89 seconds: 58.86 ops/second - unlinked 0 (time 1710101348 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 1004 (time 1710101456.41 total 10.01 last 100.26) - open/close 1949 (time 1710101466.41 total 20.02 last 94.47) - open/close 2734 (time 1710101476.42 total 30.02 last 78.45) - open/close 3437 (time 1710101486.42 total 40.03 last 70.27) - open/close 4185 (time 1710101496.43 total 50.03 last 74.75) - open/close 4810 (time 1710101506.43 total 60.04 last 62.49) - open/close 5489 (time 1710101516.43 total 70.04 last 67.88) - open/close 6245 (time 1710101526.44 total 80.05 last 75.52) - open/close 6871 (time 1710101536.45 total 90.06 last 62.56) - open/close 7483 (time 1710101546.46 total 100.07 last 61.14) - open/close 8068 (time 1710101556.47 total 110.07 last 58.46) - open/close 8649 (time 1710101566.47 total 120.07 last 58.09) - open/close 9215 (time 1710101576.48 total 130.09 last 56.54) - open/close 9750 (time 1710101586.48 total 140.09 last 53.49) total: 10000 open/close in 145.26 seconds: 68.84 ops/second - unlinked 0 (time 1710101594 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 874 (time 1710101711.67 total 19.98 last 43.75) - open/close 1653 (time 1710101721.68 total 29.99 last 77.78) - open/close 2281 (time 1710101740.35 total 48.66 last 33.64) - open/close 3063 (time 1710101750.38 total 58.69 last 77.94) - open/close 3861 (time 1710101760.39 total 68.70 last 79.74) - open/close 4566 (time 1710101770.40 total 78.71 last 70.43) - open/close 4932 (time 1710101786.80 total 95.11 last 22.31) - open/close 5616 (time 1710101796.81 total 105.12 last 68.36) - open/close 6290 (time 1710101806.82 total 115.13 last 67.37) - open/close 6851 (time 1710101816.82 total 125.13 last 56.09) - open/close 7379 (time 1710101826.82 total 135.13 last 52.77) - open/close 7941 (time 1710101836.84 total 145.15 last 56.08) - open/close 8531 (time 1710101846.85 total 155.16 last 58.95) - open/close 9123 (time 1710101856.86 total 165.17 last 59.16) - open/close 9712 (time 1710101866.87 total 175.18 last 58.84) total: 10000 open/close in 180.45 seconds: 55.42 ops/second - unlinked 0 (time 1710101874 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 928 (time 1710101980.18 total 10.01 last 92.72) - open/close 1856 (time 1710101990.19 total 20.02 last 92.70) - open/close 2737 (time 1710102000.20 total 30.03 last 88.03) - open/close 3498 (time 1710102010.20 total 40.03 last 76.07) - open/close 4288 (time 1710102020.21 total 50.04 last 78.90) - open/close 4924 (time 1710102030.21 total 60.05 last 63.58) - open/close 5259 (time 1710102046.54 total 76.37 last 20.52) - open/close 5916 (time 1710102056.55 total 86.38 last 65.68) - open/close 6562 (time 1710102066.55 total 96.38 last 64.59) - open/close 7121 (time 1710102076.55 total 106.38 last 55.89) - open/close 7678 (time 1710102086.55 total 116.38 last 55.68) - open/close 8211 (time 1710102096.56 total 126.39 last 53.24) - open/close 8730 (time 1710102106.57 total 136.40 last 51.87) - open/close 9226 (time 1710102116.58 total 146.41 last 49.54) - open/close 9753 (time 1710102126.60 total 156.44 last 52.57) total: 10000 open/close in 160.78 seconds: 62.20 ops/second - unlinked 0 (time 1710102133 ; total 0 ; last 0) total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second - open/close 745 (time 1710102263.20 total 19.18 last 38.84) - open/close 1585 (time 1710102273.21 total 29.19 last 83.94) - open/close 2416 (time 1710102283.22 total 39.19 last 83.07) - open/close 3182 (time 1710102302.82 total 58.80 last 39.07) - open/close 3966 (time 1710102312.83 total 68.81 last 78.33) - open/close 4717 (time 1710102322.84 total 78.81 last 75.06) - open/close 5368 (time 1710102332.84 total 88.81 last 65.09) - open/close 5961 (time 1710102342.85 total 98.83 last 59.21) - open/close 6023 (time 1710102354.96 total 110.93 last 5.12) - open/close 6673 (time 1710102364.97 total 120.94 last 64.90) - open/close 7345 (time 1710102374.98 total 130.95 last 67.14) - open/close 7904 (time 1710102385.01 total 140.99 last 55.72) - open/close 8376 (time 1710102405.67 total 161.65 last 22.85) - open/close 8888 (time 1710102415.68 total 171.66 last 51.14) - open/close 9475 (time 1710102425.70 total 181.67 last 58.63) total: 10000 open/close in 190.21 seconds: 52.57 ops/second - unlinked 0 (time 1710102436 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 996 (time 1710102546.93 total 10.00 last 99.57) - open/close 1822 (time 1710102556.93 total 20.00 last 82.59) - open/close 2623 (time 1710102566.94 total 30.01 last 80.08) - open/close 3437 (time 1710102576.94 total 40.01 last 81.40) - open/close 4193 (time 1710102586.94 total 50.01 last 75.59) - open/close 4473 (time 1710102602.34 total 65.41 last 18.18) - open/close 5168 (time 1710102612.34 total 75.42 last 69.44) - open/close 5638 (time 1710102630.95 total 94.02 last 25.27) - open/close 6306 (time 1710102640.98 total 104.05 last 66.59) - open/close 6916 (time 1710102650.98 total 114.05 last 60.97) - open/close 7482 (time 1710102661.04 total 124.11 last 56.28) - open/close 7506 (time 1710102672.44 total 135.51 last 2.10) - open/close 8098 (time 1710102682.45 total 145.52 last 59.18) - open/close 8661 (time 1710102692.45 total 155.52 last 56.28) - open/close 9173 (time 1710102702.50 total 165.57 last 50.93) - open/close 9676 (time 1710102712.57 total 175.64 last 49.95) total: 10000 open/close in 182.36 seconds: 54.84 ops/second - unlinked 0 (time 1710102721 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 895 (time 1710102830.13 total 10.00 last 89.49) - open/close 1771 (time 1710102840.13 total 20.00 last 87.57) - open/close 2539 (time 1710102850.14 total 30.01 last 76.78) - open/close 3274 (time 1710102860.15 total 40.02 last 73.39) - open/close 4029 (time 1710102870.16 total 50.03 last 75.44) - open/close 4683 (time 1710102880.16 total 60.03 last 65.38) - open/close 5336 (time 1710102890.17 total 70.04 last 65.28) - open/close 5975 (time 1710102900.18 total 80.05 last 63.80) - open/close 6576 (time 1710102910.19 total 90.06 last 60.05) - open/close 7181 (time 1710102920.19 total 100.06 last 60.50) - open/close 7753 (time 1710102930.20 total 110.07 last 57.16) - open/close 8319 (time 1710102940.22 total 120.09 last 56.48) - open/close 8830 (time 1710102950.22 total 130.09 last 51.07) - open/close 9136 (time 1710102967.04 total 146.90 last 18.20) - open/close 9647 (time 1710102977.04 total 156.91 last 51.06) total: 10000 open/close in 163.92 seconds: 61.00 ops/second - unlinked 0 (time 1710102986 ; total 0 ; last 0) total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second - open/close 1002 (time 1710103117.35 total 10.00 last 100.16) - open/close 1315 (time 1710103131.65 total 24.30 last 21.90) - open/close 2105 (time 1710103141.65 total 34.30 last 78.98) - open/close 2886 (time 1710103151.65 total 44.30 last 78.05) - open/close 3713 (time 1710103161.66 total 54.31 last 82.67) - open/close 4456 (time 1710103171.67 total 64.32 last 74.23) - open/close 5321 (time 1710103181.67 total 74.32 last 86.45) - open/close 6102 (time 1710103191.68 total 84.33 last 78.09) - open/close 6789 (time 1710103201.68 total 94.33 last 68.66) - open/close 7434 (time 1710103211.69 total 104.34 last 64.42) - open/close 7840 (time 1710103228.87 total 121.52 last 23.64) - open/close 8364 (time 1710103238.88 total 131.53 last 52.35) - open/close 8997 (time 1710103248.89 total 141.54 last 63.20) - open/close 9545 (time 1710103258.90 total 151.55 last 54.76) total: 10000 open/close in 158.77 seconds: 62.98 ops/second - unlinked 0 (time 1710103268 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1055 (time 1710103367.08 total 20.09 last 52.51) - open/close 2174 (time 1710103377.08 total 30.09 last 111.89) - open/close 3228 (time 1710103387.08 total 40.09 last 105.38) - open/close 4283 (time 1710103397.09 total 50.10 last 105.46) - open/close 5149 (time 1710103407.09 total 60.10 last 86.57) - open/close 6016 (time 1710103417.09 total 70.10 last 86.68) - open/close 6877 (time 1710103427.09 total 80.11 last 86.10) - open/close 7741 (time 1710103437.10 total 90.11 last 86.33) - open/close 8525 (time 1710103447.10 total 100.11 last 78.40) - open/close 9321 (time 1710103457.11 total 110.12 last 79.57) total: 10000 open/close in 118.86 seconds: 84.13 ops/second - unlinked 0 (time 1710103467 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1737 (time 1710103537.72 total 10.00 last 173.70) - open/close 3185 (time 1710103547.72 total 20.00 last 144.76) - open/close 4610 (time 1710103557.73 total 30.01 last 142.41) - open/close 6009 (time 1710103567.73 total 40.01 last 139.85) - open/close 7332 (time 1710103577.73 total 50.02 last 132.27) - open/close 8644 (time 1710103587.74 total 60.02 last 131.11) - open/close 9937 (time 1710103597.74 total 70.02 last 129.29) total: 10000 open/close in 70.50 seconds: 141.84 ops/second - unlinked 0 (time 1710103599 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.111@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.111@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg111-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg111-server: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg111-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg111-server: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg111-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock oleg111-server@tcp:/lustre /mnt/lustre On OST0, 9167 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg111-server unloading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2