== conf-sanity test 69: replace an OST with the same index ========================================================== 16:05:18 (1710101118) start mds service on oleg154-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg154-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg154-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 919 (time 1710101174.33 total 10.00 last 91.86) - open/close 1734 (time 1710101184.35 total 20.02 last 81.37) - open/close 2548 (time 1710101194.35 total 30.02 last 81.39) - open/close 3217 (time 1710101204.36 total 40.03 last 66.84) - open/close 3837 (time 1710101214.37 total 50.04 last 61.94) - open/close 4476 (time 1710101224.37 total 60.04 last 63.89) - open/close 5030 (time 1710101234.37 total 70.05 last 55.37) - open/close 5609 (time 1710101244.38 total 80.05 last 57.89) - open/close 5693 (time 1710101256.93 total 92.61 last 6.69) - open/close 6251 (time 1710101266.95 total 102.62 last 55.73) - open/close 6736 (time 1710101276.97 total 112.64 last 48.39) - open/close 7293 (time 1710101286.97 total 122.65 last 55.67) - open/close 7802 (time 1710101296.98 total 132.65 last 50.89) - open/close 8276 (time 1710101306.98 total 142.65 last 47.39) - open/close 8530 (time 1710101323.60 total 159.27 last 15.28) - open/close 8682 (time 1710101337.79 total 173.47 last 10.71) - open/close 9087 (time 1710101347.80 total 183.47 last 40.49) - open/close 9538 (time 1710101357.80 total 193.48 last 45.07) - open/close 9993 (time 1710101367.81 total 203.48 last 45.48) total: 10000 open/close in 203.70 seconds: 49.09 ops/second - unlinked 0 (time 1710101370 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 838 (time 1710101496.39 total 10.02 last 83.63) - open/close 1623 (time 1710101506.39 total 20.02 last 78.48) - open/close 2346 (time 1710101516.40 total 30.03 last 72.25) - open/close 3102 (time 1710101526.40 total 40.03 last 75.59) - open/close 3763 (time 1710101536.40 total 50.04 last 66.07) - open/close 4352 (time 1710101546.42 total 60.06 last 58.78) - open/close 4947 (time 1710101556.43 total 70.06 last 59.48) - open/close 5527 (time 1710101566.43 total 80.07 last 57.96) - open/close 5616 (time 1710101578.84 total 92.48 last 7.17) - open/close 6152 (time 1710101588.86 total 102.49 last 53.52) - open/close 6666 (time 1710101598.87 total 112.50 last 51.33) - open/close 7182 (time 1710101608.89 total 122.52 last 51.49) - open/close 7678 (time 1710101618.90 total 132.53 last 49.56) - open/close 8141 (time 1710101628.91 total 142.54 last 46.26) - open/close 8316 (time 1710101643.31 total 156.94 last 12.16) - open/close 8731 (time 1710101653.31 total 166.94 last 41.49) - open/close 9168 (time 1710101663.37 total 177.00 last 43.44) - open/close 9620 (time 1710101673.37 total 187.01 last 45.18) total: 10000 open/close in 195.26 seconds: 51.21 ops/second - unlinked 0 (time 1710101683 ; total 0 ; last 0) total: 10000 unlinks in 127 seconds: 78.740158 unlinks/second - open/close 763 (time 1710101823.06 total 10.00 last 76.30) - open/close 1531 (time 1710101833.07 total 20.01 last 76.71) - open/close 2270 (time 1710101843.08 total 30.02 last 73.82) - open/close 3002 (time 1710101853.08 total 40.03 last 73.18) - open/close 3662 (time 1710101863.10 total 50.04 last 65.91) - open/close 4277 (time 1710101873.14 total 60.08 last 61.23) - open/close 4819 (time 1710101883.15 total 70.09 last 54.17) - open/close 5453 (time 1710101893.17 total 80.11 last 63.28) - open/close 5838 (time 1710101911.14 total 98.09 last 21.42) - open/close 6332 (time 1710101921.16 total 108.10 last 49.32) - open/close 6844 (time 1710101931.18 total 118.12 last 51.12) - open/close 7355 (time 1710101941.18 total 128.12 last 51.07) - open/close 7866 (time 1710101951.19 total 138.13 last 51.06) - open/close 8377 (time 1710101961.20 total 148.14 last 51.03) - open/close 8801 (time 1710101971.22 total 158.16 last 42.34) - open/close 9241 (time 1710101981.23 total 168.17 last 43.94) - open/close 9714 (time 1710101991.24 total 178.19 last 47.24) total: 10000 open/close in 184.60 seconds: 54.17 ops/second - unlinked 0 (time 1710102000 ; total 0 ; last 0) total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second - open/close 833 (time 1710102136.75 total 10.01 last 83.18) - open/close 1562 (time 1710102146.76 total 20.03 last 72.81) - open/close 2296 (time 1710102156.76 total 30.03 last 73.39) - open/close 2932 (time 1710102166.77 total 40.03 last 63.58) - open/close 3614 (time 1710102176.78 total 50.05 last 68.10) - open/close 4267 (time 1710102186.79 total 60.06 last 65.24) - open/close 4936 (time 1710102196.80 total 70.06 last 66.87) - open/close 5511 (time 1710102206.80 total 80.06 last 57.48) - open/close 6055 (time 1710102216.80 total 90.06 last 54.39) - open/close 6594 (time 1710102226.80 total 100.07 last 53.88) - open/close 7095 (time 1710102236.81 total 110.08 last 50.05) - open/close 7502 (time 1710102246.84 total 120.10 last 40.61) - open/close 7791 (time 1710102264.26 total 137.52 last 16.59) - open/close 8250 (time 1710102274.27 total 147.53 last 45.86) - open/close 8612 (time 1710102292.91 total 166.17 last 19.42) - open/close 9059 (time 1710102302.93 total 176.20 last 44.59) - open/close 9534 (time 1710102312.94 total 186.21 last 47.44) - open/close 9980 (time 1710102322.95 total 196.22 last 44.56) total: 10000 open/close in 196.77 seconds: 50.82 ops/second - unlinked 0 (time 1710102326 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 761 (time 1710102453.09 total 10.00 last 76.10) - open/close 1540 (time 1710102463.11 total 20.02 last 77.77) - open/close 2288 (time 1710102473.11 total 30.02 last 74.80) - open/close 2899 (time 1710102483.11 total 40.02 last 61.06) - open/close 2991 (time 1710102495.36 total 52.27 last 7.51) - open/close 3626 (time 1710102505.36 total 62.27 last 63.47) - open/close 4216 (time 1710102515.37 total 72.28 last 58.93) - open/close 4808 (time 1710102525.38 total 82.29 last 59.16) - open/close 5418 (time 1710102535.39 total 92.30 last 60.95) - open/close 5933 (time 1710102545.41 total 102.32 last 51.39) - open/close 6462 (time 1710102555.43 total 112.34 last 52.78) - open/close 7010 (time 1710102565.44 total 122.34 last 54.78) - open/close 7489 (time 1710102575.44 total 132.35 last 47.86) - open/close 8038 (time 1710102585.45 total 142.36 last 54.90) - open/close 8481 (time 1710102595.45 total 152.36 last 44.29) - open/close 8940 (time 1710102605.46 total 162.37 last 45.82) - open/close 8991 (time 1710102617.53 total 174.44 last 4.23) - open/close 9390 (time 1710102627.55 total 184.46 last 39.80) - open/close 9846 (time 1710102637.56 total 194.47 last 45.58) total: 10000 open/close in 197.83 seconds: 50.55 ops/second - unlinked 0 (time 1710102643 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 821 (time 1710102771.05 total 10.00 last 82.09) - open/close 1580 (time 1710102781.05 total 20.01 last 75.84) - open/close 2343 (time 1710102791.06 total 30.02 last 76.24) - open/close 3089 (time 1710102801.08 total 40.03 last 74.50) - open/close 3695 (time 1710102811.08 total 50.03 last 60.59) - open/close 4369 (time 1710102821.08 total 60.04 last 67.34) - open/close 5010 (time 1710102831.13 total 70.09 last 63.80) - open/close 5557 (time 1710102841.15 total 80.10 last 54.62) - open/close 6071 (time 1710102851.15 total 90.11 last 51.37) - open/close 6599 (time 1710102861.16 total 100.12 last 52.74) - open/close 7099 (time 1710102871.18 total 110.13 last 49.95) - open/close 7565 (time 1710102881.20 total 120.16 last 46.47) - open/close 8055 (time 1710102891.21 total 130.17 last 48.96) - open/close 8548 (time 1710102901.24 total 140.19 last 49.18) - open/close 8987 (time 1710102911.24 total 150.19 last 43.89) - open/close 9435 (time 1710102921.25 total 160.20 last 44.76) - open/close 9886 (time 1710102931.26 total 170.22 last 45.03) total: 10000 open/close in 172.76 seconds: 57.88 ops/second - unlinked 0 (time 1710102936 ; total 0 ; last 0) total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second - open/close 909 (time 1710103065.29 total 10.02 last 90.68) - open/close 1703 (time 1710103075.36 total 20.09 last 78.85) - open/close 2437 (time 1710103085.37 total 30.10 last 73.34) - open/close 3176 (time 1710103095.38 total 40.11 last 73.81) - open/close 3883 (time 1710103105.38 total 50.12 last 70.68) - open/close 4550 (time 1710103115.39 total 60.12 last 66.68) - open/close 5218 (time 1710103125.40 total 70.14 last 66.70) - open/close 5525 (time 1710103141.13 total 85.86 last 19.52) - open/close 6082 (time 1710103151.14 total 95.87 last 55.64) - open/close 6683 (time 1710103161.17 total 105.90 last 59.90) - open/close 7199 (time 1710103171.18 total 115.91 last 51.56) - open/close 7784 (time 1710103181.18 total 125.92 last 58.48) - open/close 8364 (time 1710103191.19 total 135.92 last 57.96) - open/close 8939 (time 1710103201.21 total 145.94 last 57.39) - open/close 9416 (time 1710103211.22 total 155.95 last 47.64) - open/close 9922 (time 1710103221.23 total 165.96 last 50.58) total: 10000 open/close in 167.53 seconds: 59.69 ops/second - unlinked 0 (time 1710103225 ; total 0 ; last 0) total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second - open/close 1039 (time 1710103335.68 total 10.00 last 103.87) - open/close 1966 (time 1710103345.68 total 20.00 last 92.68) - open/close 2821 (time 1710103355.69 total 30.01 last 85.43) - open/close 3694 (time 1710103365.70 total 40.02 last 87.26) - open/close 4462 (time 1710103375.71 total 50.03 last 76.73) - open/close 5230 (time 1710103385.72 total 60.04 last 76.68) - open/close 6007 (time 1710103395.73 total 70.05 last 77.66) - open/close 6658 (time 1710103405.74 total 80.06 last 65.03) - open/close 7322 (time 1710103415.74 total 90.06 last 66.40) - open/close 7991 (time 1710103425.74 total 100.06 last 66.87) - open/close 8638 (time 1710103435.75 total 110.08 last 64.62) - open/close 9229 (time 1710103445.76 total 120.08 last 59.09) - open/close 9810 (time 1710103455.77 total 130.09 last 58.02) total: 10000 open/close in 133.64 seconds: 74.83 ops/second - unlinked 0 (time 1710103461 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1345 (time 1710103547.59 total 10.00 last 134.45) - open/close 2647 (time 1710103557.60 total 20.01 last 130.16) - open/close 3912 (time 1710103567.60 total 30.01 last 126.41) - open/close 5141 (time 1710103577.60 total 40.01 last 122.90) - open/close 6338 (time 1710103587.61 total 50.02 last 119.61) - open/close 7484 (time 1710103597.61 total 60.02 last 114.60) - open/close 8563 (time 1710103607.61 total 70.02 last 107.86) - open/close 9695 (time 1710103617.62 total 80.03 last 113.11) total: 10000 open/close in 82.80 seconds: 120.77 ops/second - unlinked 0 (time 1710103621 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 2871 (time 1710103677.07 total 10.00 last 287.08) - open/close 5295 (time 1710103687.07 total 20.00 last 242.39) - open/close 7267 (time 1710103697.07 total 30.00 last 197.16) - open/close 8994 (time 1710103707.08 total 40.01 last 172.63) total: 10000 open/close in 46.06 seconds: 217.09 ops/second - unlinked 0 (time 1710103713 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg154-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.154@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.154@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg154-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg154-server: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg154-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg154-server: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg154-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg154-client.virtnet: -o user_xattr,flock oleg154-server@tcp:/lustre /mnt/lustre On OST0, 9151 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg154-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg154-server unloading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2