== conf-sanity test 69: replace an OST with the same index ========================================================== 16:01:45 (1710100905) start mds service on oleg127-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg127-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1022 (time 1710100936.67 total 10.04 last 101.77) - open/close 2050 (time 1710100946.67 total 20.05 last 102.77) - open/close 3092 (time 1710100956.68 total 30.05 last 104.18) - open/close 3894 (time 1710100966.68 total 40.06 last 80.13) - open/close 4744 (time 1710100976.69 total 50.06 last 84.93) - open/close 5446 (time 1710100986.70 total 60.08 last 70.12) - open/close 6129 (time 1710100996.71 total 70.08 last 68.23) - open/close 6776 (time 1710101006.72 total 80.09 last 64.67) - open/close 7496 (time 1710101016.73 total 90.10 last 71.93) - open/close 8111 (time 1710101026.73 total 100.10 last 61.49) - open/close 8711 (time 1710101036.74 total 110.11 last 59.95) - open/close 9345 (time 1710101046.75 total 120.12 last 63.34) - open/close 9910 (time 1710101056.75 total 130.12 last 56.50) total: 10000 open/close in 131.39 seconds: 76.11 ops/second - unlinked 0 (time 1710101059 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 929 (time 1710101163.20 total 10.01 last 92.85) - open/close 1793 (time 1710101184.02 total 30.83 last 41.48) - open/close 2580 (time 1710101194.03 total 40.84 last 78.66) - open/close 3208 (time 1710101204.04 total 50.85 last 62.75) - open/close 3869 (time 1710101214.06 total 60.87 last 65.95) - open/close 4482 (time 1710101224.06 total 70.87 last 61.28) - open/close 4989 (time 1710101234.07 total 80.88 last 50.65) - open/close 5534 (time 1710101244.08 total 90.89 last 54.47) - open/close 6090 (time 1710101254.08 total 100.89 last 55.57) - open/close 6620 (time 1710101264.11 total 110.92 last 52.86) - open/close 7056 (time 1710101274.12 total 120.93 last 43.56) - open/close 7510 (time 1710101294.70 total 141.51 last 22.06) - open/close 7537 (time 1710101306.24 total 153.05 last 2.34) - open/close 7971 (time 1710101316.25 total 163.05 last 43.37) - open/close 8430 (time 1710101326.26 total 173.06 last 45.85) - open/close 8899 (time 1710101336.27 total 183.08 last 46.82) - open/close 9303 (time 1710101346.27 total 193.08 last 40.40) - open/close 9680 (time 1710101356.28 total 203.08 last 37.69) total: 10000 open/close in 211.02 seconds: 47.39 ops/second - unlinked 0 (time 1710101367 ; total 0 ; last 0) total: 10000 unlinks in 127 seconds: 78.740158 unlinks/second - open/close 733 (time 1710101507.57 total 10.04 last 73.04) - open/close 1486 (time 1710101517.59 total 20.05 last 75.20) - open/close 2308 (time 1710101527.60 total 30.06 last 82.10) - open/close 2992 (time 1710101537.60 total 40.07 last 68.37) - open/close 3628 (time 1710101547.61 total 50.07 last 63.58) - open/close 4226 (time 1710101557.62 total 60.08 last 59.74) - open/close 4867 (time 1710101567.62 total 70.09 last 64.05) - open/close 5082 (time 1710101582.52 total 84.98 last 14.44) - open/close 5590 (time 1710101592.53 total 94.99 last 50.74) - open/close 6125 (time 1710101602.53 total 105.00 last 53.46) - open/close 6605 (time 1710101612.54 total 115.00 last 47.97) - open/close 7068 (time 1710101622.54 total 125.00 last 46.30) - open/close 7531 (time 1710101632.55 total 135.01 last 46.26) - open/close 8007 (time 1710101642.56 total 145.03 last 47.53) - open/close 8434 (time 1710101652.58 total 155.04 last 42.65) - open/close 8868 (time 1710101662.59 total 165.05 last 43.34) - open/close 9307 (time 1710101672.60 total 175.06 last 43.85) - open/close 9622 (time 1710101691.12 total 193.59 last 17.01) total: 10000 open/close in 202.33 seconds: 49.42 ops/second - unlinked 0 (time 1710101702 ; total 0 ; last 0) total: 10000 unlinks in 126 seconds: 79.365082 unlinks/second - open/close 783 (time 1710101841.42 total 10.00 last 78.27) - open/close 1560 (time 1710101851.43 total 20.02 last 77.60) - open/close 2325 (time 1710101861.44 total 30.02 last 76.46) - open/close 2993 (time 1710101871.45 total 40.04 last 66.69) - open/close 3564 (time 1710101881.47 total 50.06 last 57.01) - open/close 4252 (time 1710101891.47 total 60.06 last 68.76) - open/close 4899 (time 1710101901.48 total 70.07 last 64.64) - open/close 5502 (time 1710101911.49 total 80.08 last 60.24) - open/close 6084 (time 1710101921.51 total 90.10 last 58.08) - open/close 6619 (time 1710101931.53 total 100.12 last 53.38) - open/close 7104 (time 1710101941.55 total 110.13 last 48.45) - open/close 7590 (time 1710101951.55 total 120.14 last 48.58) - open/close 8116 (time 1710101961.57 total 130.16 last 52.50) - open/close 8553 (time 1710101971.58 total 140.16 last 43.67) - open/close 9002 (time 1710101981.59 total 150.17 last 44.86) - open/close 9474 (time 1710101991.59 total 160.17 last 47.20) - open/close 9948 (time 1710102001.61 total 170.19 last 47.30) total: 10000 open/close in 171.40 seconds: 58.34 ops/second - unlinked 0 (time 1710102005 ; total 0 ; last 0) total: 10000 unlinks in 114 seconds: 87.719299 unlinks/second - open/close 858 (time 1710102132.08 total 10.01 last 85.72) - open/close 1635 (time 1710102142.09 total 20.02 last 77.62) - open/close 2375 (time 1710102152.10 total 30.02 last 73.97) - open/close 2984 (time 1710102162.11 total 40.04 last 60.79) - open/close 3568 (time 1710102172.12 total 50.05 last 58.35) - open/close 4201 (time 1710102182.12 total 60.05 last 63.30) - open/close 4803 (time 1710102192.13 total 70.06 last 60.16) - open/close 5406 (time 1710102202.14 total 80.07 last 60.22) - open/close 5925 (time 1710102212.15 total 90.08 last 51.84) - open/close 6404 (time 1710102222.17 total 100.09 last 47.84) - open/close 6889 (time 1710102232.17 total 110.10 last 48.49) - open/close 7361 (time 1710102242.18 total 120.11 last 47.14) - open/close 7819 (time 1710102252.19 total 130.11 last 45.77) - open/close 8308 (time 1710102262.20 total 140.12 last 48.85) - open/close 8760 (time 1710102272.22 total 150.14 last 45.12) - open/close 9204 (time 1710102282.23 total 160.15 last 44.35) - open/close 9690 (time 1710102292.23 total 170.16 last 48.60) total: 10000 open/close in 177.25 seconds: 56.42 ops/second - unlinked 0 (time 1710102301 ; total 0 ; last 0) total: 10000 unlinks in 137 seconds: 72.992699 unlinks/second - open/close 793 (time 1710102450.71 total 10.00 last 79.27) - open/close 1569 (time 1710102460.71 total 20.01 last 77.58) - open/close 2300 (time 1710102470.72 total 30.02 last 73.04) - open/close 2877 (time 1710102490.99 total 50.29 last 28.47) - open/close 3495 (time 1710102500.99 total 60.29 last 61.78) - open/close 4064 (time 1710102511.01 total 70.30 last 56.83) - open/close 4621 (time 1710102521.01 total 80.31 last 55.66) - open/close 5185 (time 1710102541.90 total 101.19 last 27.00) - open/close 5499 (time 1710102559.74 total 119.03 last 17.60) - open/close 6048 (time 1710102569.80 total 129.09 last 54.58) - open/close 6626 (time 1710102579.80 total 139.10 last 57.76) - open/close 7157 (time 1710102589.82 total 149.12 last 52.99) - open/close 7649 (time 1710102599.82 total 159.12 last 49.19) - open/close 8109 (time 1710102609.83 total 169.12 last 45.98) - open/close 8563 (time 1710102619.83 total 179.13 last 45.37) - open/close 9018 (time 1710102629.85 total 189.15 last 45.44) - open/close 9540 (time 1710102639.85 total 199.15 last 52.17) - open/close 9956 (time 1710102649.85 total 209.15 last 41.60) total: 10000 open/close in 210.16 seconds: 47.58 ops/second - unlinked 0 (time 1710102653 ; total 0 ; last 0) total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second - open/close 193 (time 1710102787.77 total 13.52 last 14.28) - open/close 1037 (time 1710102797.77 total 23.52 last 84.36) - open/close 1760 (time 1710102807.78 total 33.53 last 72.20) - open/close 2438 (time 1710102817.79 total 43.54 last 67.77) - open/close 3112 (time 1710102827.80 total 53.55 last 67.35) - open/close 3739 (time 1710102837.80 total 63.55 last 62.68) - open/close 4283 (time 1710102847.80 total 73.55 last 54.38) - open/close 4825 (time 1710102857.85 total 83.60 last 53.95) - open/close 5408 (time 1710102867.85 total 93.60 last 58.29) - open/close 5953 (time 1710102877.85 total 103.60 last 54.50) - open/close 6432 (time 1710102887.86 total 113.61 last 47.87) - open/close 6924 (time 1710102897.89 total 123.64 last 49.05) - open/close 7357 (time 1710102907.92 total 133.67 last 43.16) - open/close 7825 (time 1710102928.40 total 154.14 last 22.86) - open/close 8298 (time 1710102938.41 total 164.16 last 47.22) - open/close 8722 (time 1710102948.42 total 174.17 last 42.39) - open/close 9130 (time 1710102958.45 total 184.20 last 40.67) - open/close 9567 (time 1710102968.47 total 194.22 last 43.62) total: 10000 open/close in 204.16 seconds: 48.98 ops/second - unlinked 0 (time 1710102981 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 834 (time 1710103105.89 total 10.01 last 83.28) - open/close 1643 (time 1710103115.90 total 20.02 last 80.84) - open/close 2433 (time 1710103125.94 total 30.07 last 78.66) - open/close 3146 (time 1710103135.95 total 40.08 last 71.22) - open/close 3797 (time 1710103145.99 total 50.12 last 64.81) - open/close 4422 (time 1710103156.00 total 60.13 last 62.44) - open/close 5064 (time 1710103166.01 total 70.14 last 64.16) - open/close 5674 (time 1710103176.02 total 80.15 last 60.93) - open/close 6369 (time 1710103186.04 total 90.16 last 69.40) - open/close 6916 (time 1710103196.04 total 100.17 last 54.68) - open/close 7511 (time 1710103206.05 total 110.18 last 59.42) - open/close 8055 (time 1710103216.07 total 120.20 last 54.29) - open/close 8598 (time 1710103226.08 total 130.21 last 54.24) - open/close 9093 (time 1710103236.09 total 140.21 last 49.50) - open/close 9625 (time 1710103246.10 total 150.23 last 53.12) total: 10000 open/close in 158.22 seconds: 63.20 ops/second - unlinked 0 (time 1710103256 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 1066 (time 1710103364.66 total 10.00 last 106.57) - open/close 2028 (time 1710103374.67 total 20.01 last 96.13) - open/close 2941 (time 1710103384.67 total 30.01 last 91.29) - open/close 3865 (time 1710103394.68 total 40.02 last 92.32) - open/close 4686 (time 1710103404.68 total 50.03 last 82.06) - open/close 5445 (time 1710103414.69 total 60.03 last 75.84) - open/close 6230 (time 1710103424.70 total 70.04 last 78.44) - open/close 6971 (time 1710103434.71 total 80.05 last 74.00) - open/close 7632 (time 1710103444.72 total 90.06 last 66.05) - open/close 8319 (time 1710103454.72 total 100.07 last 68.67) - open/close 8948 (time 1710103464.73 total 110.07 last 62.87) - open/close 9598 (time 1710103474.73 total 120.08 last 64.96) total: 10000 open/close in 126.81 seconds: 78.86 ops/second - unlinked 0 (time 1710103483 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1631 (time 1710103562.25 total 10.00 last 163.08) - open/close 3236 (time 1710103572.26 total 20.01 last 160.43) - open/close 4729 (time 1710103582.26 total 30.01 last 149.28) - open/close 6129 (time 1710103592.26 total 40.01 last 139.91) - open/close 7455 (time 1710103602.26 total 50.01 last 132.60) - open/close 8704 (time 1710103612.27 total 60.02 last 124.84) - open/close 9934 (time 1710103622.27 total 70.02 last 122.95) total: 10000 open/close in 70.56 seconds: 141.72 ops/second - unlinked 0 (time 1710103623 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.127@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.127@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg127-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg127-server: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg127-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server unloading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2