== conf-sanity test 69: replace an OST with the same index ========================================================== 16:03:28 (1710101008) start mds service on oleg301-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg301-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg301-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99968. rc=0 - open/close 1399 (time 1710101051.71 total 10.01 last 139.82) - open/close 2554 (time 1710101061.72 total 20.01 last 115.44) - open/close 3745 (time 1710101071.72 total 30.01 last 119.06) - open/close 4944 (time 1710101081.73 total 40.02 last 119.86) - open/close 5961 (time 1710101091.74 total 50.03 last 101.58) - open/close 6800 (time 1710101101.74 total 60.04 last 83.84) - open/close 7628 (time 1710101111.78 total 70.07 last 82.52) - open/close 8447 (time 1710101121.79 total 80.08 last 81.83) - open/close 9202 (time 1710101131.79 total 90.09 last 75.44) - open/close 9901 (time 1710101141.81 total 100.10 last 69.79) total: 10000 open/close in 101.62 seconds: 98.41 ops/second - unlinked 0 (time 1710101145 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1294 (time 1710101236.96 total 10.01 last 129.31) - open/close 2535 (time 1710101246.97 total 20.02 last 123.99) - open/close 3692 (time 1710101256.97 total 30.02 last 115.69) - open/close 4772 (time 1710101267.00 total 40.05 last 107.68) - open/close 5769 (time 1710101277.01 total 50.05 last 99.63) - open/close 6719 (time 1710101287.02 total 60.07 last 94.88) - open/close 7621 (time 1710101297.02 total 70.07 last 90.18) - open/close 8485 (time 1710101307.03 total 80.07 last 86.34) - open/close 9299 (time 1710101317.04 total 90.08 last 81.32) total: 10000 open/close in 98.71 seconds: 101.30 ops/second - unlinked 0 (time 1710101327 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1310 (time 1710101404.83 total 10.01 last 130.83) - open/close 2499 (time 1710101414.84 total 20.02 last 118.86) - open/close 3619 (time 1710101424.85 total 30.02 last 111.90) - open/close 4660 (time 1710101434.85 total 40.03 last 104.08) - open/close 5648 (time 1710101444.86 total 50.03 last 98.72) - open/close 6599 (time 1710101454.86 total 60.04 last 95.02) - open/close 7538 (time 1710101464.88 total 70.06 last 93.78) - open/close 8384 (time 1710101474.88 total 80.06 last 84.58) - open/close 9163 (time 1710101484.88 total 90.06 last 77.87) - open/close 9918 (time 1710101494.90 total 100.08 last 75.40) total: 10000 open/close in 101.19 seconds: 98.82 ops/second - unlinked 0 (time 1710101497 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1354 (time 1710101577.36 total 10.00 last 135.34) - open/close 2601 (time 1710101587.36 total 20.01 last 124.68) - open/close 3758 (time 1710101597.36 total 30.01 last 115.69) - open/close 4798 (time 1710101607.37 total 40.02 last 103.89) - open/close 5787 (time 1710101617.37 total 50.02 last 98.85) - open/close 6639 (time 1710101637.74 total 70.39 last 41.84) - open/close 7541 (time 1710101647.74 total 80.39 last 90.14) - open/close 8340 (time 1710101657.74 total 90.39 last 79.90) - open/close 9153 (time 1710101667.75 total 100.40 last 81.25) - open/close 9968 (time 1710101677.76 total 110.41 last 81.41) total: 10000 open/close in 110.78 seconds: 90.27 ops/second - unlinked 0 (time 1710101679 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1374 (time 1710101758.01 total 10.00 last 137.34) - open/close 2654 (time 1710101768.01 total 20.01 last 127.96) - open/close 3816 (time 1710101778.02 total 30.02 last 116.09) - open/close 4896 (time 1710101788.02 total 40.02 last 107.99) - open/close 5940 (time 1710101798.02 total 50.02 last 104.36) - open/close 6898 (time 1710101808.03 total 60.03 last 95.73) - open/close 7825 (time 1710101818.04 total 70.03 last 92.65) - open/close 8712 (time 1710101828.04 total 80.04 last 88.68) - open/close 9551 (time 1710101838.05 total 90.05 last 83.77) total: 10000 open/close in 95.46 seconds: 104.76 ops/second - unlinked 0 (time 1710101845 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1183 (time 1710101925.30 total 10.03 last 117.97) - open/close 2324 (time 1710101935.30 total 20.03 last 114.10) - open/close 3398 (time 1710101945.30 total 30.03 last 107.39) - open/close 4543 (time 1710101955.30 total 40.03 last 114.47) - open/close 5579 (time 1710101965.31 total 50.04 last 103.53) - open/close 6559 (time 1710101975.31 total 60.04 last 97.96) - open/close 7509 (time 1710101985.31 total 70.05 last 94.95) - open/close 8425 (time 1710101995.32 total 80.05 last 91.54) - open/close 9288 (time 1710102005.34 total 90.07 last 86.16) total: 10000 open/close in 98.37 seconds: 101.66 ops/second - unlinked 0 (time 1710102015 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1292 (time 1710102092.30 total 10.00 last 129.18) - open/close 2553 (time 1710102102.31 total 20.01 last 126.05) - open/close 3722 (time 1710102112.32 total 30.01 last 116.80) - open/close 4798 (time 1710102122.32 total 40.02 last 107.55) - open/close 5828 (time 1710102132.32 total 50.02 last 102.97) - open/close 6816 (time 1710102142.33 total 60.02 last 98.78) - open/close 7732 (time 1710102152.33 total 70.03 last 91.56) - open/close 8723 (time 1710102162.34 total 80.03 last 99.03) - open/close 9617 (time 1710102172.34 total 90.04 last 89.38) total: 10000 open/close in 94.25 seconds: 106.10 ops/second - unlinked 0 (time 1710102177 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1505 (time 1710102246.36 total 10.00 last 150.48) - open/close 2795 (time 1710102256.36 total 20.00 last 128.99) - open/close 3998 (time 1710102266.36 total 30.01 last 120.25) - open/close 4380 (time 1710102280.70 total 44.34 last 26.64) - open/close 5482 (time 1710102290.71 total 54.35 last 110.12) - open/close 6555 (time 1710102300.72 total 64.36 last 107.20) - open/close 7653 (time 1710102310.72 total 74.36 last 109.77) - open/close 8744 (time 1710102320.72 total 84.37 last 109.07) - open/close 9783 (time 1710102330.73 total 94.37 last 103.83) total: 10000 open/close in 96.63 seconds: 103.49 ops/second - unlinked 0 (time 1710102334 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1616 (time 1710102397.94 total 10.00 last 161.59) - open/close 3053 (time 1710102407.95 total 20.00 last 143.66) - open/close 4394 (time 1710102417.95 total 30.01 last 134.01) - open/close 5586 (time 1710102427.96 total 40.01 last 119.15) - open/close 6702 (time 1710102437.96 total 50.02 last 111.57) - open/close 7739 (time 1710102447.97 total 60.02 last 103.62) - open/close 8779 (time 1710102457.97 total 70.03 last 103.99) - open/close 9820 (time 1710102467.97 total 80.03 last 104.09) total: 10000 open/close in 81.90 seconds: 122.10 ops/second - unlinked 0 (time 1710102471 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1959 (time 1710102528.58 total 10.00 last 195.86) - open/close 3650 (time 1710102538.58 total 20.00 last 169.08) - open/close 5151 (time 1710102548.58 total 30.00 last 150.09) - open/close 6657 (time 1710102558.58 total 40.01 last 150.54) - open/close 8132 (time 1710102568.59 total 50.01 last 147.43) - open/close 9719 (time 1710102578.59 total 60.02 last 158.62) total: 10000 open/close in 61.85 seconds: 161.69 ops/second - unlinked 0 (time 1710102581 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.101@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.101@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg301-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg301-server: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg301-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre On OST0, 9127 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server unloading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2