== conf-sanity test 69: replace an OST with the same index ========================================================== 16:01:43 (1710100903) start mds service on oleg213-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg213-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg213-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock oleg213-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11840 inodes available. Want 99936. rc=0 - open/close 1132 (time 1710100946.45 total 10.00 last 113.17) - open/close 2216 (time 1710100956.45 total 20.01 last 108.35) - open/close 3199 (time 1710100966.46 total 30.01 last 98.24) - open/close 4214 (time 1710100976.47 total 40.02 last 101.43) - open/close 5173 (time 1710100986.48 total 50.03 last 95.81) - open/close 6117 (time 1710100996.49 total 60.04 last 94.31) - open/close 7071 (time 1710101006.50 total 70.05 last 95.32) - open/close 7865 (time 1710101016.50 total 80.05 last 79.40) - open/close 8630 (time 1710101026.50 total 90.06 last 76.45) - open/close 9488 (time 1710101036.50 total 100.06 last 85.79) total: 10000 open/close in 106.59 seconds: 93.82 ops/second - unlinked 0 (time 1710101044 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1113 (time 1710101138.11 total 10.00 last 111.30) - open/close 2197 (time 1710101148.12 total 20.01 last 108.32) - open/close 3075 (time 1710101158.18 total 30.06 last 87.34) - open/close 3918 (time 1710101168.18 total 40.07 last 84.26) - open/close 4740 (time 1710101178.19 total 50.08 last 82.11) - open/close 5493 (time 1710101188.20 total 60.09 last 75.22) - open/close 6188 (time 1710101198.22 total 70.10 last 69.41) - open/close 6860 (time 1710101208.22 total 80.11 last 67.15) - open/close 7570 (time 1710101218.23 total 90.12 last 70.93) - open/close 8173 (time 1710101228.24 total 100.12 last 60.27) - open/close 8769 (time 1710101238.24 total 110.13 last 59.59) - open/close 9356 (time 1710101248.24 total 120.13 last 58.69) - open/close 9703 (time 1710101265.25 total 137.13 last 20.41) total: 10000 open/close in 142.31 seconds: 70.27 ops/second - unlinked 0 (time 1710101272 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1031 (time 1710101377.13 total 10.01 last 103.04) - open/close 1978 (time 1710101387.13 total 20.01 last 94.66) - open/close 2865 (time 1710101397.15 total 30.02 last 88.60) - open/close 3676 (time 1710101407.15 total 40.03 last 81.04) - open/close 4457 (time 1710101417.15 total 50.03 last 78.09) - open/close 5181 (time 1710101427.16 total 60.03 last 72.39) - open/close 5859 (time 1710101437.16 total 70.03 last 67.78) - open/close 6576 (time 1710101447.17 total 80.04 last 71.64) - open/close 7208 (time 1710101457.17 total 90.04 last 63.20) - open/close 7807 (time 1710101467.17 total 100.04 last 59.89) - open/close 8352 (time 1710101477.18 total 110.06 last 54.44) - open/close 8925 (time 1710101487.18 total 120.06 last 57.27) - open/close 9483 (time 1710101497.19 total 130.07 last 55.76) total: 10000 open/close in 139.41 seconds: 71.73 ops/second - unlinked 0 (time 1710101508 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 1073 (time 1710101614.39 total 10.01 last 107.23) - open/close 1954 (time 1710101624.39 total 20.01 last 88.10) - open/close 2854 (time 1710101634.39 total 30.01 last 89.97) - open/close 3650 (time 1710101644.40 total 40.02 last 79.51) - open/close 4432 (time 1710101654.40 total 50.03 last 78.17) - open/close 5220 (time 1710101664.41 total 60.03 last 78.79) - open/close 5937 (time 1710101674.41 total 70.03 last 71.68) - open/close 6625 (time 1710101684.42 total 80.04 last 68.72) - open/close 7314 (time 1710101694.43 total 90.05 last 68.85) - open/close 7981 (time 1710101704.43 total 100.05 last 66.68) - open/close 8610 (time 1710101714.44 total 110.06 last 62.84) - open/close 9205 (time 1710101724.44 total 120.06 last 59.49) - open/close 9805 (time 1710101734.45 total 130.07 last 59.97) total: 10000 open/close in 133.45 seconds: 74.93 ops/second - unlinked 0 (time 1710101739 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 1024 (time 1710101851.52 total 10.01 last 102.31) - open/close 1943 (time 1710101861.52 total 20.01 last 91.90) - open/close 2791 (time 1710101871.52 total 30.01 last 84.79) - open/close 3584 (time 1710101881.52 total 40.01 last 79.30) - open/close 4340 (time 1710101891.53 total 50.01 last 75.59) - open/close 5105 (time 1710101901.53 total 60.01 last 76.49) - open/close 5880 (time 1710101911.57 total 70.05 last 77.18) - open/close 6584 (time 1710101921.57 total 80.06 last 70.40) - open/close 7230 (time 1710101931.59 total 90.08 last 64.44) - open/close 7846 (time 1710101941.60 total 100.08 last 61.57) - open/close 8420 (time 1710101951.61 total 110.09 last 57.35) - open/close 9020 (time 1710101961.61 total 120.10 last 59.96) - open/close 9584 (time 1710101971.62 total 130.10 last 56.38) total: 10000 open/close in 136.77 seconds: 73.11 ops/second - unlinked 0 (time 1710101980 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1058 (time 1710102080.20 total 10.00 last 105.79) - open/close 2016 (time 1710102090.20 total 20.00 last 95.79) - open/close 2126 (time 1710102102.53 total 32.33 last 8.93) - open/close 2988 (time 1710102112.53 total 42.33 last 86.19) - open/close 3800 (time 1710102122.53 total 52.33 last 81.18) - open/close 4582 (time 1710102132.54 total 62.34 last 78.13) - open/close 5329 (time 1710102142.54 total 72.34 last 74.69) - open/close 6058 (time 1710102152.54 total 82.34 last 72.86) - open/close 6772 (time 1710102162.55 total 92.35 last 71.35) - open/close 7446 (time 1710102172.56 total 102.36 last 67.33) - open/close 8083 (time 1710102182.56 total 112.36 last 63.68) - open/close 8711 (time 1710102192.57 total 122.37 last 62.77) - open/close 9297 (time 1710102202.58 total 132.38 last 58.54) - open/close 9888 (time 1710102212.59 total 142.39 last 59.04) total: 10000 open/close in 144.18 seconds: 69.36 ops/second - unlinked 0 (time 1710102216 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 1045 (time 1710102316.08 total 10.00 last 104.46) - open/close 2060 (time 1710102326.09 total 20.01 last 101.44) - open/close 2948 (time 1710102336.10 total 30.02 last 88.72) - open/close 3839 (time 1710102346.10 total 40.02 last 89.09) - open/close 4679 (time 1710102356.10 total 50.02 last 83.98) - open/close 5449 (time 1710102366.10 total 60.02 last 76.99) - open/close 6183 (time 1710102376.11 total 70.03 last 73.33) - open/close 6906 (time 1710102386.13 total 80.05 last 72.17) - open/close 7564 (time 1710102396.14 total 90.06 last 65.73) - open/close 8184 (time 1710102406.16 total 100.08 last 61.88) - open/close 8803 (time 1710102416.16 total 110.08 last 61.89) - open/close 9424 (time 1710102426.17 total 120.09 last 62.03) total: 10000 open/close in 129.84 seconds: 77.02 ops/second - unlinked 0 (time 1710102437 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1060 (time 1710102532.71 total 10.00 last 105.95) - open/close 2073 (time 1710102542.71 total 20.01 last 101.29) - open/close 3019 (time 1710102552.72 total 30.01 last 94.54) - open/close 3858 (time 1710102562.72 total 40.02 last 83.84) - open/close 4696 (time 1710102572.73 total 50.03 last 83.74) - open/close 5486 (time 1710102582.74 total 60.03 last 78.95) - open/close 6231 (time 1710102592.74 total 70.03 last 74.49) - open/close 6978 (time 1710102602.74 total 80.04 last 74.69) - open/close 7706 (time 1710102612.74 total 90.04 last 72.77) - open/close 8406 (time 1710102622.76 total 100.05 last 69.90) - open/close 9109 (time 1710102632.76 total 110.06 last 70.26) - open/close 9759 (time 1710102642.77 total 120.06 last 64.97) total: 10000 open/close in 123.86 seconds: 80.73 ops/second - unlinked 0 (time 1710102648 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1095 (time 1710102741.36 total 10.01 last 109.43) - open/close 2057 (time 1710102751.37 total 20.01 last 96.17) - open/close 3008 (time 1710102761.37 total 30.01 last 95.06) - open/close 3979 (time 1710102771.38 total 40.03 last 96.99) - open/close 4926 (time 1710102781.39 total 50.04 last 94.60) - open/close 5767 (time 1710102791.39 total 60.04 last 84.10) - open/close 6579 (time 1710102801.40 total 70.05 last 81.11) - open/close 7411 (time 1710102811.41 total 80.05 last 83.15) - open/close 8213 (time 1710102821.41 total 90.05 last 80.20) - open/close 8982 (time 1710102831.41 total 100.05 last 76.90) - open/close 9737 (time 1710102841.41 total 110.06 last 75.48) total: 10000 open/close in 113.50 seconds: 88.10 ops/second - unlinked 0 (time 1710102846 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1412 (time 1710102923.57 total 10.01 last 141.12) - open/close 2707 (time 1710102933.57 total 20.01 last 129.47) - open/close 3929 (time 1710102943.58 total 30.02 last 122.08) - open/close 5160 (time 1710102953.59 total 40.02 last 123.05) - open/close 6372 (time 1710102963.59 total 50.03 last 121.12) - open/close 7582 (time 1710102973.60 total 60.03 last 120.92) - open/close 8822 (time 1710102983.60 total 70.04 last 123.96) total: 10000 open/close in 78.88 seconds: 126.78 ops/second - unlinked 0 (time 1710102993 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.113@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.113@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg213-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg213-server: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg213-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg213-server: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg213-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock oleg213-server@tcp:/lustre /mnt/lustre On OST0, 20421 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg213-server unloading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2