== conf-sanity test 69: replace an OST with the same index ========================================================== 15:59:43 (1710100783) start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 2433 (time 1710100810.08 total 10.00 last 243.25) - open/close 4166 (time 1710100820.08 total 20.00 last 173.29) - open/close 5485 (time 1710100830.09 total 30.01 last 131.83) - open/close 6751 (time 1710100840.09 total 40.01 last 126.52) - open/close 7937 (time 1710100850.10 total 50.02 last 118.56) - open/close 8824 (time 1710100860.10 total 60.02 last 88.66) - open/close 9764 (time 1710100870.11 total 70.03 last 93.93) total: 10000 open/close in 72.64 seconds: 137.66 ops/second - unlinked 0 (time 1710100873 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1593 (time 1710100933.37 total 10.00 last 159.27) - open/close 3002 (time 1710100943.37 total 20.01 last 140.83) - open/close 4434 (time 1710100953.38 total 30.01 last 143.15) - open/close 5624 (time 1710100963.38 total 40.01 last 118.95) - open/close 6639 (time 1710100973.39 total 50.02 last 101.43) - open/close 7384 (time 1710100983.40 total 60.03 last 74.44) - open/close 8082 (time 1710100993.41 total 70.05 last 69.67) - open/close 8927 (time 1710101003.41 total 80.05 last 84.50) - open/close 9702 (time 1710101013.43 total 90.06 last 77.39) total: 10000 open/close in 94.45 seconds: 105.88 ops/second - unlinked 0 (time 1710101019 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 1130 (time 1710101116.32 total 10.00 last 112.96) - open/close 2165 (time 1710101126.32 total 20.01 last 103.45) - open/close 3035 (time 1710101136.33 total 30.02 last 86.91) - open/close 3923 (time 1710101146.33 total 40.02 last 88.79) - open/close 4833 (time 1710101156.33 total 50.02 last 90.98) - open/close 5673 (time 1710101166.34 total 60.03 last 83.95) - open/close 6432 (time 1710101176.35 total 70.04 last 75.81) - open/close 7171 (time 1710101186.35 total 80.04 last 73.89) - open/close 7919 (time 1710101196.36 total 90.05 last 74.72) - open/close 8603 (time 1710101206.38 total 100.07 last 68.27) - open/close 9306 (time 1710101216.39 total 110.08 last 70.22) - open/close 9959 (time 1710101226.40 total 120.09 last 65.28) total: 10000 open/close in 120.75 seconds: 82.82 ops/second - unlinked 0 (time 1710101228 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1215 (time 1710101317.57 total 10.01 last 121.43) - open/close 2316 (time 1710101327.58 total 20.01 last 110.06) - open/close 3296 (time 1710101337.58 total 30.01 last 97.96) - open/close 4259 (time 1710101347.58 total 40.02 last 96.27) - open/close 5150 (time 1710101357.60 total 50.03 last 88.99) - open/close 6002 (time 1710101367.60 total 60.03 last 85.16) - open/close 6811 (time 1710101377.61 total 70.04 last 80.83) - open/close 7611 (time 1710101387.61 total 80.05 last 79.96) - open/close 8336 (time 1710101397.62 total 90.05 last 72.49) - open/close 9036 (time 1710101407.63 total 100.06 last 69.89) - open/close 9711 (time 1710101417.63 total 110.07 last 67.49) total: 10000 open/close in 114.24 seconds: 87.54 ops/second - unlinked 0 (time 1710101423 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1190 (time 1710101512.89 total 10.01 last 118.93) - open/close 2272 (time 1710101522.89 total 20.01 last 108.18) - open/close 3274 (time 1710101532.90 total 30.02 last 100.09) - open/close 4190 (time 1710101542.90 total 40.02 last 91.58) - open/close 5064 (time 1710101552.90 total 50.02 last 87.39) - open/close 5904 (time 1710101562.91 total 60.03 last 83.97) - open/close 6725 (time 1710101572.91 total 70.03 last 82.09) - open/close 7520 (time 1710101582.92 total 80.04 last 79.42) - open/close 8285 (time 1710101592.92 total 90.04 last 76.47) - open/close 8996 (time 1710101602.93 total 100.05 last 71.06) - open/close 9657 (time 1710101612.93 total 110.05 last 66.06) total: 10000 open/close in 115.05 seconds: 86.92 ops/second - unlinked 0 (time 1710101619 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1151 (time 1710101708.61 total 10.00 last 115.07) - open/close 2241 (time 1710101718.62 total 20.01 last 108.90) - open/close 3241 (time 1710101728.63 total 30.02 last 99.90) - open/close 4201 (time 1710101738.63 total 40.02 last 95.98) - open/close 5123 (time 1710101748.64 total 50.03 last 92.14) - open/close 6032 (time 1710101758.64 total 60.03 last 90.87) - open/close 6875 (time 1710101768.65 total 70.04 last 84.26) - open/close 7702 (time 1710101778.65 total 80.05 last 82.64) - open/close 8472 (time 1710101788.66 total 90.05 last 76.94) - open/close 9227 (time 1710101798.67 total 100.06 last 75.43) - open/close 9958 (time 1710101808.67 total 110.07 last 73.08) total: 10000 open/close in 110.64 seconds: 90.38 ops/second - unlinked 0 (time 1710101810 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1152 (time 1710101911.86 total 10.01 last 115.12) - open/close 2177 (time 1710101921.87 total 20.02 last 102.41) - open/close 3102 (time 1710101931.87 total 30.02 last 92.46) - open/close 4036 (time 1710101941.87 total 40.02 last 93.38) - open/close 4999 (time 1710101951.87 total 50.02 last 96.30) - open/close 5871 (time 1710101961.88 total 60.03 last 87.14) - open/close 6677 (time 1710101971.88 total 70.03 last 80.60) - open/close 7488 (time 1710101981.89 total 80.04 last 81.03) - open/close 8255 (time 1710101991.90 total 90.05 last 76.64) - open/close 9018 (time 1710102001.90 total 100.05 last 76.30) - open/close 9743 (time 1710102011.90 total 110.05 last 72.45) total: 10000 open/close in 113.67 seconds: 87.97 ops/second - unlinked 0 (time 1710102017 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 1227 (time 1710102104.46 total 10.01 last 122.60) - open/close 2350 (time 1710102114.46 total 20.01 last 112.26) - open/close 3426 (time 1710102124.47 total 30.02 last 107.54) - open/close 4399 (time 1710102134.48 total 40.03 last 97.18) - open/close 5361 (time 1710102144.49 total 50.04 last 96.08) - open/close 6250 (time 1710102154.49 total 60.04 last 88.88) - open/close 7163 (time 1710102164.50 total 70.05 last 91.29) - open/close 7972 (time 1710102174.50 total 80.05 last 80.89) - open/close 8789 (time 1710102184.51 total 90.05 last 81.64) - open/close 9599 (time 1710102194.51 total 100.06 last 80.95) total: 10000 open/close in 105.17 seconds: 95.09 ops/second - unlinked 0 (time 1710102201 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 1335 (time 1710102281.45 total 10.00 last 133.46) - open/close 2530 (time 1710102291.45 total 20.00 last 119.50) - open/close 3640 (time 1710102301.45 total 30.00 last 110.99) - open/close 4788 (time 1710102311.45 total 40.01 last 114.74) - open/close 5864 (time 1710102321.46 total 50.01 last 107.55) - open/close 6876 (time 1710102331.46 total 60.02 last 101.17) - open/close 7885 (time 1710102341.46 total 70.02 last 100.89) - open/close 8795 (time 1710102351.46 total 80.02 last 90.97) - open/close 9657 (time 1710102361.47 total 90.03 last 86.16) total: 10000 open/close in 93.96 seconds: 106.42 ops/second - unlinked 0 (time 1710102366 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1392 (time 1710102440.76 total 10.00 last 139.18) - open/close 2684 (time 1710102450.76 total 20.00 last 129.17) - open/close 3924 (time 1710102460.76 total 30.01 last 123.95) - open/close 5090 (time 1710102470.77 total 40.01 last 116.52) - open/close 6228 (time 1710102480.78 total 50.02 last 113.73) - open/close 7406 (time 1710102490.78 total 60.02 last 117.77) - open/close 8523 (time 1710102500.78 total 70.03 last 111.67) - open/close 9651 (time 1710102510.78 total 80.03 last 112.76) total: 10000 open/close in 83.29 seconds: 120.06 ops/second - unlinked 0 (time 1710102515 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.119@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.119@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg319-server: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg319-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre On OST0, 17725 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2