== conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:32 (1710100832) start mds service on oleg243-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1594 (time 1710100861.38 total 10.00 last 159.33) - open/close 3092 (time 1710100871.39 total 20.01 last 149.69) - open/close 4436 (time 1710100881.39 total 30.01 last 134.38) - open/close 5733 (time 1710100891.39 total 40.01 last 129.67) - open/close 6915 (time 1710100901.40 total 50.03 last 118.07) - open/close 8090 (time 1710100911.40 total 60.03 last 117.50) - open/close 8899 (time 1710100921.40 total 70.03 last 80.89) - open/close 9808 (time 1710100931.40 total 80.03 last 90.87) total: 10000 open/close in 82.23 seconds: 121.61 ops/second - unlinked 0 (time 1710100935 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1499 (time 1710101009.82 total 10.00 last 149.84) - open/close 2704 (time 1710101019.82 total 20.01 last 120.47) - open/close 3907 (time 1710101029.83 total 30.01 last 120.24) - open/close 5131 (time 1710101039.84 total 40.02 last 122.27) - open/close 6087 (time 1710101049.84 total 50.02 last 95.59) - open/close 6951 (time 1710101059.85 total 60.03 last 86.33) - open/close 7717 (time 1710101069.86 total 70.04 last 76.53) - open/close 8412 (time 1710101079.86 total 80.04 last 69.49) - open/close 9099 (time 1710101089.86 total 90.04 last 68.68) - open/close 9851 (time 1710101099.87 total 100.05 last 75.13) total: 10000 open/close in 102.15 seconds: 97.89 ops/second - unlinked 0 (time 1710101103 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1151 (time 1710101192.37 total 10.00 last 115.08) - open/close 2220 (time 1710101202.38 total 20.00 last 106.89) - open/close 3243 (time 1710101212.38 total 30.01 last 102.25) - open/close 4205 (time 1710101222.38 total 40.01 last 96.17) - open/close 5008 (time 1710101232.39 total 50.02 last 80.24) - open/close 5767 (time 1710101242.40 total 60.03 last 75.84) - open/close 6609 (time 1710101252.41 total 70.04 last 84.11) - open/close 7424 (time 1710101262.43 total 80.05 last 81.37) - open/close 8146 (time 1710101272.43 total 90.06 last 72.18) - open/close 8892 (time 1710101282.43 total 100.06 last 74.60) - open/close 9601 (time 1710101292.43 total 110.06 last 70.87) total: 10000 open/close in 116.32 seconds: 85.97 ops/second - unlinked 0 (time 1710101300 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1160 (time 1710101395.15 total 10.01 last 115.92) - open/close 2188 (time 1710101405.15 total 20.01 last 102.79) - open/close 3165 (time 1710101415.16 total 30.01 last 97.64) - open/close 4075 (time 1710101425.16 total 40.02 last 90.95) - open/close 4967 (time 1710101435.16 total 50.02 last 89.20) - open/close 5834 (time 1710101445.17 total 60.02 last 86.68) - open/close 6643 (time 1710101455.18 total 70.04 last 80.78) - open/close 7379 (time 1710101465.19 total 80.04 last 73.57) - open/close 8114 (time 1710101475.19 total 90.05 last 73.45) - open/close 8756 (time 1710101485.21 total 100.06 last 64.11) - open/close 9395 (time 1710101495.22 total 110.07 last 63.84) total: 10000 open/close in 119.40 seconds: 83.75 ops/second - unlinked 0 (time 1710101506 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1067 (time 1710101610.12 total 10.00 last 106.68) - open/close 2080 (time 1710101620.13 total 20.01 last 101.23) - open/close 3099 (time 1710101630.13 total 30.01 last 101.84) - open/close 4060 (time 1710101640.13 total 40.02 last 96.10) - open/close 4913 (time 1710101650.14 total 50.02 last 85.24) - open/close 5766 (time 1710101660.15 total 60.03 last 85.25) - open/close 6614 (time 1710101670.15 total 70.03 last 84.79) - open/close 7371 (time 1710101680.16 total 80.04 last 75.63) - open/close 8121 (time 1710101690.17 total 90.05 last 74.93) - open/close 8861 (time 1710101700.18 total 100.06 last 73.93) - open/close 9546 (time 1710101710.19 total 110.07 last 68.40) total: 10000 open/close in 116.56 seconds: 85.79 ops/second - unlinked 0 (time 1710101718 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1086 (time 1710101809.91 total 10.00 last 108.59) - open/close 2001 (time 1710101830.15 total 30.24 last 45.21) - open/close 2987 (time 1710101840.16 total 40.25 last 98.53) - open/close 3885 (time 1710101850.17 total 50.26 last 89.70) - open/close 4720 (time 1710101860.18 total 60.27 last 83.39) - open/close 5473 (time 1710101870.20 total 70.29 last 75.19) - open/close 6254 (time 1710101880.21 total 80.29 last 78.05) - open/close 6979 (time 1710101890.21 total 90.30 last 72.44) - open/close 7697 (time 1710101900.22 total 100.31 last 71.74) - open/close 8458 (time 1710101910.24 total 110.33 last 75.97) - open/close 8902 (time 1710101928.54 total 128.62 last 24.27) - open/close 9513 (time 1710101938.55 total 138.64 last 61.02) total: 10000 open/close in 147.30 seconds: 67.89 ops/second - unlinked 0 (time 1710101949 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1178 (time 1710102042.65 total 10.01 last 117.70) - open/close 2212 (time 1710102052.66 total 20.02 last 103.32) - open/close 3142 (time 1710102062.66 total 30.02 last 92.97) - open/close 4027 (time 1710102072.67 total 40.03 last 88.42) - open/close 4925 (time 1710102082.67 total 50.03 last 89.78) - open/close 5710 (time 1710102092.68 total 60.04 last 78.42) - open/close 6532 (time 1710102102.69 total 70.05 last 82.12) - open/close 7225 (time 1710102112.70 total 80.06 last 69.24) - open/close 7945 (time 1710102122.71 total 90.07 last 71.93) - open/close 8615 (time 1710102132.73 total 100.09 last 66.89) - open/close 9238 (time 1710102142.73 total 110.09 last 62.27) - open/close 9878 (time 1710102152.75 total 120.10 last 63.91) total: 10000 open/close in 122.00 seconds: 81.97 ops/second - unlinked 0 (time 1710102156 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1150 (time 1710102250.74 total 10.01 last 114.94) - open/close 2226 (time 1710102260.75 total 20.02 last 107.50) - open/close 3188 (time 1710102270.76 total 30.02 last 96.13) - open/close 4027 (time 1710102280.76 total 40.02 last 83.89) - open/close 4569 (time 1710102298.82 total 58.08 last 30.01) - open/close 5390 (time 1710102308.82 total 68.09 last 82.07) - open/close 6188 (time 1710102318.83 total 78.09 last 79.74) - open/close 6942 (time 1710102328.85 total 88.12 last 75.23) - open/close 7605 (time 1710102338.86 total 98.12 last 66.25) - open/close 8345 (time 1710102348.87 total 108.13 last 73.92) - open/close 9036 (time 1710102358.87 total 118.14 last 69.09) - open/close 9686 (time 1710102368.88 total 128.14 last 64.96) total: 10000 open/close in 132.94 seconds: 75.22 ops/second - unlinked 0 (time 1710102375 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 1164 (time 1710102479.26 total 10.00 last 116.38) - open/close 2202 (time 1710102489.27 total 20.01 last 103.71) - open/close 3207 (time 1710102499.27 total 30.01 last 100.49) - open/close 4141 (time 1710102509.27 total 40.02 last 93.35) - open/close 4989 (time 1710102519.28 total 50.02 last 84.75) - open/close 5793 (time 1710102529.28 total 60.02 last 80.40) - open/close 6576 (time 1710102539.29 total 70.03 last 78.24) - open/close 7388 (time 1710102549.29 total 80.04 last 81.13) - open/close 8086 (time 1710102559.29 total 90.04 last 69.80) - open/close 8798 (time 1710102569.30 total 100.04 last 71.18) - open/close 9472 (time 1710102579.31 total 110.06 last 67.30) total: 10000 open/close in 118.07 seconds: 84.69 ops/second - unlinked 0 (time 1710102589 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1135 (time 1710102681.47 total 10.01 last 113.41) - open/close 2184 (time 1710102691.48 total 20.02 last 104.80) - open/close 3285 (time 1710102701.49 total 30.02 last 110.02) - open/close 4211 (time 1710102711.49 total 40.03 last 92.58) - open/close 5077 (time 1710102721.49 total 50.03 last 86.58) - open/close 5871 (time 1710102731.50 total 60.04 last 79.32) - open/close 6710 (time 1710102741.51 total 70.05 last 83.82) - open/close 7432 (time 1710102751.53 total 80.06 last 72.10) - open/close 8170 (time 1710102761.55 total 90.09 last 73.59) - open/close 8938 (time 1710102771.57 total 100.10 last 76.70) - open/close 9664 (time 1710102781.59 total 110.12 last 72.44) total: 10000 open/close in 114.81 seconds: 87.10 ops/second - unlinked 0 (time 1710102787 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.143@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.143@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg243-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg243-server: oleg243-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg243-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server unloading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2