== conf-sanity test 69: replace an OST with the same index ========================================================== 15:59:25 (1710100765) start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1638 (time 1710100793.06 total 10.00 last 163.75) - open/close 2966 (time 1710100803.06 total 20.01 last 132.76) - open/close 4112 (time 1710100813.06 total 30.01 last 114.59) - open/close 5168 (time 1710100823.06 total 40.01 last 105.58) - open/close 6181 (time 1710100833.06 total 50.01 last 101.30) - open/close 7279 (time 1710100843.06 total 60.01 last 109.77) - open/close 8291 (time 1710100853.07 total 70.02 last 101.16) - open/close 9128 (time 1710100863.07 total 80.02 last 83.70) total: 10000 open/close in 89.88 seconds: 111.26 ops/second - unlinked 0 (time 1710100874 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1149 (time 1710100951.80 total 10.00 last 114.84) - open/close 2106 (time 1710100961.81 total 20.01 last 95.67) - open/close 3219 (time 1710100971.81 total 30.01 last 111.27) - open/close 4182 (time 1710100981.81 total 40.01 last 96.26) - open/close 5145 (time 1710100991.82 total 50.02 last 96.22) - open/close 6111 (time 1710101001.83 total 60.03 last 96.51) - open/close 6928 (time 1710101011.84 total 70.04 last 81.61) - open/close 7658 (time 1710101021.84 total 80.04 last 72.99) - open/close 8485 (time 1710101031.85 total 90.05 last 82.67) - open/close 9233 (time 1710101041.85 total 100.05 last 74.78) - open/close 9855 (time 1710101051.86 total 110.06 last 62.15) total: 10000 open/close in 112.33 seconds: 89.02 ops/second - unlinked 0 (time 1710101055 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1039 (time 1710101158.32 total 10.00 last 103.89) - open/close 1999 (time 1710101168.32 total 20.00 last 95.99) - open/close 2937 (time 1710101178.33 total 30.01 last 93.71) - open/close 3760 (time 1710101188.34 total 40.02 last 82.24) - open/close 4558 (time 1710101198.35 total 50.03 last 79.69) - open/close 5271 (time 1710101208.35 total 60.03 last 71.28) - open/close 6014 (time 1710101218.36 total 70.04 last 74.25) - open/close 6632 (time 1710101228.37 total 80.05 last 61.73) - open/close 7257 (time 1710101238.37 total 90.06 last 62.47) - open/close 7894 (time 1710101248.38 total 100.07 last 63.63) - open/close 8505 (time 1710101258.39 total 110.07 last 61.07) - open/close 9070 (time 1710101268.40 total 120.09 last 56.42) - open/close 9625 (time 1710101278.42 total 130.10 last 55.42) total: 10000 open/close in 137.18 seconds: 72.89 ops/second - unlinked 0 (time 1710101287 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 1028 (time 1710101398.46 total 10.00 last 102.76) - open/close 1922 (time 1710101408.47 total 20.01 last 89.30) - open/close 2784 (time 1710101418.49 total 30.03 last 86.07) - open/close 3619 (time 1710101428.51 total 40.05 last 83.35) - open/close 4410 (time 1710101438.51 total 50.05 last 79.08) - open/close 5174 (time 1710101448.52 total 60.06 last 76.34) - open/close 5866 (time 1710101458.53 total 70.07 last 69.11) - open/close 6511 (time 1710101468.54 total 80.08 last 64.45) - open/close 7112 (time 1710101478.54 total 90.08 last 60.09) - open/close 7712 (time 1710101488.64 total 100.18 last 59.42) - open/close 8305 (time 1710101498.64 total 110.18 last 59.27) - open/close 8885 (time 1710101508.66 total 120.20 last 57.92) - open/close 9465 (time 1710101518.66 total 130.20 last 57.98) total: 10000 open/close in 139.57 seconds: 71.65 ops/second - unlinked 0 (time 1710101530 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1097 (time 1710101628.35 total 10.01 last 109.61) - open/close 2111 (time 1710101638.35 total 20.01 last 101.39) - open/close 2985 (time 1710101648.36 total 30.02 last 87.33) - open/close 3866 (time 1710101658.36 total 40.02 last 88.03) - open/close 4753 (time 1710101668.37 total 50.04 last 88.60) - open/close 5548 (time 1710101678.37 total 60.04 last 79.50) - open/close 6330 (time 1710101688.39 total 70.05 last 78.11) - open/close 7102 (time 1710101698.39 total 80.05 last 77.15) - open/close 7809 (time 1710101708.39 total 90.06 last 70.69) - open/close 8518 (time 1710101718.41 total 100.07 last 70.77) - open/close 9154 (time 1710101728.42 total 110.08 last 63.53) - open/close 9829 (time 1710101738.49 total 120.15 last 67.08) total: 10000 open/close in 123.05 seconds: 81.27 ops/second - unlinked 0 (time 1710101743 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 1086 (time 1710101838.95 total 10.00 last 108.58) - open/close 2100 (time 1710101848.96 total 20.01 last 101.34) - open/close 3004 (time 1710101858.96 total 30.01 last 90.37) - open/close 3831 (time 1710101868.98 total 40.03 last 82.53) - open/close 4658 (time 1710101878.99 total 50.04 last 82.67) - open/close 5445 (time 1710101889.00 total 60.04 last 78.63) - open/close 6217 (time 1710101899.00 total 70.05 last 77.15) - open/close 7028 (time 1710101909.02 total 80.06 last 80.99) - open/close 7125 (time 1710101921.64 total 92.69 last 7.68) - open/close 7792 (time 1710101931.67 total 102.72 last 66.50) - open/close 8417 (time 1710101941.68 total 112.73 last 62.47) - open/close 9002 (time 1710101951.68 total 122.73 last 58.49) - open/close 9587 (time 1710101961.68 total 132.73 last 58.47) total: 10000 open/close in 139.44 seconds: 71.71 ops/second - unlinked 0 (time 1710101970 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1097 (time 1710102062.26 total 10.01 last 109.64) - open/close 2145 (time 1710102072.26 total 20.01 last 104.78) - open/close 3155 (time 1710102082.26 total 30.01 last 100.98) - open/close 4073 (time 1710102092.27 total 40.01 last 91.78) - open/close 4977 (time 1710102102.28 total 50.02 last 90.29) - open/close 5746 (time 1710102112.28 total 60.03 last 76.86) - open/close 6509 (time 1710102122.29 total 70.04 last 76.23) - open/close 7240 (time 1710102132.30 total 80.05 last 73.03) - open/close 7909 (time 1710102142.31 total 90.05 last 66.87) - open/close 8621 (time 1710102152.31 total 100.06 last 71.17) - open/close 9271 (time 1710102162.32 total 110.07 last 64.93) - open/close 9911 (time 1710102172.34 total 120.08 last 63.91) total: 10000 open/close in 121.45 seconds: 82.34 ops/second - unlinked 0 (time 1710102175 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1150 (time 1710102267.18 total 10.01 last 114.92) - open/close 2162 (time 1710102277.19 total 20.01 last 101.19) - open/close 3066 (time 1710102287.19 total 30.01 last 90.36) - open/close 3689 (time 1710102304.66 total 47.49 last 35.65) - open/close 4588 (time 1710102314.67 total 57.50 last 89.81) - open/close 5491 (time 1710102324.68 total 67.50 last 90.24) - open/close 6255 (time 1710102334.70 total 77.53 last 76.22) - open/close 7033 (time 1710102344.72 total 87.55 last 77.64) - open/close 7796 (time 1710102354.76 total 97.58 last 76.06) - open/close 8505 (time 1710102364.76 total 107.58 last 70.88) - open/close 9208 (time 1710102374.77 total 117.59 last 70.25) - open/close 9890 (time 1710102384.78 total 127.60 last 68.11) total: 10000 open/close in 129.19 seconds: 77.41 ops/second - unlinked 0 (time 1710102388 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1186 (time 1710102478.93 total 10.00 last 118.59) - open/close 2276 (time 1710102488.95 total 20.02 last 108.84) - open/close 3256 (time 1710102498.95 total 30.02 last 97.97) - open/close 4226 (time 1710102508.95 total 40.02 last 96.98) - open/close 5117 (time 1710102518.96 total 50.03 last 89.05) - open/close 5950 (time 1710102528.96 total 60.03 last 83.29) - open/close 6784 (time 1710102538.97 total 70.04 last 83.34) - open/close 7588 (time 1710102548.97 total 80.04 last 80.36) - open/close 8268 (time 1710102559.03 total 90.10 last 67.59) - open/close 8999 (time 1710102569.04 total 100.11 last 73.05) - open/close 9689 (time 1710102579.04 total 110.11 last 68.99) total: 10000 open/close in 114.89 seconds: 87.04 ops/second - unlinked 0 (time 1710102585 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1185 (time 1710102676.01 total 10.00 last 118.44) - open/close 2256 (time 1710102686.01 total 20.01 last 107.09) - open/close 3324 (time 1710102696.02 total 30.02 last 106.69) - open/close 4287 (time 1710102706.03 total 40.02 last 96.22) - open/close 5123 (time 1710102716.03 total 50.03 last 83.54) - open/close 5962 (time 1710102726.04 total 60.04 last 83.81) - open/close 6324 (time 1710102741.48 total 75.48 last 23.45) - open/close 7111 (time 1710102751.49 total 85.49 last 78.59) - open/close 7877 (time 1710102761.50 total 95.50 last 76.50) - open/close 8696 (time 1710102771.51 total 105.51 last 81.86) - open/close 9453 (time 1710102781.52 total 115.52 last 75.60) total: 10000 open/close in 122.74 seconds: 81.47 ops/second - unlinked 0 (time 1710102790 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.147@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.147@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre On OST0, 7277 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server unloading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2