== conf-sanity test 69: replace an OST with the same index ========================================================== 18:41:39 (1736984499) start mds service on oleg406-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock oleg406-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1401 (time 1736984550.96 total 10.00 last 140.06) - open/close 2861 (time 1736984560.96 total 20.01 last 145.89) - open/close 4410 (time 1736984570.96 total 30.01 last 154.88) - open/close 6143 (time 1736984580.97 total 40.01 last 173.26) - open/close 8486 (time 1736984590.97 total 50.02 last 234.27) total: 10000 open/close in 57.05 seconds: 175.27 ops/second - unlinked 0 (time 1736984599 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2092 (time 1736984635.37 total 10.00 last 209.13) - open/close 4069 (time 1736984645.37 total 20.00 last 197.67) - open/close 6047 (time 1736984655.38 total 30.01 last 197.73) - open/close 8064 (time 1736984665.38 total 40.01 last 201.67) total: 10000 open/close in 48.68 seconds: 205.41 ops/second - unlinked 0 (time 1736984675 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2285 (time 1736984708.69 total 10.00 last 228.50) - open/close 4797 (time 1736984718.69 total 20.00 last 251.17) - open/close 6988 (time 1736984728.69 total 30.00 last 219.10) - open/close 9484 (time 1736984738.69 total 40.01 last 249.51) total: 10000 open/close in 42.28 seconds: 236.52 ops/second - unlinked 0 (time 1736984742 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1128 (time 1736984786.16 total 10.00 last 112.77) - open/close 2626 (time 1736984796.16 total 20.01 last 149.76) - open/close 4195 (time 1736984806.17 total 30.01 last 156.82) - open/close 5821 (time 1736984816.17 total 40.01 last 162.53) - open/close 7349 (time 1736984826.18 total 50.02 last 152.75) - open/close 8569 (time 1736984836.18 total 60.02 last 121.99) total: 10000 open/close in 69.96 seconds: 142.93 ops/second - unlinked 0 (time 1736984847 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1682 (time 1736984897.90 total 10.00 last 168.19) - open/close 3680 (time 1736984907.90 total 20.00 last 199.74) - open/close 4990 (time 1736984917.91 total 30.01 last 130.95) - open/close 6679 (time 1736984927.91 total 40.01 last 168.86) - open/close 8227 (time 1736984937.91 total 50.01 last 154.79) total: 10000 open/close in 59.08 seconds: 169.27 ops/second - unlinked 0 (time 1736984948 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1438 (time 1736984998.20 total 10.00 last 143.73) - open/close 3015 (time 1736985008.20 total 20.01 last 157.68) - open/close 4850 (time 1736985018.20 total 30.01 last 183.46) - open/close 6481 (time 1736985028.20 total 40.01 last 163.06) - open/close 8159 (time 1736985038.21 total 50.01 last 167.74) - open/close 9919 (time 1736985048.21 total 60.02 last 175.89) total: 10000 open/close in 60.43 seconds: 165.47 ops/second - unlinked 0 (time 1736985050 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2019 (time 1736985089.09 total 10.00 last 201.81) - open/close 4206 (time 1736985099.10 total 20.01 last 218.59) - open/close 6198 (time 1736985109.10 total 30.01 last 199.17) - open/close 8362 (time 1736985119.10 total 40.01 last 216.32) total: 10000 open/close in 48.42 seconds: 206.51 ops/second - unlinked 0 (time 1736985128 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1705 (time 1736985172.91 total 10.00 last 170.45) - open/close 3425 (time 1736985182.91 total 20.00 last 171.97) - open/close 5478 (time 1736985192.91 total 30.01 last 205.23) - open/close 7668 (time 1736985202.92 total 40.01 last 218.99) - open/close 9514 (time 1736985212.92 total 50.01 last 184.59) total: 10000 open/close in 52.16 seconds: 191.72 ops/second - unlinked 0 (time 1736985216 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2082 (time 1736985254.46 total 10.00 last 208.14) - open/close 4294 (time 1736985264.46 total 20.00 last 221.18) - open/close 6704 (time 1736985274.46 total 30.01 last 240.97) - open/close 8863 (time 1736985284.47 total 40.01 last 215.87) total: 10000 open/close in 45.45 seconds: 220.02 ops/second - unlinked 0 (time 1736985291 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1889 (time 1736985328.23 total 10.01 last 188.73) - open/close 3764 (time 1736985338.24 total 20.01 last 187.48) - open/close 5951 (time 1736985348.24 total 30.01 last 218.66) - open/close 8181 (time 1736985358.24 total 40.01 last 222.96) total: 10000 open/close in 49.07 seconds: 203.81 ops/second - unlinked 0 (time 1736985368 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.106@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.106@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg406-server: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg406-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock oleg406-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server LNET ready to unload unloading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: LNET ready to unload modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2