== conf-sanity test 69: replace an OST with the same index ========================================================== 17:11:42 (1747257102) start mds service on oleg112-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg112-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2354 (time 1747257146.01 total 10.00 last 235.39) - open/close 4852 (time 1747257156.01 total 20.00 last 249.79) - open/close 7686 (time 1747257166.01 total 30.00 last 283.35) total: 10000 open/close in 39.49 seconds: 253.23 ops/second - unlinked 0 (time 1747257176 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 2791 (time 1747257204.17 total 10.00 last 279.08) - open/close 4998 (time 1747257214.17 total 20.00 last 220.66) - open/close 6997 (time 1747257224.17 total 30.00 last 199.85) - open/close 9237 (time 1747257234.17 total 40.01 last 223.99) total: 10000 open/close in 42.79 seconds: 233.72 ops/second - unlinked 0 (time 1747257237 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2544 (time 1747257274.92 total 10.00 last 254.32) - open/close 5261 (time 1747257284.93 total 20.01 last 271.64) - open/close 7339 (time 1747257294.93 total 30.01 last 207.74) - open/close 9542 (time 1747257304.93 total 40.01 last 220.27) total: 10000 open/close in 42.23 seconds: 236.78 ops/second - unlinked 0 (time 1747257308 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2129 (time 1747257337.78 total 10.00 last 212.89) - open/close 4102 (time 1747257347.78 total 20.00 last 197.28) - open/close 6210 (time 1747257357.78 total 30.00 last 210.80) - open/close 8552 (time 1747257367.78 total 40.00 last 234.15) total: 10000 open/close in 45.23 seconds: 221.10 ops/second - unlinked 0 (time 1747257374 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2672 (time 1747257405.23 total 10.00 last 267.16) - open/close 5541 (time 1747257415.23 total 20.00 last 286.85) - open/close 8496 (time 1747257425.24 total 30.01 last 295.42) total: 10000 open/close in 35.65 seconds: 280.50 ops/second - unlinked 0 (time 1747257431 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2556 (time 1747257460.92 total 10.00 last 255.55) - open/close 5173 (time 1747257470.92 total 20.00 last 261.65) - open/close 8075 (time 1747257480.93 total 30.01 last 290.06) total: 10000 open/close in 37.05 seconds: 269.92 ops/second - unlinked 0 (time 1747257489 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 2972 (time 1747257517.91 total 10.00 last 297.17) - open/close 5923 (time 1747257527.91 total 20.00 last 295.06) - open/close 8615 (time 1747257537.91 total 30.00 last 269.15) total: 10000 open/close in 34.80 seconds: 287.37 ops/second - unlinked 0 (time 1747257543 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3689 (time 1747257570.04 total 10.00 last 368.88) - open/close 6929 (time 1747257580.04 total 20.00 last 323.91) total: 10000 open/close in 29.59 seconds: 337.94 ops/second - unlinked 0 (time 1747257590 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3566 (time 1747257616.34 total 10.00 last 356.60) - open/close 7044 (time 1747257626.34 total 20.00 last 347.79) total: 10000 open/close in 29.62 seconds: 337.64 ops/second - unlinked 0 (time 1747257636 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4375 (time 1747257660.38 total 10.00 last 437.50) - open/close 8209 (time 1747257670.38 total 20.00 last 383.38) total: 10000 open/close in 24.58 seconds: 406.86 ops/second - unlinked 0 (time 1747257675 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.112@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.112@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg112-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg112-server: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg112-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock oleg112-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server LNET ready to unload unloading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: LNET ready to unload modules unloaded. pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-client: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2