== conf-sanity test 69: replace an OST with the same index ========================================================== 02:37:18 (1747809438) start mds service on oleg127-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: detected 4 online CPUs by sysfs oleg127-server: MODOPTS_LIBCFS= oleg127-server: Force libcfs to create 2 CPU partitions oleg127-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg127-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg127-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 5334 (time 1747809474.03 total 10.00 last 533.38) total: 10000 open/close in 19.10 seconds: 523.52 ops/second - unlinked 0 (time 1747809483 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5175 (time 1747809506.10 total 10.00 last 517.50) total: 10000 open/close in 19.46 seconds: 513.75 ops/second - unlinked 0 (time 1747809516 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4930 (time 1747809538.70 total 10.00 last 492.94) - open/close 9943 (time 1747809548.70 total 20.00 last 501.27) total: 10000 open/close in 20.11 seconds: 497.20 ops/second - unlinked 0 (time 1747809549 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 5110 (time 1747809572.03 total 10.00 last 510.88) total: 10000 open/close in 19.95 seconds: 501.30 ops/second - unlinked 0 (time 1747809582 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 5362 (time 1747809605.47 total 10.00 last 536.14) total: 10000 open/close in 19.12 seconds: 522.94 ops/second - unlinked 0 (time 1747809615 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 5227 (time 1747809638.08 total 10.00 last 522.65) - open/close 9972 (time 1747809648.08 total 20.00 last 474.49) total: 10000 open/close in 20.06 seconds: 498.53 ops/second - unlinked 0 (time 1747809649 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 5101 (time 1747809672.91 total 10.00 last 510.07) - open/close 9617 (time 1747809682.91 total 20.00 last 451.56) total: 10000 open/close in 20.86 seconds: 479.49 ops/second - unlinked 0 (time 1747809684 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4784 (time 1747809708.28 total 10.00 last 478.39) - open/close 9326 (time 1747809718.28 total 20.00 last 454.12) total: 10000 open/close in 21.37 seconds: 467.88 ops/second - unlinked 0 (time 1747809720 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4819 (time 1747809744.13 total 10.00 last 481.82) - open/close 9599 (time 1747809754.13 total 20.00 last 477.94) total: 10000 open/close in 20.80 seconds: 480.68 ops/second - unlinked 0 (time 1747809755 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 5166 (time 1747809778.57 total 10.00 last 516.55) total: 10000 open/close in 18.60 seconds: 537.73 ops/second - unlinked 0 (time 1747809787 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.127@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.127@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg127-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg127-server: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg127-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre On OST0, 4357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server unloading modules via unload_modules_local on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2