== conf-sanity test 69: replace an OST with the same index ========================================================== 17:34:16 (1751319256) start mds service on oleg435-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_size: failed to read coresize from /sys/module/intel_rapl libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 3840 (time 1751319300.75 total 10.00 last 383.96) - open/close 7801 (time 1751319310.75 total 20.00 last 396.05) total: 10000 open/close in 24.12 seconds: 414.65 ops/second - unlinked 0 (time 1751319315 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5018 (time 1751319337.41 total 10.00 last 501.74) - open/close 9996 (time 1751319347.41 total 20.00 last 497.80) total: 10000 open/close in 20.01 seconds: 499.83 ops/second - unlinked 0 (time 1751319348 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4486 (time 1751319370.03 total 10.00 last 448.56) - open/close 8986 (time 1751319380.03 total 20.00 last 449.95) total: 10000 open/close in 22.21 seconds: 450.25 ops/second - unlinked 0 (time 1751319383 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4956 (time 1751319408.00 total 10.00 last 495.51) total: 10000 open/close in 19.17 seconds: 521.64 ops/second - unlinked 0 (time 1751319418 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 6005 (time 1751319438.86 total 10.00 last 600.49) total: 10000 open/close in 17.68 seconds: 565.66 ops/second - unlinked 0 (time 1751319447 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 5047 (time 1751319469.71 total 10.00 last 504.70) - open/close 9351 (time 1751319479.71 total 20.00 last 430.38) total: 10000 open/close in 21.60 seconds: 463.07 ops/second - unlinked 0 (time 1751319482 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3770 (time 1751319508.25 total 10.00 last 376.98) - open/close 7510 (time 1751319518.25 total 20.00 last 373.95) total: 10000 open/close in 27.58 seconds: 362.59 ops/second - unlinked 0 (time 1751319527 ; total 1 ; last 1) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3376 (time 1751319552.78 total 10.00 last 337.56) - open/close 6649 (time 1751319562.78 total 20.00 last 327.27) total: 10000 open/close in 28.85 seconds: 346.66 ops/second - unlinked 0 (time 1751319572 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 3155 (time 1751319602.24 total 10.00 last 315.40) - open/close 6074 (time 1751319612.24 total 20.01 last 291.81) - open/close 9842 (time 1751319622.24 total 30.01 last 376.75) total: 10000 open/close in 30.51 seconds: 327.81 ops/second - unlinked 0 (time 1751319623 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3379 (time 1751319650.78 total 10.00 last 337.80) - open/close 6457 (time 1751319660.78 total 20.01 last 307.72) - open/close 9498 (time 1751319670.78 total 30.01 last 304.05) total: 10000 open/close in 31.61 seconds: 316.34 ops/second - unlinked 0 (time 1751319673 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg435-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server LNET ready to unload unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: LNET ready to unload modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2