== conf-sanity test 69: replace an OST with the same index ========================================================== 11:02:04 (1712761324) start mds service on oleg251-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg251-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 524 (time 1712761367.01 total 13.06 last 40.13) - open/close 4872 (time 1712761377.01 total 23.06 last 434.72) - open/close 8432 (time 1712761387.01 total 33.06 last 355.97) total: 10000 open/close in 37.21 seconds: 268.74 ops/second - unlinked 0 (time 1712761392 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3386 (time 1712761417.93 total 10.00 last 338.57) - open/close 6674 (time 1712761427.93 total 20.00 last 328.77) total: 10000 open/close in 29.11 seconds: 343.56 ops/second - unlinked 0 (time 1712761437 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2665 (time 1712761466.34 total 10.00 last 266.45) - open/close 5112 (time 1712761476.35 total 20.01 last 244.59) - open/close 8791 (time 1712761486.35 total 30.01 last 367.85) total: 10000 open/close in 32.99 seconds: 303.11 ops/second - unlinked 0 (time 1712761490 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3854 (time 1712761517.10 total 10.00 last 385.40) - open/close 7733 (time 1712761527.10 total 20.00 last 387.88) total: 10000 open/close in 26.78 seconds: 373.43 ops/second - unlinked 0 (time 1712761534 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3204 (time 1712761562.88 total 10.01 last 320.22) - open/close 6590 (time 1712761572.88 total 20.01 last 338.56) total: 10000 open/close in 28.98 seconds: 345.06 ops/second - unlinked 0 (time 1712761582 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3817 (time 1712761610.73 total 10.00 last 381.68) - open/close 7263 (time 1712761620.74 total 20.00 last 344.50) total: 10000 open/close in 27.57 seconds: 362.70 ops/second - unlinked 0 (time 1712761629 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4768 (time 1712761653.18 total 10.00 last 476.69) - open/close 8750 (time 1712761663.18 total 20.00 last 398.17) total: 10000 open/close in 23.39 seconds: 427.59 ops/second - unlinked 0 (time 1712761667 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4600 (time 1712761691.92 total 10.00 last 459.92) - open/close 8947 (time 1712761701.92 total 20.00 last 434.60) total: 10000 open/close in 22.61 seconds: 442.25 ops/second - unlinked 0 (time 1712761705 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3687 (time 1712761732.30 total 10.00 last 368.58) - open/close 7401 (time 1712761742.30 total 20.01 last 371.30) total: 10000 open/close in 26.11 seconds: 382.93 ops/second - unlinked 0 (time 1712761749 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4742 (time 1712761774.25 total 10.00 last 474.17) - open/close 8992 (time 1712761784.25 total 20.00 last 424.93) total: 10000 open/close in 22.40 seconds: 446.37 ops/second - unlinked 0 (time 1712761787 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.151@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.151@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg251-server: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg251-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre On OST0, 18069 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2