== conf-sanity test 69: replace an OST with the same index ========================================================== 18:17:54 (1736983074) start mds service on oleg219-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1382 (time 1736983127.21 total 10.00 last 138.14) - open/close 2549 (time 1736983137.24 total 20.03 last 116.37) - open/close 3711 (time 1736983147.24 total 30.04 last 116.16) - open/close 4915 (time 1736983157.24 total 40.04 last 120.40) - open/close 6247 (time 1736983167.24 total 50.04 last 133.17) - open/close 7558 (time 1736983177.25 total 60.04 last 131.04) - open/close 8844 (time 1736983187.26 total 70.05 last 128.51) total: 10000 open/close in 78.42 seconds: 127.52 ops/second - unlinked 0 (time 1736983196 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 3858 (time 1736983238.94 total 10.00 last 385.78) - open/close 6885 (time 1736983248.94 total 20.00 last 302.65) - open/close 9095 (time 1736983258.94 total 30.00 last 220.98) total: 10000 open/close in 34.39 seconds: 290.82 ops/second - unlinked 0 (time 1736983264 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 3069 (time 1736983299.73 total 10.00 last 306.83) - open/close 6097 (time 1736983309.73 total 20.00 last 302.77) - open/close 9312 (time 1736983319.73 total 30.01 last 321.40) total: 10000 open/close in 32.04 seconds: 312.14 ops/second - unlinked 0 (time 1736983322 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3272 (time 1736983348.16 total 10.00 last 327.13) - open/close 6175 (time 1736983358.16 total 20.00 last 290.28) - open/close 9447 (time 1736983368.16 total 30.00 last 327.14) total: 10000 open/close in 31.57 seconds: 316.80 ops/second - unlinked 0 (time 1736983370 ; total 0 ; last 0) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 3945 (time 1736983396.24 total 10.00 last 394.44) - open/close 8306 (time 1736983406.24 total 20.00 last 436.02) total: 10000 open/close in 25.10 seconds: 398.34 ops/second - unlinked 0 (time 1736983412 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4877 (time 1736983436.67 total 10.00 last 487.65) - open/close 9595 (time 1736983446.67 total 20.00 last 471.77) total: 10000 open/close in 21.09 seconds: 474.25 ops/second - unlinked 0 (time 1736983448 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4229 (time 1736983471.84 total 10.00 last 422.89) - open/close 8160 (time 1736983481.84 total 20.00 last 392.97) total: 10000 open/close in 24.83 seconds: 402.79 ops/second - unlinked 0 (time 1736983487 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4942 (time 1736983510.95 total 10.00 last 494.17) - open/close 9770 (time 1736983520.95 total 20.00 last 482.74) total: 10000 open/close in 20.53 seconds: 487.10 ops/second - unlinked 0 (time 1736983522 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4022 (time 1736983544.75 total 10.00 last 402.13) - open/close 9553 (time 1736983554.75 total 20.00 last 553.03) total: 10000 open/close in 20.87 seconds: 479.26 ops/second - unlinked 0 (time 1736983556 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second - open/close 4630 (time 1736983578.30 total 10.00 last 462.96) total: 10000 open/close in 19.13 seconds: 522.70 ops/second - unlinked 0 (time 1736983588 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.119@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg219-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre On OST0, 291 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server LNET ready to unload unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local oleg219-server: LNET ready to unload modules unloaded. pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2