== conf-sanity test 69: replace an OST with the same index ========================================================== 17:16:47 (1715203007) start mds service on oleg238-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: detected 4 online CPUs by sysfs oleg238-server: Force libcfs to create 2 CPU partitions oleg238-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg238-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg238-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 5530 (time 1715203038.06 total 10.00 last 552.94) - open/close 9529 (time 1715203048.07 total 20.01 last 399.68) total: 10000 open/close in 23.04 seconds: 434.06 ops/second - unlinked 0 (time 1715203052 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 2522 (time 1715203091.85 total 10.00 last 252.20) - open/close 4767 (time 1715203101.86 total 20.00 last 224.41) - open/close 6244 (time 1715203111.86 total 30.01 last 147.67) - open/close 8316 (time 1715203121.86 total 40.01 last 207.16) total: 10000 open/close in 48.03 seconds: 208.21 ops/second - unlinked 0 (time 1715203130 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3461 (time 1715203159.14 total 10.00 last 346.10) - open/close 6080 (time 1715203169.14 total 20.00 last 261.89) - open/close 8006 (time 1715203179.14 total 30.01 last 192.51) total: 10000 open/close in 39.81 seconds: 251.22 ops/second - unlinked 0 (time 1715203190 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1969 (time 1715203226.35 total 10.00 last 196.87) - open/close 4135 (time 1715203236.36 total 20.00 last 216.58) - open/close 6484 (time 1715203246.36 total 30.00 last 234.89) - open/close 8707 (time 1715203256.36 total 40.00 last 222.30) total: 10000 open/close in 46.62 seconds: 214.52 ops/second - unlinked 0 (time 1715203264 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2363 (time 1715203300.01 total 10.00 last 236.26) - open/close 4925 (time 1715203310.01 total 20.00 last 256.14) - open/close 7033 (time 1715203320.01 total 30.01 last 210.75) - open/close 9847 (time 1715203330.01 total 40.01 last 281.37) total: 10000 open/close in 40.53 seconds: 246.70 ops/second - unlinked 0 (time 1715203331 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2823 (time 1715203364.60 total 10.00 last 282.27) - open/close 5781 (time 1715203374.60 total 20.00 last 295.75) - open/close 8966 (time 1715203384.60 total 30.00 last 318.45) total: 10000 open/close in 33.48 seconds: 298.73 ops/second - unlinked 0 (time 1715203389 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3354 (time 1715203418.69 total 10.00 last 335.34) - open/close 6434 (time 1715203428.69 total 20.00 last 307.96) - open/close 9418 (time 1715203438.69 total 30.00 last 298.39) total: 10000 open/close in 31.76 seconds: 314.87 ops/second - unlinked 0 (time 1715203441 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 4814 (time 1715203470.28 total 10.00 last 481.35) - open/close 9895 (time 1715203480.28 total 20.00 last 508.05) total: 10000 open/close in 20.23 seconds: 494.40 ops/second - unlinked 0 (time 1715203481 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 5026 (time 1715203504.57 total 10.00 last 502.54) - open/close 9947 (time 1715203514.58 total 20.00 last 492.01) total: 10000 open/close in 20.14 seconds: 496.53 ops/second - unlinked 0 (time 1715203515 ; total 0 ; last 0) total: 10000 unlinks in 10 seconds: 1000.000000 unlinks/second - open/close 5746 (time 1715203536.44 total 10.00 last 574.46) total: 10000 open/close in 18.97 seconds: 527.18 ops/second - unlinked 0 (time 1715203546 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.138@tcp sys.timeout=20 mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.138@tcp lustre:sys.timeout=20 lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg238-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg238-server: oleg238-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg238-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock oleg238-server@tcp:/lustre /mnt/lustre On OST0, 9005 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server LNET ready to unload unloading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: LNET ready to unload modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2