== conf-sanity test 69: replace an OST with the same index ========================================================== 04:53:25 (1726304005) start mds service on oleg114-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: detected 4 online CPUs by sysfs oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2056 (time 1726304046.45 total 10.00 last 205.55) - open/close 4077 (time 1726304056.45 total 20.01 last 201.98) - open/close 6335 (time 1726304066.45 total 30.01 last 225.72) - open/close 8835 (time 1726304076.46 total 40.01 last 249.99) total: 10000 open/close in 44.27 seconds: 225.86 ops/second - unlinked 0 (time 1726304081 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2321 (time 1726304113.71 total 10.00 last 232.06) - open/close 4406 (time 1726304123.71 total 20.00 last 208.47) - open/close 6395 (time 1726304133.72 total 30.01 last 198.74) - open/close 8669 (time 1726304143.72 total 40.01 last 227.37) total: 10000 open/close in 45.12 seconds: 221.65 ops/second - unlinked 0 (time 1726304149 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2436 (time 1726304180.76 total 10.00 last 243.59) - open/close 5280 (time 1726304190.76 total 20.00 last 284.34) - open/close 8549 (time 1726304200.76 total 30.00 last 326.85) total: 10000 open/close in 34.03 seconds: 293.85 ops/second - unlinked 0 (time 1726304205 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3009 (time 1726304233.91 total 10.00 last 300.87) - open/close 6316 (time 1726304243.91 total 20.00 last 330.68) - open/close 9444 (time 1726304253.91 total 30.00 last 312.74) total: 10000 open/close in 31.77 seconds: 314.72 ops/second - unlinked 0 (time 1726304256 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3707 (time 1726304287.49 total 10.00 last 370.65) - open/close 7660 (time 1726304297.49 total 20.00 last 395.19) total: 10000 open/close in 26.44 seconds: 378.24 ops/second - unlinked 0 (time 1726304304 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3984 (time 1726304329.40 total 10.00 last 398.33) - open/close 7741 (time 1726304339.41 total 20.00 last 375.59) total: 10000 open/close in 25.90 seconds: 386.08 ops/second - unlinked 0 (time 1726304346 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 3897 (time 1726304371.71 total 10.00 last 389.68) - open/close 7806 (time 1726304381.71 total 20.00 last 390.82) total: 10000 open/close in 24.97 seconds: 400.47 ops/second - unlinked 0 (time 1726304387 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4700 (time 1726304411.66 total 10.00 last 469.93) - open/close 9183 (time 1726304421.66 total 20.00 last 448.17) total: 10000 open/close in 22.03 seconds: 453.92 ops/second - unlinked 0 (time 1726304424 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4184 (time 1726304448.00 total 10.00 last 418.40) - open/close 8658 (time 1726304458.00 total 20.00 last 447.39) total: 10000 open/close in 22.90 seconds: 436.72 ops/second - unlinked 0 (time 1726304461 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second - open/close 4837 (time 1726304484.89 total 10.00 last 483.68) - open/close 9367 (time 1726304494.89 total 20.00 last 452.96) total: 10000 open/close in 21.81 seconds: 458.41 ops/second - unlinked 0 (time 1726304497 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg114-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 15 sec mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre On OST0, 16005 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server unloading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2