== conf-sanity test 69: replace an OST with the same index ========================================================== 07:14:56 (1750590896) start mds service on oleg258-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing load_modules_local oleg258-server: Loading modules from /home/green/git/lustre-release/lustre oleg258-server: detected 4 online CPUs by sysfs oleg258-server: MODOPTS_LIBCFS= oleg258-server: Force libcfs to create 2 CPU partitions oleg258-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg258-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg258-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1502 (time 1750590943.95 total 10.00 last 150.19) - open/close 2990 (time 1750590953.95 total 20.00 last 148.80) - open/close 4494 (time 1750590963.95 total 30.00 last 150.37) - open/close 5977 (time 1750590973.95 total 40.01 last 148.26) - open/close 7448 (time 1750590983.95 total 50.01 last 147.10) - open/close 9051 (time 1750590993.96 total 60.01 last 160.24) total: 10000 open/close in 66.30 seconds: 150.84 ops/second - unlinked 0 (time 1750591001 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1660 (time 1750591048.97 total 10.00 last 166.00) - open/close 3122 (time 1750591058.98 total 20.00 last 146.16) - open/close 4759 (time 1750591068.98 total 30.01 last 163.66) - open/close 6296 (time 1750591078.98 total 40.01 last 153.68) - open/close 7855 (time 1750591088.98 total 50.01 last 155.89) - open/close 9425 (time 1750591098.98 total 60.01 last 156.92) total: 10000 open/close in 63.28 seconds: 158.03 ops/second - unlinked 0 (time 1750591103 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1413 (time 1750591153.15 total 10.00 last 141.29) - open/close 2804 (time 1750591163.17 total 20.02 last 138.84) - open/close 4212 (time 1750591173.17 total 30.02 last 140.75) - open/close 5597 (time 1750591183.17 total 40.02 last 138.48) - open/close 7026 (time 1750591193.17 total 50.03 last 142.87) - open/close 8484 (time 1750591203.18 total 60.03 last 145.73) total: 10000 open/close in 69.26 seconds: 144.38 ops/second - unlinked 0 (time 1750591213 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1796 (time 1750591256.70 total 10.00 last 179.51) - open/close 3327 (time 1750591266.70 total 20.01 last 153.09) - open/close 4939 (time 1750591276.70 total 30.01 last 161.14) - open/close 6471 (time 1750591286.70 total 40.01 last 153.17) - open/close 7904 (time 1750591296.70 total 50.01 last 143.27) - open/close 9693 (time 1750591306.71 total 60.02 last 178.83) total: 10000 open/close in 62.10 seconds: 161.04 ops/second - unlinked 0 (time 1750591310 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1796 (time 1750591360.42 total 10.00 last 179.53) - open/close 3339 (time 1750591370.42 total 20.01 last 154.24) - open/close 4843 (time 1750591380.42 total 30.01 last 150.40) - open/close 6371 (time 1750591390.42 total 40.01 last 152.78) - open/close 8180 (time 1750591400.43 total 50.02 last 180.78) - open/close 9931 (time 1750591410.43 total 60.02 last 175.06) total: 10000 open/close in 60.39 seconds: 165.60 ops/second - unlinked 0 (time 1750591412 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1466 (time 1750591457.05 total 10.00 last 146.54) - open/close 2896 (time 1750591467.05 total 20.01 last 142.97) - open/close 4366 (time 1750591477.05 total 30.01 last 146.93) - open/close 6003 (time 1750591487.06 total 40.02 last 163.61) - open/close 7438 (time 1750591497.06 total 50.02 last 143.45) - open/close 8723 (time 1750591507.06 total 60.02 last 128.46) total: 10000 open/close in 69.60 seconds: 143.68 ops/second - unlinked 0 (time 1750591518 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1333 (time 1750591576.38 total 10.00 last 133.29) - open/close 2731 (time 1750591586.38 total 20.00 last 139.80) - open/close 4145 (time 1750591596.38 total 30.00 last 141.38) - open/close 5527 (time 1750591606.39 total 40.01 last 138.12) - open/close 6894 (time 1750591616.39 total 50.01 last 136.65) - open/close 8238 (time 1750591626.39 total 60.01 last 134.38) - open/close 9534 (time 1750591636.39 total 70.02 last 129.55) total: 10000 open/close in 73.40 seconds: 136.25 ops/second - unlinked 0 (time 1750591641 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1485 (time 1750591690.50 total 10.01 last 148.41) - open/close 2792 (time 1750591700.51 total 20.01 last 130.65) - open/close 4203 (time 1750591710.51 total 30.01 last 141.08) - open/close 5590 (time 1750591720.51 total 40.01 last 138.67) - open/close 7012 (time 1750591730.52 total 50.02 last 142.10) - open/close 8452 (time 1750591740.52 total 60.02 last 143.99) - open/close 9960 (time 1750591750.52 total 70.02 last 150.76) total: 10000 open/close in 70.29 seconds: 142.26 ops/second - unlinked 0 (time 1750591752 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 2098 (time 1750591797.90 total 10.00 last 209.75) - open/close 4160 (time 1750591807.90 total 20.00 last 206.18) - open/close 6371 (time 1750591817.90 total 30.01 last 221.04) - open/close 8686 (time 1750591827.90 total 40.01 last 231.50) total: 10000 open/close in 45.66 seconds: 219.01 ops/second - unlinked 0 (time 1750591834 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1633 (time 1750591882.44 total 10.00 last 163.24) - open/close 3359 (time 1750591892.44 total 20.00 last 172.59) - open/close 4860 (time 1750591902.45 total 30.01 last 150.02) - open/close 6468 (time 1750591912.45 total 40.01 last 160.77) - open/close 8112 (time 1750591922.45 total 50.02 last 164.34) - open/close 9967 (time 1750591932.46 total 60.02 last 185.41) total: 10000 open/close in 60.18 seconds: 166.16 ops/second - unlinked 0 (time 1750591933 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.158@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.158@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg258-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug -1 all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg258-server: oleg258-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg258-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre On OST0, 17685 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg258-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg258-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg258-server stop mds service on oleg258-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg258-server unloading modules via unload_modules_local on: 'oleg258-server' oleg258-server: oleg258-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-client: ssh exited with exit code 2 pdsh@oleg258-client: oleg258-server: ssh exited with exit code 2