== conf-sanity test 69: replace an OST with the same index ========================================================== 16:20:56 (1746130856) start mds service on oleg440-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1968 (time 1746130898.94 total 10.01 last 196.69) - open/close 3804 (time 1746130908.94 total 20.01 last 183.45) - open/close 5797 (time 1746130918.95 total 30.02 last 199.17) - open/close 7881 (time 1746130928.95 total 40.02 last 208.33) - open/close 9835 (time 1746130938.96 total 50.03 last 195.34) total: 10000 open/close in 50.82 seconds: 196.75 ops/second - unlinked 0 (time 1746130941 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1557 (time 1746130990.98 total 10.00 last 155.65) - open/close 3540 (time 1746131000.99 total 20.01 last 198.09) - open/close 5587 (time 1746131010.99 total 30.02 last 204.60) - open/close 8185 (time 1746131021.00 total 40.02 last 259.68) total: 10000 open/close in 49.47 seconds: 202.13 ops/second - unlinked 0 (time 1746131031 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2505 (time 1746131065.67 total 10.00 last 250.45) - open/close 4759 (time 1746131075.67 total 20.00 last 225.37) - open/close 6725 (time 1746131085.67 total 30.01 last 196.51) - open/close 8968 (time 1746131095.67 total 40.01 last 224.27) total: 10000 open/close in 46.08 seconds: 216.99 ops/second - unlinked 0 (time 1746131103 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1905 (time 1746131150.97 total 10.01 last 190.28) - open/close 3370 (time 1746131160.97 total 20.01 last 146.48) - open/close 5476 (time 1746131170.97 total 30.02 last 210.55) - open/close 7473 (time 1746131180.97 total 40.02 last 199.64) - open/close 8679 (time 1746131190.98 total 50.02 last 120.57) total: 10000 open/close in 59.43 seconds: 168.27 ops/second - unlinked 0 (time 1746131201 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 2284 (time 1746131247.00 total 10.00 last 228.31) - open/close 4259 (time 1746131257.00 total 20.01 last 197.48) - open/close 6542 (time 1746131267.01 total 30.01 last 228.24) - open/close 9698 (time 1746131277.01 total 40.01 last 315.43) total: 10000 open/close in 41.43 seconds: 241.36 ops/second - unlinked 0 (time 1746131279 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1545 (time 1746131325.90 total 10.00 last 154.47) - open/close 3405 (time 1746131335.90 total 20.00 last 185.96) - open/close 5091 (time 1746131345.90 total 30.00 last 168.58) - open/close 7131 (time 1746131355.90 total 40.01 last 204.00) - open/close 9415 (time 1746131365.90 total 50.01 last 228.38) total: 10000 open/close in 51.61 seconds: 193.75 ops/second - unlinked 0 (time 1746131368 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 2135 (time 1746131412.17 total 10.01 last 213.31) - open/close 4815 (time 1746131422.18 total 20.02 last 267.81) - open/close 6330 (time 1746131432.18 total 30.02 last 151.46) - open/close 8544 (time 1746131442.18 total 40.02 last 221.34) total: 10000 open/close in 47.44 seconds: 210.79 ops/second - unlinked 0 (time 1746131450 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1195 (time 1746131496.73 total 10.00 last 119.47) - open/close 2182 (time 1746131506.74 total 20.01 last 98.65) - open/close 3248 (time 1746131516.74 total 30.01 last 106.56) - open/close 4358 (time 1746131526.74 total 40.01 last 110.99) - open/close 5329 (time 1746131536.75 total 50.02 last 97.03) - open/close 6498 (time 1746131546.76 total 60.02 last 116.85) - open/close 8110 (time 1746131556.76 total 70.03 last 161.18) - open/close 9332 (time 1746131566.76 total 80.03 last 122.18) total: 10000 open/close in 84.42 seconds: 118.45 ops/second - unlinked 0 (time 1746131572 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1148 (time 1746131614.69 total 10.00 last 114.76) - open/close 2403 (time 1746131624.70 total 20.01 last 125.45) - open/close 3570 (time 1746131634.70 total 30.01 last 116.61) - open/close 4818 (time 1746131644.71 total 40.02 last 124.77) - open/close 6242 (time 1746131654.72 total 50.03 last 142.28) - open/close 7599 (time 1746131664.72 total 60.03 last 135.69) - open/close 8808 (time 1746131674.73 total 70.04 last 120.72) total: 10000 open/close in 79.81 seconds: 125.29 ops/second - unlinked 0 (time 1746131686 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 928 (time 1746131750.48 total 10.02 last 92.65) - open/close 1951 (time 1746131760.48 total 20.02 last 102.27) - open/close 3059 (time 1746131770.49 total 30.03 last 110.72) - open/close 4102 (time 1746131780.50 total 40.04 last 104.15) - open/close 5022 (time 1746131790.51 total 50.05 last 91.92) - open/close 5993 (time 1746131800.52 total 60.05 last 97.08) - open/close 6888 (time 1746131810.52 total 70.06 last 89.48) - open/close 7926 (time 1746131820.52 total 80.06 last 103.75) - open/close 9118 (time 1746131830.53 total 90.07 last 119.13) total: 10000 open/close in 98.86 seconds: 101.15 ops/second - unlinked 0 (time 1746131841 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.140@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2