== conf-sanity test 69: replace an OST with the same index ========================================================== 20:39:17 (1746146357) start mds service on oleg332-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1831 (time 1746146396.14 total 10.00 last 183.06) - open/close 3401 (time 1746146406.15 total 20.01 last 156.94) - open/close 5237 (time 1746146416.15 total 30.01 last 183.53) - open/close 7022 (time 1746146426.15 total 40.01 last 178.46) - open/close 8818 (time 1746146436.15 total 50.01 last 179.57) total: 10000 open/close in 55.13 seconds: 181.39 ops/second - unlinked 0 (time 1746146442 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 3054 (time 1746146479.49 total 10.00 last 305.34) - open/close 5538 (time 1746146489.49 total 20.00 last 248.39) - open/close 7592 (time 1746146499.49 total 30.00 last 205.37) total: 10000 open/close in 39.89 seconds: 250.67 ops/second - unlinked 0 (time 1746146510 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2271 (time 1746146552.85 total 10.00 last 227.08) - open/close 4987 (time 1746146562.85 total 20.00 last 271.50) - open/close 7608 (time 1746146572.85 total 30.01 last 262.05) total: 10000 open/close in 38.86 seconds: 257.36 ops/second - unlinked 0 (time 1746146582 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 2781 (time 1746146609.36 total 10.00 last 278.06) - open/close 6417 (time 1746146619.36 total 20.00 last 363.58) - open/close 8635 (time 1746146629.36 total 30.00 last 221.75) total: 10000 open/close in 36.43 seconds: 274.49 ops/second - unlinked 0 (time 1746146637 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1765 (time 1746146679.12 total 10.01 last 176.39) - open/close 4566 (time 1746146689.12 total 20.01 last 280.09) - open/close 7043 (time 1746146699.12 total 30.01 last 247.68) total: 10000 open/close in 39.12 seconds: 255.63 ops/second - unlinked 0 (time 1746146709 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3403 (time 1746146740.06 total 10.00 last 340.22) - open/close 6530 (time 1746146750.06 total 20.00 last 312.68) - open/close 8657 (time 1746146760.06 total 30.00 last 212.68) total: 10000 open/close in 36.58 seconds: 273.39 ops/second - unlinked 0 (time 1746146767 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2949 (time 1746146798.82 total 10.00 last 294.88) - open/close 5265 (time 1746146808.82 total 20.00 last 231.58) - open/close 7845 (time 1746146818.82 total 30.00 last 257.99) - open/close 9646 (time 1746146828.83 total 40.01 last 179.94) total: 10000 open/close in 42.47 seconds: 235.49 ops/second - unlinked 0 (time 1746146832 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2448 (time 1746146870.62 total 10.00 last 244.78) - open/close 4472 (time 1746146880.63 total 20.00 last 202.32) - open/close 5894 (time 1746146890.63 total 30.00 last 142.20) - open/close 7693 (time 1746146900.63 total 40.01 last 179.86) - open/close 9158 (time 1746146910.63 total 50.01 last 146.41) total: 10000 open/close in 55.54 seconds: 180.06 ops/second - unlinked 0 (time 1746146917 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2491 (time 1746146953.12 total 10.00 last 249.07) - open/close 4746 (time 1746146963.12 total 20.01 last 225.38) - open/close 7232 (time 1746146973.13 total 30.01 last 248.57) - open/close 9417 (time 1746146983.13 total 40.01 last 218.49) total: 10000 open/close in 43.30 seconds: 230.95 ops/second - unlinked 0 (time 1746146987 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1637 (time 1746147032.58 total 10.00 last 163.67) - open/close 3413 (time 1746147042.59 total 20.01 last 177.53) - open/close 4961 (time 1746147052.59 total 30.01 last 154.77) - open/close 6274 (time 1746147062.59 total 40.01 last 131.23) - open/close 7714 (time 1746147072.60 total 50.02 last 143.95) - open/close 9052 (time 1746147082.60 total 60.02 last 133.73) total: 10000 open/close in 68.36 seconds: 146.28 ops/second - unlinked 0 (time 1746147092 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg332-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre On OST0, 4157 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2