== conf-sanity test 69: replace an OST with the same index ========================================================== 17:09:46 (1746133786) start mds service on oleg406-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2219 (time 1746133825.47 total 10.01 last 221.77) - open/close 4060 (time 1746133835.48 total 20.01 last 184.07) - open/close 6337 (time 1746133845.48 total 30.01 last 227.64) - open/close 8722 (time 1746133855.48 total 40.01 last 238.45) total: 10000 open/close in 45.97 seconds: 217.53 ops/second - unlinked 0 (time 1746133862 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1716 (time 1746133908.07 total 10.01 last 171.49) - open/close 3197 (time 1746133918.07 total 20.01 last 148.06) - open/close 4481 (time 1746133928.09 total 30.02 last 128.25) - open/close 5865 (time 1746133938.09 total 40.03 last 138.34) - open/close 7641 (time 1746133948.09 total 50.03 last 177.55) total: 10000 open/close in 58.77 seconds: 170.15 ops/second - unlinked 0 (time 1746133957 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2119 (time 1746133988.04 total 10.01 last 211.77) - open/close 3658 (time 1746133998.04 total 20.01 last 153.90) - open/close 5150 (time 1746134008.04 total 30.01 last 149.18) - open/close 6617 (time 1746134018.04 total 40.01 last 146.68) - open/close 8125 (time 1746134028.05 total 50.01 last 150.74) - open/close 9673 (time 1746134038.05 total 60.02 last 154.73) total: 10000 open/close in 63.23 seconds: 158.16 ops/second - unlinked 0 (time 1746134042 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2147 (time 1746134084.22 total 10.00 last 214.66) - open/close 4022 (time 1746134094.22 total 20.00 last 187.46) - open/close 6488 (time 1746134104.22 total 30.00 last 246.59) - open/close 8301 (time 1746134114.22 total 40.00 last 181.29) - open/close 9771 (time 1746134124.22 total 50.01 last 146.93) total: 10000 open/close in 51.55 seconds: 193.99 ops/second - unlinked 0 (time 1746134127 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1649 (time 1746134178.87 total 10.00 last 164.83) - open/close 3470 (time 1746134188.87 total 20.01 last 182.02) - open/close 5523 (time 1746134198.88 total 30.01 last 205.29) - open/close 7197 (time 1746134208.88 total 40.01 last 167.39) - open/close 8457 (time 1746134218.88 total 50.02 last 125.92) - open/close 9694 (time 1746134228.88 total 60.02 last 123.69) total: 10000 open/close in 62.37 seconds: 160.34 ops/second - unlinked 0 (time 1746134232 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1675 (time 1746134273.64 total 10.00 last 167.50) - open/close 3430 (time 1746134283.64 total 20.00 last 175.46) - open/close 5720 (time 1746134293.64 total 30.01 last 228.93) - open/close 8232 (time 1746134303.64 total 40.01 last 251.17) total: 10000 open/close in 46.30 seconds: 215.97 ops/second - unlinked 0 (time 1746134310 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2348 (time 1746134344.78 total 10.00 last 234.70) - open/close 4148 (time 1746134354.78 total 20.01 last 179.90) - open/close 5786 (time 1746134364.78 total 30.01 last 163.78) - open/close 7125 (time 1746134374.78 total 40.01 last 133.88) - open/close 8681 (time 1746134384.79 total 50.01 last 155.58) total: 10000 open/close in 56.20 seconds: 177.95 ops/second - unlinked 0 (time 1746134392 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1753 (time 1746134439.73 total 10.00 last 175.29) - open/close 3674 (time 1746134449.74 total 20.01 last 191.98) - open/close 5623 (time 1746134459.74 total 30.01 last 194.85) - open/close 7079 (time 1746134469.74 total 40.01 last 145.60) - open/close 8454 (time 1746134479.75 total 50.02 last 137.42) - open/close 9574 (time 1746134489.75 total 60.02 last 111.90) total: 10000 open/close in 63.03 seconds: 158.65 ops/second - unlinked 0 (time 1746134494 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1513 (time 1746134549.75 total 10.00 last 151.24) - open/close 3150 (time 1746134559.75 total 20.01 last 163.68) - open/close 4673 (time 1746134569.75 total 30.01 last 152.29) - open/close 6295 (time 1746134579.76 total 40.01 last 162.17) - open/close 8005 (time 1746134589.76 total 50.01 last 170.97) - open/close 9798 (time 1746134599.76 total 60.01 last 179.27) total: 10000 open/close in 61.45 seconds: 162.73 ops/second - unlinked 0 (time 1746134602 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1258 (time 1746134657.56 total 10.00 last 125.80) - open/close 2429 (time 1746134667.58 total 20.01 last 116.96) - open/close 3893 (time 1746134677.58 total 30.02 last 146.30) - open/close 5641 (time 1746134687.59 total 40.02 last 174.74) - open/close 7250 (time 1746134697.59 total 50.03 last 160.83) - open/close 8412 (time 1746134707.59 total 60.03 last 116.17) - open/close 9753 (time 1746134717.60 total 70.03 last 134.04) total: 10000 open/close in 71.60 seconds: 139.67 ops/second - unlinked 0 (time 1746134720 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.106@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.106@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg406-server: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg406-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre On OST0, 18357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2