== conf-sanity test 69: replace an OST with the same index ========================================================== 08:51:37 (1788439897) start mds service on oleg129-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: kmemleak disabled oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: MODOPTS_LIBCFS= oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg129-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg129-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 585 (time 1788439989.09 total 10.01 last 58.41) - open/close 1112 (time 1788439999.09 total 20.02 last 52.67) - open/close 1759 (time 1788440009.11 total 30.04 last 64.58) - open/close 2566 (time 1788440019.12 total 40.05 last 80.64) - open/close 3174 (time 1788440029.12 total 50.05 last 60.79) - open/close 3875 (time 1788440039.12 total 60.05 last 70.07) - open/close 4579 (time 1788440049.14 total 70.07 last 70.28) - open/close 5226 (time 1788440059.14 total 80.07 last 64.68) - open/close 5902 (time 1788440069.15 total 90.08 last 67.53) - open/close 6563 (time 1788440079.16 total 100.09 last 66.03) - open/close 7147 (time 1788440089.17 total 110.10 last 58.34) - open/close 7715 (time 1788440099.17 total 120.10 last 56.80) - open/close 8302 (time 1788440109.19 total 130.12 last 58.61) - open/close 8898 (time 1788440119.22 total 140.14 last 59.43) - open/close 9647 (time 1788440129.23 total 150.16 last 74.80) total: 10000 open/close in 155.80 seconds: 64.18 ops/second - unlinked 0 (time 1788440140 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 627 (time 1788440245.64 total 10.00 last 62.69) - open/close 1393 (time 1788440255.64 total 20.01 last 76.57) - open/close 1879 (time 1788440265.65 total 30.01 last 48.58) - open/close 2495 (time 1788440275.66 total 40.02 last 61.55) - open/close 3250 (time 1788440285.66 total 50.02 last 75.49) - open/close 3923 (time 1788440295.66 total 60.02 last 67.29) - open/close 4556 (time 1788440305.67 total 70.03 last 63.24) - open/close 5155 (time 1788440315.67 total 80.03 last 59.89) - open/close 5743 (time 1788440325.68 total 90.04 last 58.77) - open/close 6344 (time 1788440335.71 total 100.07 last 59.92) - open/close 6920 (time 1788440345.71 total 110.07 last 57.57) - open/close 7538 (time 1788440355.71 total 120.07 last 61.80) - open/close 8286 (time 1788440365.72 total 130.08 last 74.77) - open/close 9059 (time 1788440375.73 total 140.10 last 77.17) - open/close 9751 (time 1788440385.74 total 150.10 last 69.17) total: 10000 open/close in 153.79 seconds: 65.02 ops/second - unlinked 0 (time 1788440393 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 632 (time 1788440503.00 total 10.00 last 63.18) - open/close 1291 (time 1788440513.00 total 20.01 last 65.86) - open/close 1940 (time 1788440523.01 total 30.02 last 64.83) - open/close 2671 (time 1788440533.01 total 40.02 last 73.10) - open/close 3380 (time 1788440543.02 total 50.03 last 70.84) - open/close 4143 (time 1788440553.03 total 60.04 last 76.23) - open/close 4946 (time 1788440563.03 total 70.04 last 80.28) - open/close 5794 (time 1788440573.05 total 80.05 last 84.69) - open/close 6497 (time 1788440583.06 total 90.06 last 70.22) - open/close 7230 (time 1788440593.07 total 100.07 last 73.22) - open/close 7948 (time 1788440603.08 total 110.08 last 71.73) - open/close 8650 (time 1788440613.08 total 120.08 last 70.20) - open/close 9316 (time 1788440623.09 total 130.09 last 66.56) total: 10000 open/close in 139.64 seconds: 71.61 ops/second - unlinked 0 (time 1788440636 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 705 (time 1788440742.24 total 10.00 last 70.47) - open/close 1414 (time 1788440752.25 total 20.01 last 70.85) - open/close 2141 (time 1788440762.26 total 30.03 last 72.58) - open/close 2791 (time 1788440772.27 total 40.03 last 64.96) - open/close 3583 (time 1788440782.28 total 50.04 last 79.13) - open/close 4271 (time 1788440792.29 total 60.05 last 68.74) - open/close 4976 (time 1788440802.30 total 70.06 last 70.43) - open/close 5610 (time 1788440812.31 total 80.07 last 63.33) - open/close 6336 (time 1788440822.32 total 90.08 last 72.54) - open/close 6966 (time 1788440832.33 total 100.09 last 62.93) - open/close 7671 (time 1788440842.34 total 110.10 last 70.43) - open/close 8375 (time 1788440852.36 total 120.12 last 70.28) - open/close 9096 (time 1788440862.36 total 130.13 last 72.06) - open/close 9734 (time 1788440872.36 total 140.13 last 63.78) total: 10000 open/close in 144.07 seconds: 69.41 ops/second - unlinked 0 (time 1788440880 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 644 (time 1788440988.17 total 10.01 last 64.34) - open/close 1333 (time 1788440998.18 total 20.02 last 68.81) - open/close 2093 (time 1788441008.18 total 30.03 last 75.98) - open/close 2745 (time 1788441018.19 total 40.03 last 65.18) - open/close 3371 (time 1788441028.20 total 50.04 last 62.53) - open/close 3993 (time 1788441038.21 total 60.06 last 62.11) - open/close 4617 (time 1788441048.22 total 70.06 last 62.35) - open/close 5321 (time 1788441058.23 total 80.08 last 70.31) - open/close 5922 (time 1788441068.23 total 90.08 last 60.09) - open/close 6559 (time 1788441078.25 total 100.09 last 63.62) - open/close 7209 (time 1788441088.27 total 110.11 last 64.88) - open/close 7934 (time 1788441098.28 total 120.13 last 72.38) - open/close 8681 (time 1788441108.28 total 130.13 last 74.69) - open/close 9342 (time 1788441118.30 total 140.14 last 66.00) total: 10000 open/close in 149.48 seconds: 66.90 ops/second - unlinked 0 (time 1788441132 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 696 (time 1788441242.41 total 10.00 last 69.59) - open/close 1389 (time 1788441252.42 total 20.01 last 69.26) - open/close 2158 (time 1788441262.42 total 30.01 last 76.86) - open/close 2854 (time 1788441272.42 total 40.01 last 69.58) - open/close 3506 (time 1788441282.43 total 50.03 last 65.12) - open/close 4200 (time 1788441292.44 total 60.03 last 69.34) - open/close 4986 (time 1788441302.46 total 70.05 last 78.47) - open/close 5624 (time 1788441312.46 total 80.05 last 63.77) - open/close 6297 (time 1788441322.47 total 90.07 last 67.22) - open/close 6937 (time 1788441332.48 total 100.07 last 63.96) - open/close 7633 (time 1788441342.49 total 110.08 last 69.56) - open/close 8403 (time 1788441352.49 total 120.08 last 76.98) - open/close 9147 (time 1788441362.50 total 130.09 last 74.30) - open/close 9825 (time 1788441372.51 total 140.10 last 67.76) total: 10000 open/close in 142.42 seconds: 70.22 ops/second - unlinked 0 (time 1788441379 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 606 (time 1788441489.09 total 10.01 last 60.52) - open/close 1142 (time 1788441499.10 total 20.02 last 53.54) - open/close 1802 (time 1788441509.12 total 30.05 last 65.84) - open/close 2460 (time 1788441519.13 total 40.05 last 65.76) - open/close 3241 (time 1788441529.13 total 50.06 last 78.06) - open/close 3871 (time 1788441539.14 total 60.06 last 62.99) - open/close 4507 (time 1788441549.14 total 70.06 last 63.58) - open/close 5163 (time 1788441559.14 total 80.07 last 65.58) - open/close 5759 (time 1788441569.16 total 90.08 last 59.50) - open/close 6358 (time 1788441579.16 total 100.08 last 59.90) - open/close 6998 (time 1788441589.16 total 110.09 last 63.99) - open/close 7626 (time 1788441599.17 total 120.09 last 62.76) - open/close 8250 (time 1788441609.17 total 130.09 last 62.39) - open/close 8930 (time 1788441619.18 total 140.10 last 67.95) - open/close 9510 (time 1788441629.21 total 150.14 last 57.80) total: 10000 open/close in 157.92 seconds: 63.32 ops/second - unlinked 0 (time 1788441641 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 669 (time 1788441757.02 total 10.01 last 66.85) - open/close 1334 (time 1788441767.02 total 20.01 last 66.49) - open/close 2018 (time 1788441777.03 total 30.02 last 68.31) - open/close 2751 (time 1788441787.03 total 40.02 last 73.29) - open/close 3452 (time 1788441797.04 total 50.03 last 70.02) - open/close 4067 (time 1788441807.05 total 60.04 last 61.45) - open/close 4684 (time 1788441817.06 total 70.05 last 61.68) - open/close 5330 (time 1788441827.07 total 80.06 last 64.52) - open/close 5901 (time 1788441837.07 total 90.06 last 57.08) - open/close 6546 (time 1788441847.09 total 100.08 last 64.39) - open/close 7215 (time 1788441857.10 total 110.09 last 66.85) - open/close 7841 (time 1788441867.11 total 120.10 last 62.54) - open/close 8501 (time 1788441877.12 total 130.11 last 65.95) - open/close 9149 (time 1788441887.13 total 140.12 last 64.69) - open/close 9865 (time 1788441897.13 total 150.13 last 71.59) total: 10000 open/close in 152.13 seconds: 65.73 ops/second - unlinked 0 (time 1788441903 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 761 (time 1788442013.87 total 10.00 last 76.08) - open/close 1381 (time 1788442023.88 total 20.00 last 62.00) - open/close 2096 (time 1788442033.88 total 30.01 last 71.48) - open/close 2713 (time 1788442043.88 total 40.01 last 61.68) - open/close 3371 (time 1788442053.89 total 50.02 last 65.75) - open/close 4042 (time 1788442063.90 total 60.03 last 67.04) - open/close 4705 (time 1788442073.90 total 70.03 last 66.27) - open/close 5234 (time 1788442083.90 total 80.03 last 52.90) - open/close 5898 (time 1788442093.92 total 90.04 last 66.32) - open/close 6429 (time 1788442103.95 total 100.08 last 52.92) - open/close 7133 (time 1788442113.96 total 110.09 last 70.34) - open/close 7850 (time 1788442123.97 total 120.10 last 71.59) - open/close 8509 (time 1788442133.99 total 130.12 last 65.79) - open/close 9016 (time 1788442143.99 total 140.12 last 50.68) - open/close 9660 (time 1788442154.00 total 150.13 last 64.37) total: 10000 open/close in 155.80 seconds: 64.18 ops/second - unlinked 0 (time 1788442164 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 718 (time 1788442274.13 total 10.01 last 71.76) - open/close 1451 (time 1788442284.14 total 20.01 last 73.26) - open/close 2157 (time 1788442294.15 total 30.02 last 70.55) - open/close 2973 (time 1788442304.16 total 40.03 last 81.52) - open/close 3723 (time 1788442314.17 total 50.04 last 74.89) - open/close 4443 (time 1788442324.17 total 60.04 last 71.99) - open/close 5081 (time 1788442334.18 total 70.05 last 63.75) - open/close 5705 (time 1788442344.20 total 80.07 last 62.30) - open/close 6377 (time 1788442354.20 total 90.07 last 67.18) - open/close 6997 (time 1788442364.21 total 100.08 last 61.93) - open/close 7653 (time 1788442374.21 total 110.08 last 65.59) - open/close 8292 (time 1788442384.22 total 120.09 last 63.87) - open/close 8974 (time 1788442394.23 total 130.10 last 68.10) - open/close 9651 (time 1788442404.25 total 140.12 last 67.58) total: 10000 open/close in 144.95 seconds: 68.99 ops/second - unlinked 0 (time 1788442414 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg129-server: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg129-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.129@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg129-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg129-server: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg129-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock 192.168.201.129@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules via unload_modules_local on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local oleg129-server: kmemleak disabled modules unloaded. pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2