== conf-sanity test 69: replace an OST with the same index ========================================================== 21:20:07 (1789176007) start mds service on oleg623-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 11 09:35 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing load_modules_local oleg623-server: kmemleak disabled oleg623-server: Loading modules from /home/green/git/lustre-release/lustre oleg623-server: detected 4 online CPUs by sysfs oleg623-server: MODOPTS_LIBCFS= oleg623-server: Force libcfs to create 2 CPU partitions oleg623-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg623-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg623-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg623-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg623-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 662 (time 1789176087.36 total 10.01 last 66.15) - open/close 1267 (time 1789176097.38 total 20.02 last 60.40) - open/close 1829 (time 1789176107.38 total 30.03 last 56.17) - open/close 2462 (time 1789176117.39 total 40.04 last 63.26) - open/close 3180 (time 1789176127.41 total 50.06 last 71.65) - open/close 3813 (time 1789176137.42 total 60.07 last 63.22) - open/close 4518 (time 1789176147.42 total 70.07 last 70.50) - open/close 5254 (time 1789176157.44 total 80.08 last 73.50) - open/close 5944 (time 1789176167.44 total 90.09 last 68.98) - open/close 6665 (time 1789176177.45 total 100.09 last 72.04) - open/close 7380 (time 1789176187.45 total 110.10 last 71.46) - open/close 8124 (time 1789176197.46 total 120.11 last 74.32) - open/close 8854 (time 1789176207.46 total 130.11 last 72.99) - open/close 9606 (time 1789176217.47 total 140.12 last 75.16) total: 10000 open/close in 145.71 seconds: 68.63 ops/second - unlinked 0 (time 1789176227 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 760 (time 1789176327.42 total 10.01 last 75.91) - open/close 1551 (time 1789176337.43 total 20.02 last 79.04) - open/close 2333 (time 1789176347.43 total 30.03 last 78.16) - open/close 3038 (time 1789176357.44 total 40.03 last 70.45) - open/close 3729 (time 1789176367.45 total 50.04 last 69.05) - open/close 4476 (time 1789176377.46 total 60.05 last 74.62) - open/close 5194 (time 1789176387.46 total 70.05 last 71.78) - open/close 5973 (time 1789176397.47 total 80.06 last 77.87) - open/close 6719 (time 1789176407.47 total 90.06 last 74.56) - open/close 7446 (time 1789176417.48 total 100.07 last 72.65) - open/close 8109 (time 1789176427.49 total 110.08 last 66.22) - open/close 8828 (time 1789176437.49 total 120.09 last 71.87) - open/close 9450 (time 1789176447.51 total 130.10 last 62.13) total: 10000 open/close in 137.83 seconds: 72.55 ops/second - unlinked 0 (time 1789176459 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 722 (time 1789176558.06 total 10.01 last 72.13) - open/close 1404 (time 1789176568.06 total 20.01 last 68.20) - open/close 2061 (time 1789176578.07 total 30.02 last 65.64) - open/close 2621 (time 1789176588.07 total 40.02 last 55.98) - open/close 3368 (time 1789176598.08 total 50.03 last 74.61) - open/close 4078 (time 1789176608.10 total 60.05 last 70.85) - open/close 4777 (time 1789176618.11 total 70.06 last 69.87) - open/close 5514 (time 1789176628.11 total 80.06 last 73.69) - open/close 6232 (time 1789176638.11 total 90.06 last 71.79) - open/close 6931 (time 1789176648.11 total 100.06 last 69.89) - open/close 7666 (time 1789176658.12 total 110.07 last 73.45) - open/close 8398 (time 1789176668.12 total 120.07 last 73.17) - open/close 9142 (time 1789176678.13 total 130.08 last 74.35) - open/close 9941 (time 1789176688.14 total 140.09 last 79.81) total: 10000 open/close in 140.83 seconds: 71.01 ops/second - unlinked 0 (time 1789176692 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 775 (time 1789176790.97 total 10.01 last 77.46) - open/close 1511 (time 1789176800.99 total 20.02 last 73.48) - open/close 2255 (time 1789176810.99 total 30.03 last 74.34) - open/close 3004 (time 1789176821.00 total 40.03 last 74.89) - open/close 3695 (time 1789176831.00 total 50.03 last 69.09) - open/close 4460 (time 1789176841.00 total 60.04 last 76.45) - open/close 5208 (time 1789176851.01 total 70.05 last 74.72) - open/close 5880 (time 1789176861.04 total 80.07 last 67.06) - open/close 6631 (time 1789176871.05 total 90.08 last 75.03) - open/close 7304 (time 1789176881.05 total 100.08 last 67.29) - open/close 7940 (time 1789176891.07 total 110.10 last 63.46) - open/close 8641 (time 1789176901.07 total 120.11 last 70.07) - open/close 9338 (time 1789176911.09 total 130.12 last 69.60) total: 10000 open/close in 138.69 seconds: 72.10 ops/second - unlinked 0 (time 1789176923 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 704 (time 1789177023.41 total 10.01 last 70.32) - open/close 1415 (time 1789177033.41 total 20.01 last 71.09) - open/close 2182 (time 1789177043.42 total 30.02 last 76.68) - open/close 2901 (time 1789177053.42 total 40.02 last 71.88) - open/close 3649 (time 1789177063.42 total 50.02 last 74.80) - open/close 4418 (time 1789177073.43 total 60.03 last 76.84) - open/close 5164 (time 1789177083.43 total 70.04 last 74.54) - open/close 5895 (time 1789177093.44 total 80.04 last 73.09) - open/close 6647 (time 1789177103.44 total 90.04 last 75.19) - open/close 7363 (time 1789177113.45 total 100.05 last 71.50) - open/close 8116 (time 1789177123.47 total 110.07 last 75.16) - open/close 8865 (time 1789177133.48 total 120.08 last 74.86) - open/close 9563 (time 1789177143.49 total 130.09 last 69.69) total: 10000 open/close in 136.58 seconds: 73.22 ops/second - unlinked 0 (time 1789177155 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 789 (time 1789177249.92 total 10.00 last 78.89) - open/close 1525 (time 1789177259.92 total 20.00 last 73.57) - open/close 2267 (time 1789177269.93 total 30.01 last 74.17) - open/close 3003 (time 1789177279.94 total 40.02 last 73.51) - open/close 3726 (time 1789177289.95 total 50.03 last 72.21) - open/close 4553 (time 1789177299.96 total 60.04 last 82.66) - open/close 5230 (time 1789177309.96 total 70.04 last 67.70) - open/close 5913 (time 1789177319.96 total 80.04 last 68.27) - open/close 6589 (time 1789177329.97 total 90.06 last 67.52) - open/close 7237 (time 1789177339.98 total 100.06 last 64.78) - open/close 7899 (time 1789177349.98 total 110.06 last 66.19) - open/close 8646 (time 1789177359.99 total 120.07 last 74.67) - open/close 9504 (time 1789177369.99 total 130.07 last 85.77) total: 10000 open/close in 136.54 seconds: 73.24 ops/second - unlinked 0 (time 1789177380 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 726 (time 1789177475.08 total 10.01 last 72.51) - open/close 1449 (time 1789177485.09 total 20.02 last 72.25) - open/close 2179 (time 1789177495.10 total 30.03 last 72.96) - open/close 2950 (time 1789177505.10 total 40.03 last 77.06) - open/close 3695 (time 1789177515.11 total 50.04 last 74.46) - open/close 4457 (time 1789177525.12 total 60.05 last 76.12) - open/close 5217 (time 1789177535.12 total 70.05 last 75.98) - open/close 6000 (time 1789177545.13 total 80.06 last 78.22) - open/close 6798 (time 1789177555.13 total 90.07 last 79.76) - open/close 7548 (time 1789177565.14 total 100.07 last 74.98) - open/close 8301 (time 1789177575.14 total 110.07 last 75.28) - open/close 9043 (time 1789177585.14 total 120.07 last 74.17) - open/close 9814 (time 1789177595.16 total 130.09 last 76.96) total: 10000 open/close in 132.62 seconds: 75.40 ops/second - unlinked 0 (time 1789177601 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 714 (time 1789177696.63 total 10.01 last 71.33) - open/close 1163 (time 1789177706.66 total 20.03 last 44.80) - open/close 1594 (time 1789177716.66 total 30.04 last 43.08) - open/close 2237 (time 1789177726.68 total 40.05 last 64.20) - open/close 2948 (time 1789177736.69 total 50.06 last 71.02) - open/close 3695 (time 1789177746.69 total 60.07 last 74.69) - open/close 4427 (time 1789177756.69 total 70.07 last 73.14) - open/close 5150 (time 1789177766.70 total 80.08 last 72.23) - open/close 5955 (time 1789177776.71 total 90.08 last 80.49) - open/close 6730 (time 1789177786.71 total 100.08 last 77.49) - open/close 7451 (time 1789177796.72 total 110.09 last 72.04) - open/close 8215 (time 1789177806.73 total 120.11 last 76.28) - open/close 8907 (time 1789177816.74 total 130.12 last 69.12) - open/close 9618 (time 1789177826.74 total 140.12 last 71.09) total: 10000 open/close in 144.94 seconds: 69.00 ops/second - unlinked 0 (time 1789177835 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 742 (time 1789177928.53 total 10.03 last 74.01) - open/close 1485 (time 1789177938.53 total 20.03 last 74.28) - open/close 2237 (time 1789177948.54 total 30.04 last 75.13) - open/close 2961 (time 1789177958.56 total 40.05 last 72.27) - open/close 3702 (time 1789177968.57 total 50.07 last 74.01) - open/close 4415 (time 1789177978.57 total 60.07 last 71.26) - open/close 5180 (time 1789177988.58 total 70.08 last 76.46) - open/close 5913 (time 1789177998.58 total 80.08 last 73.28) - open/close 6693 (time 1789178008.59 total 90.08 last 77.98) - open/close 7428 (time 1789178018.59 total 100.09 last 73.45) - open/close 8156 (time 1789178028.60 total 110.10 last 72.77) - open/close 8940 (time 1789178038.60 total 120.10 last 78.36) - open/close 9611 (time 1789178048.61 total 130.11 last 67.02) total: 10000 open/close in 136.04 seconds: 73.51 ops/second - unlinked 0 (time 1789178061 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 701 (time 1789178169.78 total 10.01 last 70.06) - open/close 1444 (time 1789178179.78 total 20.01 last 74.27) - open/close 2290 (time 1789178189.79 total 30.02 last 84.53) - open/close 3030 (time 1789178199.80 total 40.03 last 73.91) - open/close 3652 (time 1789178209.81 total 50.03 last 62.17) - open/close 4357 (time 1789178219.82 total 60.05 last 70.38) - open/close 5026 (time 1789178229.83 total 70.06 last 66.84) - open/close 5715 (time 1789178239.86 total 80.08 last 68.74) - open/close 6462 (time 1789178249.88 total 90.11 last 74.50) - open/close 7188 (time 1789178259.89 total 100.12 last 72.52) - open/close 7852 (time 1789178269.89 total 110.12 last 66.40) - open/close 8513 (time 1789178279.91 total 120.14 last 66.00) - open/close 9254 (time 1789178289.91 total 130.14 last 74.08) total: 10000 open/close in 140.14 seconds: 71.36 ops/second - unlinked 0 (time 1789178303 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.123@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.123@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg623-server: oleg623-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg623-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.123@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.123@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.206.123@tcp autodegrade=on sys.timeout=20 Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.123@tcp autodegrade=on sys.timeout=20 autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.123@tcp lustre:autodegrade=on lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg623-server: oleg623-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg623-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server stop mds service on oleg623-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg623-server unloading modules via unload_modules_local on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing unload_modules_local oleg623-server: kmemleak disabled modules unloaded. pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2